UniFi Controller 5.8 on Debian 9

 

VPS services in Australia have become cheap enough for me to have a dedicated VM to run a UniFi controller, which changes how I’ve gone about setting it all up as compared with a past effort.

The UniFi controller is pretty memory hungry; I’ve got my VM configured with 1.5 GB of RAM and wouldn’t recommend anything less than that. 2 GB of RAM may be better on busier controllers. Aside from RAM, the system requirements are fairly minimal.

This recipe includes setting up the pre-requisites, installing UniFi, adding a firewall and an automatically-updating SSL certificate from Let’s Encrypt.

Pre-requisites

It’s a brand new VM, so make sure everything is up to date – plus install a few dependencies. I’ve had problems with package dependency conflicts when installing the UniFi .deb file, so I prefer to get a few things installed ahead of time.

Note that the versions of the Linux kernel and MongoDB are important! If you’re using this guide with versions of Linux other than Debian 9 – the latest version of Ubuntu has an incompatible version of MongoDB and earlier versions of Debian need a patched kernel. Caveat emptor.. 😉

sudo apt-get update && sudo apt-get upgrade && sudo apt-get dist-upgrade
sudo apt-get install mongodb openjdk-8-jre-headless jsvc ufw curl

Install the UniFi controller

As of now, version 5.8 of the UniFi controller is limited to beta testers – but joining the beta program is easy/quick/painless.

The latest 5.8 controller is available in the UniFi Beta Blog – download the Debian/Ubuntu .deb file and run the following:

sudo dpkg -i unifi_sysvinit_all.deb

If you’re upgrading from an earlier release, expect things to take a little while to stabilise post-install.

Configure the ufw firewall

If you’ve never configured ufw before, please see this DigitalOcean guide – it’s pretty well written and covers how to open up SSH. For obvious reasons, don’t turn on the firewall if you haven’t confirmed that you’ll still have access afterwards!

Create /etc/ufw/applications.d/unifi:

[UniFi]
title=UniFi Controller
description=UniFi Controller
ports=8080,8443,8880,8843/tcp|3478/udp

Enable the UniFi rules and confirm they’re working:

sudo ufw allow unifi
sudo ufw status verbose

Add Let’s Encrypt SSL

By default, the UniFi controller includes an untrusted certificate – which is almost useless, given how much the browsers complain about such certificates. While you can definitely get a certificate from a commercial CA, getting a certificate from Let’s Encrypt is fairly painless (and free!)

The first time you run the certbot command, you’ll need to agree to the Terms of Service and provide an email address – this is a first-time-only thing. You’ll also need to update the DNS name in the second command:

sudo apt-get install certbot
sudo certbot certonly --standalone --preferred-challenges http --pre-hook "ufw allow http" --post-hook "ufw deny http" -d your.dns.name.here

Once you’ve got the certificate, run a script to replace the certificate that comes with the UniFi controller, then add the script to cron to ensure the certificate is renewed and installed into the UniFi controller automagically.

Create /usr/local/bin/unifi-ssl-import.sh:

#!/usr/bin/env bash

# unifi_ssl_import.sh
# UniFi Controller SSL Certificate Import Script for Unix/Linux Systems
# by Steve Jenkins 
# Part of https://github.com/stevejenkins/ubnt-linux-utils/
# Incorporates ideas from https://source.sosdg.org/brielle/lets-encrypt-scripts
# Version 2.8
# Last Updated Jan 13, 2017

# Minor adjustments by Zac Ariel, called out with {ZA}
# Updated Sep 28, 2017

# REQUIREMENTS
# 1) Assumes you have a UniFi Controller installed and running on your system.
# 2) Assumes you already have a valid 2048-bit private key, signed certificate, and certificate authority
#    chain file. The Controller UI will not work with a 4096-bit certificate. See http://wp.me/p1iGgP-2wU
#    for detailed instructions on how to generate those files and use them with this script.

# KEYSTORE BACKUP
# Even though this script attempts to be clever and careful in how it backs up your existing keystore,
# it's never a bad idea to manually back up your keystore (located at $UNIFI_DIR/data/keystore on RedHat
# systems or /$UNIFI_DIR/keystore on Debian/Ubuntu systems) to a separate directory before running this
# script. If anything goes wrong, you can restore from your backup, restart the UniFi Controller service,
# and be back online immediately.

# CONFIGURATION OPTIONS
#UNIFI_HOSTNAME=hostname.example.com
UNIFI_HOSTNAME=your.dns.name.here
UNIFI_SERVICE=unifi

# Uncomment following three lines for Fedora/RedHat/CentOS
#UNIFI_DIR=/opt/UniFi
#JAVA_DIR=${UNIFI_DIR}
#KEYSTORE=${UNIFI_DIR}/data/keystore

# Uncomment following three lines for Debian/Ubuntu
UNIFI_DIR=/var/lib/unifi
JAVA_DIR=/usr/lib/unifi
KEYSTORE=${UNIFI_DIR}/keystore

# FOR LET'S ENCRYPT SSL CERTIFICATES ONLY
# Generate your Let's Encrtypt key & cert with certbot before running this script
#LE_MODE=no
LE_MODE=yes
LE_LIVE_DIR=/etc/letsencrypt/live

# THE FOLLOWING OPTIONS NOT REQUIRED IF LE_MODE IS ENABLED
#PRIV_KEY=/etc/ssl/private/hostname.example.com.key
#SIGNED_CRT=/etc/ssl/certs/hostname.example.com.crt
#CHAIN_FILE=/etc/ssl/certs/startssl-chain.crt

# CONFIGURATION OPTIONS YOU PROBABLY SHOULDN'T CHANGE
ALIAS=unifi
PASSWORD=aircontrolenterprise

#### SHOULDN'T HAVE TO TOUCH ANYTHING PAST THIS POINT ####

printf "\nStarting UniFi Controller SSL Import...\n"

# Check to see whether Let's Encrypt Mode (LE_MODE) is enabled

if [[ ${LE_MODE} == "YES" || ${LE_MODE} == "yes" || ${LE_MODE} == "Y" || ${LE_MODE} == "y" || ${LE_MODE} == "TRUE" || ${LE_MODE} == "true" || ${LE_MODE} == "ENABLED" || ${LE_MODE} == "enabled" || ${LE_MODE} == 1 ]] ; then
        LE_MODE=true
        printf "\nRunning in Let's Encrypt Mode...\n"
        PRIV_KEY=${LE_LIVE_DIR}/${UNIFI_HOSTNAME}/privkey.pem
        SIGNED_CRT=${LE_LIVE_DIR}/${UNIFI_HOSTNAME}/cert.pem
        CHAIN_FILE=${LE_LIVE_DIR}/${UNIFI_HOSTNAME}/chain.pem
else
        LE_MODE=false
        printf "\nRunning in Standard Mode...\n"
fi

if [ ${LE_MODE} == "true" ]; then
        # Check to see whether LE certificate has changed
        printf "\nInspecting current SSL certificate...\n"
        if md5sum -c ${LE_LIVE_DIR}/${UNIFI_HOSTNAME}/cert.pem.md5 &>/dev/null; then
                # MD5 remains unchanged, exit the script
                printf "\nCertificate is unchanged, no update is necessary.\n"
                exit 0
        else
        # MD5 is different, so it's time to get busy!
        printf "\nUpdated SSL certificate available. Proceeding with import...\n"
        fi
fi

# Verify required files exist
if [ ! -f ${PRIV_KEY} ] || [ ! -f ${SIGNED_CRT} ] || [ ! -f ${CHAIN_FILE} ]; then
        printf "\nMissing one or more required files. Check your settings.\n"
        exit 1
else
        # Everything looks OK to proceed
        printf "\nImporting the following files:\n"
        printf "Private Key: %s\n" "$PRIV_KEY"
        printf "Signed Certificate: %s\n" "$SIGNED_CRT"
        printf "CA File: %s\n" "$CHAIN_FILE"
fi

# Create temp files
P12_TEMP=$(mktemp)
CA_TEMP=$(mktemp)

# Stop the UniFi Controller
printf "\nStopping UniFi Controller...\n"
# {ZA} update to a systemd style command
#service ${UNIFI_SERVICE} stop
systemctl stop ${UNIFI_SERVICE}

if [ ${LE_MODE} == "true" ]; then
        
        # Write a new MD5 checksum based on the updated certificate     
        printf "\nUpdating certificate MD5 checksum...\n"

        md5sum ${LE_LIVE_DIR}/${UNIFI_HOSTNAME}/cert.pem > ${LE_LIVE_DIR}/${UNIFI_HOSTNAME}/cert.pem.md5 
        
        # Create local copy of cross-signed CA File (required for keystore import)
        # Verify original @ https://www.identrust.com/certificates/trustid/root-download-x3.html
  cat > "${CA_TEMP}" <<'_EOF'
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----
_EOF
fi

# Create double-safe keystore backup
if [ -s "${KEYSTORE}.orig" ]; then
        printf "\nBackup of original keystore exists!\n"
        printf "\nCreating non-destructive backup as keystore.bak...\n"
        cp ${KEYSTORE} ${KEYSTORE}.bak
else
        cp ${KEYSTORE} ${KEYSTORE}.orig
        printf "\nNo original keystore backup found.\n"
        printf "\nCreating backup as keystore.orig...\n"
fi
         
# Export your existing SSL key, cert, and CA data to a PKCS12 file
printf "\nExporting SSL certificate and key data into temporary PKCS12 file...\n"

openssl pkcs12 -export \
-in ${SIGNED_CRT} \
-inkey ${PRIV_KEY} \
-CAfile ${CHAIN_FILE} \
-out ${P12_TEMP} -passout pass:${PASSWORD} \
-caname root -name ${ALIAS}
        
# Delete the previous certificate data from keystore to avoid "already exists" message
printf "\nRemoving previous certificate data from UniFi keystore...\n"
keytool -delete -alias ${ALIAS} -keystore ${KEYSTORE} -deststorepass ${PASSWORD}
        
# Import the temp PKCS12 file into the UniFi keystore
printf "\nImporting SSL certificate into UniFi keystore...\n"
keytool -importkeystore \
-srckeystore ${P12_TEMP} -srcstoretype PKCS12 \
-srcstorepass ${PASSWORD} \
-destkeystore ${KEYSTORE} \
-deststorepass ${PASSWORD} \
-destkeypass ${PASSWORD} \
-alias ${ALIAS} -trustcacerts

# Import the certificate authority data into the UniFi keystore
printf "\nImporting certificate authority into UniFi keystore...\n\n"
if [ ${LE_MODE} == "true" ]; then
        # Import with additional cross-signed CA file
        java -jar ${JAVA_DIR}/lib/ace.jar import_cert \
        ${SIGNED_CRT} \
        ${CHAIN_FILE} \
        ${CA_TEMP}
else
        # Import in standard mode
        java -jar ${JAVA_DIR}/lib/ace.jar import_cert \
        ${SIGNED_CRT} \
        ${CHAIN_FILE}
fi      

# Clean up temp files
printf "\nRemoving temporary files...\n"
rm -f ${P12_TEMP}
rm -f ${CA_TEMP}
        
# Restart the UniFi Controller to pick up the updated keystore
printf "\nRestarting UniFi Controller to apply new Let's Encrypt SSL certificate...\n"
# {ZA} update to a systemd style command
#service ${UNIFI_SERVICE} start
systemctl start ${UNIFI_SERVICE}

# That's all, folks!
printf "\nDone!\n"

exit 0

Make the file executable and then run it to ensure it actually replaces the certificate in the UniFi controller – note that this script may take some time to complete:

sudo chmod 755 /usr/local/bin/unifi-ssl-import.sh
sudo /usr/local/bin/unifi-ssl-import.sh

Once the import process is working properly, automate it with a daily cron job.

Create /etc/cron.daily/unifi-ssl-import:

#!/bin/sh
/usr/local/bin/unifi-ssl-import.sh

Finally, make the cron script executable:

sudo chmod 755 /etc/cron.daily/unifi-ssl-import

HTTP/2 with Apache 2.4 on Debian 9 (Stretch)

Note: there are multiple reported security issues with HTTP/2 and Apache versions less than 2.4.26 – Debian 9 includes 2.4.25, so I’ve disabled the below for now.

A quick Google didn’t tell me the answer, but setting up HTTP/2 with Apache 2.4 on Debian Stretch is way easier than I thought it would be:

Create the following file: /etc/apache2/conf-available/http2.conf

Protocols h2 http/1.1

Run the following three commands:

sudo a2enmod http2
sudo a2enconf http2
sudo systemctl restart apache2

You can add more, but so long as you’re specifying a reasonable list of SSL ciphers (and if you’re using Let’s Encrypt – you’re already doing so), the defaults for other settings are probably fine.

Rspamd with Postfix and Dovecot in Debian Stretch

As far as spam filters go, SpamAssassin isn’t terrible – but it’s slow and chews through more CPU than I’d like. As I needed to set up a new mail server, I decided to look for alternatives – and after reading many positive reviews, I decided to have a go at running Rspamd instead of setting up SpamAssassin again. I was also able to do away with Policyd-SPF/OpenDKIM/OpenDMARC, as Rspamd handles all these functions as well!

Like pretty much every other antispam suite, Rspamd needs quite a bit of configuration – here are my notes for a minimally configured setup that seems to do a decent job. For larger servers with a higher email volume, further tuning will almost certainly be required – but this should get you going.

Assumptions

Here’s what you should already have:

  • A Debian Stretch server with Internet access
    (this guide will probably also work on Debian Jessie and versions of Ubuntu, but I haven’t tested it)
  • A functioning LAMP stack
    (note that I’m using Apache rather than nginx)
  • Postfix and Dovecot already configured and delivering mail
    (I recommend Christoph Hass’ ISPMail tutorials – just skip the Spamassassin instructions)
  • Ability to update DNS records for your domain(s)
  • Knowledge of how and when to use sudo

Overview

Here’s what you need to do:

Install Rspamd and Redis

Note: if you’re following the ISPMail guide (or other similar guide) for Postfix and Dovecot, don’t install spamassassin or spamass-milter!

The packages for Rspamd in Stretch are out of date and unsupported – add the official Rspamd repository before installing Rspamd and Redis.

First, create /etc/apt/sources.list.d/rspamd.list and add the following line (you’ll need to change the release code name if you’re not using Stretch):

deb http://rspamd.com/apt-stable/ stretch main

Next, get the GPG key and install stuff:

wget https://rspamd.com/apt-stable/gpg.key
apt-key add gpg.key
apt-get update
apt-get install rspamd redis-server

Create config files for Rspamd and Redis

First, create a secure password for the Rspamd controller using rspamadm – you’ll need the output from this command in one of the config files:

rspamadm pw

Next, create a bunch of config files – these can be adjusted to taste, but the aim here is a minimal config to get things up and running – you can check the official documentation for additional options.

Set host and port for the main Rspamd worker:
/etc/rspamd/local.d/worker-normal.inc

bind_socket = "localhost:11333";

Configure the Rspamd proxy to talk to Postfix via a milter:
/etc/rspamd/local.d/worker-proxy.inc

bind_socket = "localhost:11332";
milter = yes;
timeout = 120s;
upstream "local" {
  default = yes;
  self_scan = yes;
}

Configure a socket for the Rspamd controller (used for the web interface and for Dovecot sieve filters). This is where you use the password generated by rspamadm earlier – you may want to use separate standard and “enable” (superuser) passwords:
/etc/rspamd/local.d/worker-controller.inc

password = "$2$s164jny.....";
enable_password = "$2$s164jny.....";
bind_socket = "localhost:11334";

Make the Rspamd bayes classifier use Redis:
/etc/rspamd/local.d/classifier-bayes.conf

servers = "127.0.0.1";
backend = "redis";
autolearn = true;

Choose which headers get added to emails. This is a matter of personal taste, but I like the following as it’s not too verbose:
/etc/rspamd/local.d/milter_headers.conf

use = ["authentication-results", "x-spam-status"];
authenticated_headers = ["authentication-results"];

If a user has replied to an email, don’t mark other emails in the same thread as spam:
/etc/rspamd/local.d/replies.conf

action = "no action";

Add URL redirect checks to the URL blacklist check – note that hitting some of these blacklist services more frequently might require purchasing a license:
/etc/rspamd/local.d/surbl.conf

redirector_hosts_map = "/etc/rspamd/redirectors.inc";

Assign dynamic reputation to certain TLDs:
/etc/rspamd/local.d/url_reputation.conf

enabled = true;

Cache URL tags in Redis:
/etc/rspamd/local.d/url_tags.conf

enabled = true;

Where to find Redis:
/etc/rspamd/local.d/redis.conf

servers = "127.0.0.1";

Now we know where to find Redis, adjust Redis’ configuration so it binds to the appropriate IP addresses and is a little less memory hungry – /etc/redis/redis.conf already exists, you just need to adjust one line and add two others:

bind 127.0.0.1 ::1
maxmemory 500mb
maxmemory-policy volatile-lru

A couple other configuration items to consider –

Configure a local DNS resolver – not really necessary for low volumes of mail, as Rspamd will use whatever is in /etc/resolv.conf – however for higher volumes of mail, install something like Unbound and then:
/etc/rspamd/local.d/options.inc

dns {
  nameserver = ["127.0.0.1:53:1"];
}

Check for phishing emails. Phishing checks chew up an insane amount of memory (~800 MB for the few minutes I had it switched on), so you may not want this..:
/etc/rspamd/local.d/phishing.conf

openphish_enabled = true;
phishtank_enabled = true;

Get Postfix talking to Rspamd

We’ve already set up Rspamd to talk to Postfix via the milter protocol – now we need to set up Postfix to join the conversation. Update /etc/postfix/main.cf – if you already have a milter section, it will need to be adjusted:

milter_protocol = 6
milter_default_action = accept
smtpd_milters = inet:localhost:11332
non_smtpd_milters = $smtpd_milters
milter_mail_macros =  i {mail_addr} {client_addr} {client_name} {auth_authen}

Now that Rspamd and Postfix are talking to each other, restart both services. New inbound email should have headers like “X-Spam-Status:” added by Rspamd.

systemctl restart postfix
systemctl restart rspamd

Get Dovecot’s sieve and imapsieve plugins to train Rspamd

If Rspamd gets spam detection wrong for a given email, users can retrain Rspamd themselves by moving the email either out of the Junk folder or into the Junk folder as appropriate.

First, enable a couple sieve plugins within the protocol imap { } and protocol lmtp { } sections:

/etc/dovecot/conf.d/20-imap.conf

protocol imap {
  mail_plugins = $mail_plugins imap_sieve
}

/etc/dovecot/conf.d/20-lmtp.conf

protocol lmtp {
  mail_plugins = $mail_plugins sieve
}

Next, create a directory for the sieve filters:

mkdir /etc/dovecot/sieve

Tell Dovecot to pay attention to emails being shifted between folders and to trigger the sieve filters:

/etc/dovecot/conf.d/90-imapsieve.conf

plugin {
  sieve_plugins = sieve_imapsieve sieve_extprograms

  # From elsewhere to Junk folder
  imapsieve_mailbox1_name = Junk
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_before = file:/etc/dovecot/sieve/report-spam.sieve

  # From Junk folder to elsewhere
  imapsieve_mailbox2_name = *
  imapsieve_mailbox2_from = Junk
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_before = file:/etc/dovecot/sieve/report-ham.sieve

  sieve_pipe_bin_dir = /etc/dovecot/sieve

  sieve_global_extensions = +vnd.dovecot.pipe
}

Create the relevant sieve filters:

/etc/dovecot/sieve/report-spam.sieve

require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"];

if environment :matches "imap.email" "*" {
  set "email" "${1}";
}

pipe :copy "learn-spam.sh" [ "${email}" ];

/etc/dovecot/sieve/report-ham.sieve

require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"];

if environment :matches "imap.mailbox" "*" {
  set "mailbox" "${1}";
}

if string "${mailbox}" "Trash" {
  stop;
}

if environment :matches "imap.user" "*" {
  set "username" "${1}";
}

pipe :copy "learn-ham.sh" [ "${email}" ];

Restart Dovecot, then compile the two sieve filters:

systemctl restart dovecot
sievec /etc/dovecot/sieve/report-spam.sieve
sievec /etc/dovecot/sieve/report-ham.sieve

Finally, add the scripts referenced in the sieve filters that tell Rspamd to learn as spam or learn as ham:

/etc/dovecot/sieve/learn-spam.sh

exec /usr/bin/rspamc -h localhost:11334 learn_spam

/etc/dovecot/sieve/learn-ham.sh

exec /usr/bin/rspamc -h localhost:11334 learn_ham

Make the scripts executable, then restart Dovecot one more time:

chmod 755 /etc/dovecot/sieve/learn-spam.sh
chmod 755 /etc/dovecot/sieve/learn-ham.sh
systemctl restart dovecot

Create SPF records for your domain

Configuring outbound SPF involves working out what mail servers you could potentially send through and then crafting an appropriate DNS TXT record (note that DNS SPF records have been deprecated and shouldn’t be used).

There are plenty of decent wizards and tutorials out there, but I like the easySPF wizard. Because I run my own mail server and never, ever send through any other mail service, my record looks like this:

example.com. IN TXT "v=spf1 mx -all"

If I also wanted to use Gmail or Google Apps to send email from my domain, I’d need to change it to look like this:

example.com. IN TXT "v=spf1 mx include:_spf.google.com -all"

Configure DKIM signing in Rspamd

DKIM won’t improve spam detection rates, so this section can be skipped – but valid DKIM signatures are often used by antispam systems as a negative score, so it may improve delivery rates for your outbound email.

Thankfully, DKIM signing in Rspamd is even easier than setting up OpenDKIM! Create a directory for DKIM keys and then generate a key – I’ve used the domain name in the file name (this makes it easier to have different keys for different domains), and I’m using “dkim” as a selector (use something like the server name for multi-server configs) – adjust the file name in the command below to taste:

sudo mkdir /var/lib/rspamd/dkim
sudo rspamadm dkim_keygen -k /var/lib/rspamd/dkim/example.com.dkim.key -b 2048 -s dkim -d example.com

The rspamadm dkim_keygen command above will create the private key file and display the public key for the DNS TXT record on screen – the public key isn’t saved anywhere, so add the DNS TXT record immediately as follows:

  • The name of the record should be the selector, followed by _domainkey, followed by the domain
    (so dkim._domainkey.example.com for this example).
  • The value for the record needs to be copied carefully – the three lines should end up on a single line in the record, with spaces in between each:
dkim._domainkey.example.com. IN TXT "v=DKIM1; k=rsa;" "p=MIIBI.....HtByA" "504pO.....DAQAB"

Protect the private key:

chown -R _rspamd._rspamd /var/lib/rspamd/dkim
chmod 640 /var/lib/rspamd/dkim/example.com.dkim.key

Enable DKIM signing and ARC signing in Rspamd with two new config files – both files have the same content, but they both need to exist. Note that both the domain and selector are called out – this is to allow for multiple domains and different keys. Additionally, allow username mismatches when signing:
/etc/rspamd/local.d/dkim_signing.conf
/etc/rspamd/local.d/arc.conf

path = "/var/lib/rspamd/dkim/$domain.$selector.key";
selector = "dkim";
allow_username_mismatch = true;

Restart Rspamd:

systemctl restart rspamd

It’s extremely important to test the SPF and DKIM setup at this point – don’t just assume everything is working. Incorrect SPF/DKIM configs will break email delivery.

First, wait for your DNS updates to propagate – then check the SPF record and the DKIM record to make sure they are valid.

If these checks pass, use DKIMValidator.com to ensure that outbound mail is being correctly signed.

Configure Apache to reverse proxy Rspamd’s web interface

If you’ve already got admin websites up and running, add access to the shiny Rspamd web interface by using an Apache reverse proxy – first, enable two Apache modules:

a2enmod proxy
a2enmod proxy_http

Add the following to an existing virtual host – the Rspamd web interface will be available at https://example.com/rspamd/ (note the trailing slash):

RewriteEngine On
RewriteRule ^/rspamd/(.*) http://127.0.0.1:11334/$1 [P,L]
<Location /rspamd>
  Options FollowSymLinks
  Require all granted
</Location>

Restart Apache:

systemctl restart apache2

Train Rspamd with existing ham/spam corpora

Rspamd works without training – but training will definitely improve accuracy! For my training, I used the contents of my Inbox as a source of ham and the contents of my Junk folder as a source of spam – modify the following to suit your environment:

rspamc learn_ham /var/vmail/example.com/user/cur/
rspamc learn_spam /var/vmail/example.com/user/.Junk/cur/

You will probably get a few errors during the training process – some of the more common ones include:

  • <MessageID> contains less tokens than required for bayes classifier
    (the email was too short to classify)
  • <MessageID> has been already learned as ham, ignore it
    (means exactly what it says – the email is similar enough to one already in the database)

Acquire spam for additional training

If you’re like me, you have lots of ham for training purposes – but you’ve deleted most of your spam. It’s important to train both, so you really need some recent spam to tell Rspamd what to look for (old spam isn’t as useful, as spam changes over time).

Art Invoice make their spam corpora available to download – each day has a gzipped file filled with fresh spam ready to train with! Just one problem – it’s in mbox format, rather than Maildir format.

I found a python script to convert mbox files to Maildirs on GitHub – grab that, call it mb2md.py, make it executable and then put it in the same folder as this shell script (which needs to be executable as well):

/path/to/getspam.sh

#!/bin/bash
spamfile=spam--`date '+%Y-%m-%d'`.gz
spamfile_unpacked=spam--`date '+%Y-%m-%d'`
wget http://artinvoice.hu/spams/$spamfile
gunzip $spamfile
./mb2md.py -i $spamfile_unpacked -o spam/
rspamc learn_spam spam/cur/
rm -r $spamfile_unpacked spam/
exit 0

Run it once (as root) to test, then add it to cron for automated daily spam training. Rspamd will already detect most of the messages as spam – you’ll see errors like this:

HTTP error: 410, <MessageID> is skipped for bayes classifier: already in class spam; probability 100.00%

..but there will be enough that get learned to make this a worthwhile exercise.

Filter everyone’s spam into the Junk folder

This step is definitely optional and I’d recommend holding off on this until you’re comfortable that Rspamd is doing a good job of detecting spam.

Out of the box, Rspamd has a few too many false positives for my liking – so I decided to raise the score at which the X-Spam: Yes header gets added from 6 to 8. You can modify this setting with the Rspamd web interface, in the Configuration tab (it’s the setting called “Probably Spam”). I’d recommend adjusting this before creating the following global sieve filter..

First, edit /etc/dovecot/conf.d/90-sieve.conf and add the following line:

sieve_after = /etc/dovecot/sieve/after.d/

Create the directory mentioned above:

mkdir /etc/dovecot/sieve/after.d/

Create the sieve filter – /etc/dovecot/sieve/after.d/junk.sieve

require ["fileinto","mailbox"];
if header :contains "X-Spam" "Yes" {
 fileinto :create "Junk";
 stop;
}

Finally, compile the sieve filter, update privileges and restart Dovecot:

sievec /etc/dovecot/sieve/after.d/junk.sieve
chown -R vmail.vmail /etc/dovecot/sieve/
systemctl restart dovecot

References and notes

I couldn’t have written this guide without extensive Googling – among other sites, these were the most helpful:

I will probably tweak this guide as I learn more about Rspamd – I will note any updates I make here.

  • 2017-10-15: Added notes on how to acquire additional spam for training purposes
  • 2017-10-17: Included instructions for enabling the Dovecot imap_sieve and sieve plugins
  • 2017-10-17: Simplified worker controller configuration to use a single bind_socket
  • 2018-01-28: Updated Apache proxy configuration to remove redundant config option
  • 2018-06-05: Change privileges on the Dovecot sieve script directory

Any feedback is greatly appreciated – I hope you find this guide useful!