SpamAssassin sa-learn cron script for virtual users

zac 2 Comments

Because I couldn’t find something to fully automate the SpamAssassin sa-learn process for virtual email users in a MySQL database, I wrote my own (borrowing liberally from Jason Schaefer’s SpamAssassin training and spam cleanup script).

This script assumes that:

  • You have a MySQL database with virtual users, with a user table called ‘virtual_users’ and the full email address stored in a field called ’email’.
  • Your email is stored in Maildir folders, with a heirarchy starting from /var/vmail/domain.com/username/…
#!/bin/bash

## Database details

USER=''
PASS=''
HOST=''
DB=''

## Where to log stuff

LOG='/var/log/sa-learn.log'

## How many days to wait before deleting spam
## Comment out to disable

CLEAN=30

echo -e "\n\nRun started `date +%c`"  >> $LOG 2>&1

## Spam and ham training for all virtual users
## Delete spam older than $CLEAN days

mysql --skip-column-names -u$USER -p$PASS -h$HOST -D$DB -e "SELECT SUBSTRING(email, 1, LOCATE('@', email) - 1) AS user, SUBSTRING(email, LOCATE('@', email) + 1) AS domain FROM virtual_users" | while read user domain;

do
  ## Spam
  echo "Spam training for $user@$domain" >> $LOG 2>&1
  /usr/bin/sa-learn --no-sync --spam /var/vmail/$domain/$user/.Junk/{cur,new} >> $LOG 2>&1
  ## Ham
  echo "Ham training for $user@$domain" >> $LOG 2>&1
  /usr/bin/sa-learn --no-sync --ham /var/vmail/$domain/$user/{cur} >> $LOG 2>&1
  ## Delete
  if [ -n $CLEAN ]; then
    echo "Deleting spam for $user@$domain older than $CLEAN days" >> $LOG 2>&1
    find /var/vmail/$domain/$user/.Junk/cur/ -type f -mtime +$CLEAN -exec rm {} \;
  fi
done

## Sync the SpamAssassin journal and print out stats

echo "Syncing the SpamAssassin journal" >> $LOG 2>&1
/usr/bin/sa-learn --sync >> $LOG 2>&1
echo "Statistics for this run:" >> $LOG 2>&1
/usr/bin/sa-learn --dump magic >> $LOG 2>&1

echo -e "Run finished `date +%c`"  >> $LOG 2>&1

exit

Postfix with SPF, DKIM and DMARC

zac 1 Comment

Using the tutorials at workaround.org for setting up Postfix with virtual users and domains is generally excellent, but the tutorial for Debian Jessie (with Postfix 2.11) doesn’t contain instructions for setting up authenticated mail delivery (SPF, DKIM and DMARC). Here are my notes:

Pre-requisites

  • An otherwise-functioning email setup (these notes build on the workaround.org tutorial)
  • Understanding of (and ability to modify) DNS records

Inbound SPF

SPF has its pros and cons, but on the whole it’s pretty useful. I’d recommend configuring inbound SPF checks to only reject mail that has a “hard fail” for SPF (vs. “soft fail” or “neutral”).

Install the following:

sudo aptitude install postfix-policyd-spf-python

In /etc/postfix/main.cf, in the section shown below – add the check_policy_service line before any RBL checks:

smtpd_recipient_restrictions =
  ...
  check_policy_service unix:private/policy-spf
  ...

Other SPF tutorials recommend extending the SPF time limit to 3600 seconds (from a default of 1000 seconds) – assuming your server is on a decently quick connection, I can’t see a good reason to change the default.

In /etc/postfix/master.cf, enable the SPF service as follows:

policy-spf unix -       n       n       -       -       spawn
  user=nobody argv=/usr/bin/policyd-spf

In /etc/postfix-policyd-spf-python/policyd-spf.conf, ensure the configuration is set to only reject messages that “hard fail” the SPF check – remove the existing HELO_reject line and replace as follows:

HELO_reject = Fail

Restart postfix:

sudo systemctl restart postfix

Send yourself a test email (from Gmail or similar) to ensure you’re getting lines like this in your mail.log:

policyd-spf[22747]: None; identity=helo; client-ip=2607:f8b0:4003:c06::230; helo=mail-oi0-x230.google.com; envelope-from=user@gmail.com; receiver=user@example.com
policyd-spf[22747]: Pass; identity=mailfrom; client-ip=2607:f8b0:4003:c06::230; helo=mail-oi0-x230.google.com; envelope-from=user@gmail.com; receiver=user@example.com

Outbound SPF

Configuring outbound SPF involves working out what mail servers you could potentially send through and then crafting an appropriate DNS TXT record (note that DNS SPF records have been deprecated and shouldn’t be used).

There are plenty of decent wizards and tutorials out there, but I like the easySPF wizard. Because I run my own mail server and never, ever send through any other mail service, my record looks like this:

example.com. IN TXT "v=spf1 mx -all"

If I also wanted to use Gmail or Google Apps to send email from my domain, I’d need to change it to look like this:

example.com. IN TXT "v=spf1 mx include:_spf.google.com -all"

You can absolutely break mail delivery for your domain using SPF, so it’s really important to understand what you’re putting in to your DNS records.

DKIM

DKIM is much, much easier than it used to be – and is becoming more and more important in the fight against spam. For a more comprehensive introduction, Eric Allman from Sendmail put together a nice overview of DKIM.

Different tutorials use different packages – I install the following:

sudo aptitude install opendkim opendkim-tools libmail-dkim-perl libcrypt-openssl-random-perl libcrypt-openssl-rsa-perl

The first two packages install OpenDKIM; the remaining packages are so SpamAssassin can handle DKIM signed messages.

Make sure the following directories exist (create them if they don’t):

sudo mkdir /etc/opendkim
sudo mkdir /etc/opendkim/keys

My /etc/opendkim.conf file is as follows. You’ll note that it differs from other tutorials in two main ways:

  • I’m using virtual domains, so I’m not putting everything in the main configuration file
  • I’m not using PID files to communicate with Postfix, so I don’t have things like UMask and PidFile set
AutoRestart             yes
AutoRestartRate         10/1h

Syslog                  yes
SyslogSuccess           yes
LogWhy                  yes

Canonicalization        relaxed/simple

ExternalIgnoreList      refile:/etc/opendkim/TrustedHosts
InternalHosts           refile:/etc/opendkim/TrustedHosts
KeyTable                refile:/etc/opendkim/KeyTable
SigningTable            refile:/etc/opendkim/SigningTable

Mode                    sv
SignatureAlgorithm      rsa-sha256

UserID                  opendkim:opendkim

Socket                  inet:12345@localhost

OversignHeaders         From

Note the Socket line above – that needs to be configured elsewhere as well.

In /etc/default/opendkim, comment out everything except the following:

SOCKET="inet:12345@localhost" # listen on loopback on port 12345

In /etc/postfix/main.cf:

milter_protocol = 6
milter_default_action = accept
smtpd_milters = inet:localhost:12345
non_smtpd_milters = $smtpd_milters

If you’ve followed the workaround.org tutorial, you will already have smtpd_milters = unix:/spamass/spamass.sock – if so, the smtpd_milters line above should be changed to:

smtpd_milters = unix:/spamass/spamass.sock, inet:localhost:12345

Create the TrustedHosts, KeyTable and SigningTable files as follows (substitute your domain(s) for example.com in all of them)

/etc/opendkim/TrustedHosts:

127.0.0.1
::1
localhost

8.8.8.8
2001:4860:4860::8888/64

*.example.com

Substitute the second block of IP addresses with real addresses from your server – don’t leave the Google DNS addresses there! Note that the /64 at the end of the IPV6 address is required.

/etc/opendkim/KeyTable

Note multiple domain substitutions; DKIM also uses ‘selectors’ which are defined here. I’ve used “mail” as the selector (some folk will recommend something like “jan2016”), but you can use whatever you’d like:

mail._domainkey.example.com example.com:mail:/etc/opendkim/keys/example.com/mail.private

/etc/opendkim/SigningTable (again with the multiple domain substitutions):

*@example.com mail._domainkey.example.com

You’ll notice that the KeyTable file mentions a directory for your domain – create that now:

sudo mkdir /etc/opendkim/keys/example.com

In the directory you just created, create the private/public key pair (note another domain substitution, and if you changed the selector above from “mail” to something else, you’ll need to change it here too):

sudo opendkim-genkey -b 2048 -s mail -d example.com

Change the owner of the private key:

sudo chown opendkim:opendkim mail.private

The mail.txt file contains the TXT record you need to add to your DNS. Note that the key is broken up in to multiple quoted sections – be careful when adding it to your DNS records to make sure it’s all added correctly.

Wait until the DNS change has propagated, then confirm that the DKIM key in your DNS records passes a validation check. If everything looks good, restart the opendkim and postfix processes:

sudo systemctl restart opendkim
sudo systemctl restart postfix

Send a test email to a Gmail account. Here’s what you should see in the mail logs for the outbound message:

opendkim[11862]: D11F114939: DKIM-Signature field added (s=mail, d=example.com)

In Gmail, when the test email arrives, it should have the following headers:

...
Authentication-Results: mx.google.com;
       spf=pass (google.com: domain of user@example.com designates [IP Address] as permitted sender) smtp.mailfrom=user@example.com;
       dkim=pass header.i=@example.com;
...
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=example.com; s=mail;
...

Reply back to the email and check the mail logs for the following:

opendkim[11862]: 0EBC014939: mail-ob0-x235.google.com [2607:f8b0:4003:c01::235] not internal
opendkim[11862]: 0EBC014939: not authenticated
opendkim[11862]: 0EBC014939: DKIM verification successful
opendkim[11862]: 0EBC014939: s=20120113 d=gmail.com SSL

If you have SpamAssassin installed, you should also see something like the following:

spamd[18803]: spamd: result: . 0 - DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,UNPARSEABLE_RELAY scantime=0.4,size=2698,user=user,uid=5000,required_score=5.0,rhost=::1,raddr=::1,rport=52616,mid=<CAEbJLUEKOOGQabWPgjwXzW-_Y0PjUFH1+jo7cz50X3BOxYNf7w@mail.gmail.com>,autolearn=ham autolearn_force=no

Another test you can (and should) run is the Newsletters spam test, as it will make sure your mail is correctly signed using DKIM.

Once you are absolutely certain that both SPF and DKIM are working correctly…

Outbound DMARC

Similar to outbound SPF, outbound DMARC is set up as a DNS TXT record that defines how receiving mail servers should handle messages that fail SPF and DKIM checks. DMARC is only useful if SPF and DKIM are working for your domain, AND if all outbound mail for your domain goes through your servers.

If other servers are used (e.g., using an ISP mail server as a smarthost), then do not configure a DMARC record!

I really like Google’s guide for Preventing outgoing spam with DMARC and consider it mandatory reading.

Google’s guide will help you configure your own DMARC record, but as an example – here’s mine:

_dmarc.example.com. IN TXT "v=DMARC1; p=quarantine; pct=100; rua=mailto:postmaster@example.com"

Inbound DMARC

Install the following:

sudo aptitude install opendmarc

My /etc/opendmarc.conf – I’ve chosen to not send forensic/aggregate reports and to not reject mail that fails DMARC (for the time being):

AutoRestart             Yes
AutoRestartRate         10/1h

UserID                  opendmarc:opendmarc
Socket                  inet:54321@localhost
Syslog                  true
SyslogFacility          mail

AuthservID              mail.server.fqdn
#TrustedAuthservIDs      other.mail.server, another.mail.server
IgnoreHosts             /etc/opendkim/TrustedHosts

RejectFailures          false

Uncomment the TrustedAuthservIDs if you have additional mail server(s) configured for the same domain(s) that also have OpenDMARC installed.

Next, configure the socket by adding the following line to /etc/default/opendmarc (don’t uncomment anything):

SOCKET="inet:54321@localhost"

Add to the list of milters in /etc/postfix/main.cf:

smtpd_milters = unix:/spamass/spamass.sock, inet:localhost:12345, inet:localhost:54321

Start and restart the appropriate processes:

sudo systemctl start opendmarc
sudo systemctl restart postfix

Send yet another test email from Gmail and then look for the following in the mail logs:

opendmarc[23774]: E24C514939: gmail.com pass

You’ll also see the following in the mail header:

Authentication-Results: mail.server.fqdn; dmarc=pass header.from=gmail.com

Tomcat proxified by Apache

zac

Notes on how to set up Tomcat with an Apache proxy in front of it – useful for adding SSL or serving other content via Apache on the same port – written up because a *lot* of the online howtos out there are either out of date or generally unhelpful. This is a quick-and-dirty setup and works well enough for a small server with not much traffic; it’s probably not the best for a larger-scale site.

Ingredients:

  • Tomcat 7.0
  • OpenJDK Runtime Environment 1.7 (aka, Java 7)
  • Apache 2.2
  • Debian Wheezy (Linux)

Assumptions:

Already functioning LAMP server .. there are stacks of good howtos out there for getting Apache up and running, so I won’t rehash it here.

Recipe:

JRE/Tomcat Install and Config

sudo aptitude install openjdk-7-jre
sudo aptitude install tomcat7 tomcat-common tomcat-admin

Once Tomcat is installed and any dependencies are satisfied, modify /etc/tomcat7/tomcat-users.xml to allow access to the Manager webapp – add the following to the bottom of the file before </tomcat-users> (obviously substituting admin and password with something a little more creative..):

<user username="admin" password="password" roles="manager-gui,admin-gui"/>

Save and then restart Tomcat:

sudo service tomcat7 restart

Add a rule in to the firewall to allow access to port 8080, then try to load http://server.name.here:8080 – the default Tomcat page (It works !) should appear – navigate to the Manager webapp link and log in with the user/pass configured earlier in tomcat-users.xml.

At this point, it would be helpful to deploy and test a Java webapp (in a .war file) using the Manager webapp if you have one handy..

If everything works up to this point, enable the AJP connector for Tomcat – in /etc/tomcat7/server.xml, find and remove the following highlighted lines:

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <!--
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
    -->

Save and then restart Tomcat:

sudo service tomcat7 restart

Apache Proxy Config

Enable the proxy and proxy_ajp modules:

sudo a2enmod proxy proxy_ajp

Configure the Apache proxy for the deployed webapp in an already-functioning VirtualHost – this can be done a number of ways, but easiest inside a Location declaration (the ProxyPassReverse line should reference http:// or https:// as appropriate):

  <Location /WebappName>
    ProxyPass ajp://server.name.here:8009/WebappName
    ProxyPassReverse https://server.name.here/WebappName
  </Location>

Restart Apache:

sudo service apache2 restart

The deployed webapp should now be available at http(s)://server.name.here/WebappName.