Postfix with SPF, DKIM and DMARC

Using the tutorials at workaround.org for setting up Postfix with virtual users and domains is generally excellent, but the tutorial for Debian Jessie (with Postfix 2.11) doesn’t contain instructions for setting up authenticated mail delivery (SPF, DKIM and DMARC). Here are my notes:

Pre-requisites

  • An otherwise-functioning email setup (these notes build on the workaround.org tutorial)
  • Understanding of (and ability to modify) DNS records

Inbound SPF

SPF has its pros and cons, but on the whole it’s pretty useful. I’d recommend configuring inbound SPF checks to only reject mail that has a “hard fail” for SPF (vs. “soft fail” or “neutral”).

Install the following:

sudo aptitude install postfix-policyd-spf-python

In /etc/postfix/main.cf, in the section shown below – add the check_policy_service line before any RBL checks:

smtpd_recipient_restrictions =
  ...
  check_policy_service unix:private/policy-spf
  ...

Other SPF tutorials recommend extending the SPF time limit to 3600 seconds (from a default of 1000 seconds) – assuming your server is on a decently quick connection, I can’t see a good reason to change the default.

In /etc/postfix/master.cf, enable the SPF service as follows:

policy-spf unix -       n       n       -       -       spawn
  user=nobody argv=/usr/bin/policyd-spf

In /etc/postfix-policyd-spf-python/policyd-spf.conf, ensure the configuration is set to only reject messages that “hard fail” the SPF check – remove the existing HELO_reject line and replace as follows:

HELO_reject = Fail

Restart postfix:

sudo systemctl restart postfix

Send yourself a test email (from Gmail or similar) to ensure you’re getting lines like this in your mail.log:

policyd-spf[22747]: None; identity=helo; client-ip=2607:f8b0:4003:c06::230; helo=mail-oi0-x230.google.com; envelope-from=user@gmail.com; receiver=user@example.com
policyd-spf[22747]: Pass; identity=mailfrom; client-ip=2607:f8b0:4003:c06::230; helo=mail-oi0-x230.google.com; envelope-from=user@gmail.com; receiver=user@example.com

Outbound SPF

Configuring outbound SPF involves working out what mail servers you could potentially send through and then crafting an appropriate DNS TXT record (note that DNS SPF records have been deprecated and shouldn’t be used).

There are plenty of decent wizards and tutorials out there, but I like the easySPF wizard. Because I run my own mail server and never, ever send through any other mail service, my record looks like this:

example.com. IN TXT "v=spf1 mx -all"

If I also wanted to use Gmail or Google Apps to send email from my domain, I’d need to change it to look like this:

example.com. IN TXT "v=spf1 mx include:_spf.google.com -all"

You can absolutely break mail delivery for your domain using SPF, so it’s really important to understand what you’re putting in to your DNS records.

DKIM

DKIM is much, much easier than it used to be – and is becoming more and more important in the fight against spam. For a more comprehensive introduction, Eric Allman from Sendmail put together a nice overview of DKIM.

Different tutorials use different packages – I install the following:

sudo aptitude install opendkim opendkim-tools libmail-dkim-perl libcrypt-openssl-random-perl libcrypt-openssl-rsa-perl

The first two packages install OpenDKIM; the remaining packages are so SpamAssassin can handle DKIM signed messages.

Make sure the following directories exist (create them if they don’t):

sudo mkdir /etc/opendkim
sudo mkdir /etc/opendkim/keys

My /etc/opendkim.conf file is as follows. You’ll note that it differs from other tutorials in two main ways:

  • I’m using virtual domains, so I’m not putting everything in the main configuration file
  • I’m not using PID files to communicate with Postfix, so I don’t have things like UMask and PidFile set
AutoRestart             yes
AutoRestartRate         10/1h

Syslog                  yes
SyslogSuccess           yes
LogWhy                  yes

Canonicalization        relaxed/simple

ExternalIgnoreList      refile:/etc/opendkim/TrustedHosts
InternalHosts           refile:/etc/opendkim/TrustedHosts
KeyTable                refile:/etc/opendkim/KeyTable
SigningTable            refile:/etc/opendkim/SigningTable

Mode                    sv
SignatureAlgorithm      rsa-sha256

UserID                  opendkim:opendkim

Socket                  inet:12345@localhost

OversignHeaders         From

Note the Socket line above – that needs to be configured elsewhere as well.

In /etc/default/opendkim, comment out everything except the following:

SOCKET="inet:12345@localhost" # listen on loopback on port 12345

In /etc/postfix/main.cf:

milter_protocol = 6
milter_default_action = accept
smtpd_milters = inet:localhost:12345
non_smtpd_milters = $smtpd_milters

If you’ve followed the workaround.org tutorial, you will already have smtpd_milters = unix:/spamass/spamass.sock – if so, the smtpd_milters line above should be changed to:

smtpd_milters = unix:/spamass/spamass.sock, inet:localhost:12345

Create the TrustedHosts, KeyTable and SigningTable files as follows (substitute your domain(s) for example.com in all of them)

/etc/opendkim/TrustedHosts:

127.0.0.1
::1
localhost

8.8.8.8
2001:4860:4860::8888/64

*.example.com

Substitute the second block of IP addresses with real addresses from your server – don’t leave the Google DNS addresses there! Note that the /64 at the end of the IPV6 address is required.

/etc/opendkim/KeyTable

Note multiple domain substitutions; DKIM also uses ‘selectors’ which are defined here. I’ve used “mail” as the selector (some folk will recommend something like “jan2016”), but you can use whatever you’d like:

mail._domainkey.example.com example.com:mail:/etc/opendkim/keys/example.com/mail.private

/etc/opendkim/SigningTable (again with the multiple domain substitutions):

*@example.com mail._domainkey.example.com

You’ll notice that the KeyTable file mentions a directory for your domain – create that now:

sudo mkdir /etc/opendkim/keys/example.com

In the directory you just created, create the private/public key pair (note another domain substitution, and if you changed the selector above from “mail” to something else, you’ll need to change it here too):

sudo opendkim-genkey -b 2048 -s mail -d example.com

Change the owner of the private key:

sudo chown opendkim:opendkim mail.private

The mail.txt file contains the TXT record you need to add to your DNS. Note that the key is broken up in to multiple quoted sections – be careful when adding it to your DNS records to make sure it’s all added correctly.

Wait until the DNS change has propagated, then confirm that the DKIM key in your DNS records passes a validation check. If everything looks good, restart the opendkim and postfix processes:

sudo systemctl restart opendkim
sudo systemctl restart postfix

Send a test email to a Gmail account. Here’s what you should see in the mail logs for the outbound message:

opendkim[11862]: D11F114939: DKIM-Signature field added (s=mail, d=example.com)

In Gmail, when the test email arrives, it should have the following headers:

...
Authentication-Results: mx.google.com;
       spf=pass (google.com: domain of user@example.com designates [IP Address] as permitted sender) smtp.mailfrom=user@example.com;
       dkim=pass header.i=@example.com;
...
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=example.com; s=mail;
...

Reply back to the email and check the mail logs for the following:

opendkim[11862]: 0EBC014939: mail-ob0-x235.google.com [2607:f8b0:4003:c01::235] not internal
opendkim[11862]: 0EBC014939: not authenticated
opendkim[11862]: 0EBC014939: DKIM verification successful
opendkim[11862]: 0EBC014939: s=20120113 d=gmail.com SSL

If you have SpamAssassin installed, you should also see something like the following:

spamd[18803]: spamd: result: . 0 - DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,UNPARSEABLE_RELAY scantime=0.4,size=2698,user=user,uid=5000,required_score=5.0,rhost=::1,raddr=::1,rport=52616,mid=<CAEbJLUEKOOGQabWPgjwXzW-_Y0PjUFH1+jo7cz50X3BOxYNf7w@mail.gmail.com>,autolearn=ham autolearn_force=no

Another test you can (and should) run is the Newsletters spam test, as it will make sure your mail is correctly signed using DKIM.

Once you are absolutely certain that both SPF and DKIM are working correctly…

Outbound DMARC

Similar to outbound SPF, outbound DMARC is set up as a DNS TXT record that defines how receiving mail servers should handle messages that fail SPF and DKIM checks. DMARC is only useful if SPF and DKIM are working for your domain, AND if all outbound mail for your domain goes through your servers.

If other servers are used (e.g., using an ISP mail server as a smarthost), then do not configure a DMARC record!

I really like Google’s guide for Preventing outgoing spam with DMARC and consider it mandatory reading.

Google’s guide will help you configure your own DMARC record, but as an example – here’s mine:

_dmarc.example.com. IN TXT "v=DMARC1; p=quarantine; pct=100; rua=mailto:postmaster@example.com"

Inbound DMARC

Install the following:

sudo aptitude install opendmarc

My /etc/opendmarc.conf – I’ve chosen to not send forensic/aggregate reports and to not reject mail that fails DMARC (for the time being):

AutoRestart             Yes
AutoRestartRate         10/1h

UserID                  opendmarc:opendmarc
Socket                  inet:54321@localhost
Syslog                  true
SyslogFacility          mail

AuthservID              mail.server.fqdn
#TrustedAuthservIDs      other.mail.server, another.mail.server
IgnoreHosts             /etc/opendkim/TrustedHosts

RejectFailures          false

Uncomment the TrustedAuthservIDs if you have additional mail server(s) configured for the same domain(s) that also have OpenDMARC installed.

Next, configure the socket by adding the following line to /etc/default/opendmarc (don’t uncomment anything):

SOCKET="inet:54321@localhost"

Add to the list of milters in /etc/postfix/main.cf:

smtpd_milters = unix:/spamass/spamass.sock, inet:localhost:12345, inet:localhost:54321

Start and restart the appropriate processes:

sudo systemctl start opendmarc
sudo systemctl restart postfix

Send yet another test email from Gmail and then look for the following in the mail logs:

opendmarc[23774]: E24C514939: gmail.com pass

You’ll also see the following in the mail header:

Authentication-Results: mail.server.fqdn; dmarc=pass header.from=gmail.com

Tomcat proxified by Apache

Notes on how to set up Tomcat with an Apache proxy in front of it – useful for adding SSL or serving other content via Apache on the same port – written up because a *lot* of the online howtos out there are either out of date or generally unhelpful. This is a quick-and-dirty setup and works well enough for a small server with not much traffic; it’s probably not the best for a larger-scale site.

Ingredients:

  • Tomcat 7.0
  • OpenJDK Runtime Environment 1.7 (aka, Java 7)
  • Apache 2.2
  • Debian Wheezy (Linux)

Assumptions:

Already functioning LAMP server .. there are stacks of good howtos out there for getting Apache up and running, so I won’t rehash it here.

Recipe:

JRE/Tomcat Install and Config

sudo aptitude install openjdk-7-jre
sudo aptitude install tomcat7 tomcat-common tomcat-admin

Once Tomcat is installed and any dependencies are satisfied, modify /etc/tomcat7/tomcat-users.xml to allow access to the Manager webapp – add the following to the bottom of the file before </tomcat-users> (obviously substituting admin and password with something a little more creative..):

<user username="admin" password="password" roles="manager-gui,admin-gui"/>

Save and then restart Tomcat:

sudo service tomcat7 restart

Add a rule in to the firewall to allow access to port 8080, then try to load http://server.name.here:8080 – the default Tomcat page (It works !) should appear – navigate to the Manager webapp link and log in with the user/pass configured earlier in tomcat-users.xml.

At this point, it would be helpful to deploy and test a Java webapp (in a .war file) using the Manager webapp if you have one handy..

If everything works up to this point, enable the AJP connector for Tomcat – in /etc/tomcat7/server.xml, find and remove the following highlighted lines:

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <!--
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
    -->

Save and then restart Tomcat:

sudo service tomcat7 restart

Apache Proxy Config

Enable the proxy and proxy_ajp modules:

sudo a2enmod proxy proxy_ajp

Configure the Apache proxy for the deployed webapp in an already-functioning VirtualHost – this can be done a number of ways, but easiest inside a Location declaration (the ProxyPassReverse line should reference http:// or https:// as appropriate):

  <Location /WebappName>
    ProxyPass ajp://server.name.here:8009/WebappName
    ProxyPassReverse https://server.name.here/WebappName
  </Location>

Restart Apache:

sudo service apache2 restart

The deployed webapp should now be available at http(s)://server.name.here/WebappName.

SABnzbd+ and Sick Beard on Ubuntu Linux

There are heaps of guides out there on the Internets for setting up SABnzbd+ and Sick Beard for automated download goodness – but most of them assume you’re setting up SAB and Sick Beard on a computer that’s being used as a regular PC, rather than a media server. As a result, the instructions out there could use some tweaking..

Recipe, pre-requisites and assumptions:

  • You’re using Ubuntu Linux
  • You have a user account to use on the media server (your own account, another account .. just not the root account)
  • You already have details for a news server

Step 1: Install SABnzbd+

Add the SABnzbd+ repository, update apt:

sudo add-apt-repository ppa:jcfp/ppa
sudo aptitude update

Install SAB and all its’ dependencies:

sudo aptitude install sabnzbdplus

Configure SAB as a service – set USER= to the user account you’ve chosen and set HOST= to 0.0.0.0 – everything else is fine to leave as is:

sudo vim /etc/default/sabnzbdplus
USER=username
HOST=0.0.0.0

Start SAB:

sudo service sabnzbdplus start

Run the configuration wizard by going to http://servername:8080/sabnzbd/ and following the prompts. You’ll need to enter the news server details; also make sure that ‘I want SABnzbd to be viewable by any pc on my network’ is selected and set a username and password (optional).

References
How to install SABnzbd on Ubuntu?
SABnzbd Manual – Ubuntu PPA Repository

Step 2: Install Sick Beard

Sick Beard installation needs git – if you don’t have it already:

sudo aptitude install git

Create the directory for installation:

sudo mkdir /var/lib/sickbeard

Pull down a copy of Sick Beard via git, then change ownership of the whole lot to the chosen user account:

sudo git clone git://github.com/midgetspy/Sick-Beard.git /var/lib/sickbeard
sudo chown -R username.groupname /var/lib/sickbeard/

Change to the Sick Beard directory, copy the init file out to where it belongs:

cd /var/lib/sickbeard
sudo cp init.ubuntu /etc/init.d/sickbeard
sudo update-rc.d sickbeard defaults

Edit the defaults file and add the following lines (changing out the username and home directory):

sudo vim /etc/default/sickbeard
SB_USER=username
SB_HOME=/var/lib/sickbeard
SB_DATA=/var/lib/sickbeard/sickbeard_data
SB_PIDFILE=/var/lib/sickbeard/pid

Start the Sick Beard service:

sudo service sickbeard start

Sick Beard should now be available at http://servername:8081/

References
Installing Sick Beard on Ubuntu Server

Step 3: Configure SABnzbd

Wherein the download directories are selected and Sick Beard configuration magic is made:

Load the SAB configuration section: http://servername:8080/sabnzbd/config/

In Folders, configure the Temporary Download Folder and Completed Download Folder items. Don’t choose directories that are the final destination for the TV – and make a note of the Completed Download Folder for later. Next, configure the Post-Processing Scripts Folder item – it should be /var/lib/sickbeard/autoProcessTV, but it depends on where Sick Beard is installed.

Scroll to the bottom, Save Changes and then Restart SABnzbd.

In General, make a note of the API Key for later.

In Categories, the magic to notify Sick Beard that the download has completed is configured – this is where the Post-Processing Scripts Folder comes in to play. Configure a new line on the table as follows:

ItemConfiguration
Categorytv
PriorityDefault
Processing+Delete
ScriptsabToSickBeard.py
Folder/PathThe Completed Download Folder from above
Groups / Indexer tagstv

Save – SABnzbd+ is now configured.

Step 4: Configure Sick Beard

Wherein the newznab services are configured and Sick Beard learns how to play with SAB . A few assumptions:

  • You have an account with a newznab service
  • The directory structure for your TV shows looks a little like this:
    /drive/tv/showname/season/episode

Load the Sick Beard configuration section: http://servername:8081/config/

In Search Settings, check Search NZBs, select SABnzbd from the dropdown and enter details for the SAB installation..:

ItemConfiguration
SABnzbd URLhttp://servername:8080/
SABnzbd UsernameUsername, if any
SABnzbd PasswordPassword, if any
SABnzbd API KeyAPI Key from above
SABnzbd Categorytv

Test SABnzbd – if everything works, Save Changes.

In Search Providers, configure a newznab service according to the documentation from the provider. None of the built-in providers seem to work particularly well; configure a Custom provider for best results.

In Post Processing, the downloaded file gets picked up and put in to the final directory structure. The base directory for the TV show is configured elsewhere; this is just the season and file name configuration. The important things to change are..:

ItemConfiguration
TV Download DirFolder from above (SAB’s Completed Download Folder)
Keep Original FilesUnchecked
Move Associated FilesUnchecked
Rename EpisodesChecked
Scan and ProcessUnchecked
Name PatternSeason 02/Show.Name.S02E03.HD.TV-RLSGROUP (from dropdown)
Multi-Episode StyleRepeat

Metadata is probably important if using XBMC, but it can be configured later.

Step 5: Test!

Wherein a new TV show (that hasn’t previously been downloaded) is added and an episode is downloaded to make sure the automation works:

Load the Sick Beard home page: http://servername:8081/home/

Select Add Shows then Add New Show. Start with a show that hasn’t been downloaded to test; add the others later. In Step 1, search for and select the show to download. In Step 2, select the directory that contains subdirectories with TV shows (/drive/tv/ from the above directory structure). Set this as the default. In Step 3, select the download quality, Save Defaults and Add Show. Wait a minute while Sick Beard gets the show details from the Internets..

The show ought to have been added with all episodes set to ‘Skipped’ – change one episode to ‘Wanted’ and within a minute it should begin downloading in SAB (and be transferred in to the appropriate directory once the download is complete). 🙂

If everything works, go back to Add Shows and then Add Existing Show. Based on the defaults set when adding a new show, Sick Beard should be able to see any other show downloaded; they can all be added to Sick Beard by stepping through the wizard to make sure the show name and directory have been correctly guessed. Enjoy! 😀