Padlock with Chain Header

Let’s Encrypt wildcard certificates with Hurricane Electric DNS

Renewing Let’s Encrypt wildcard certificates is generally a massive pain. You need to be able to automatically update DNS records for the domain – which is fine if you use a DNS provider that has an official Let’s Encrypt DNS plugin, but less so if you use a DNS provider that doesn’t – such as Hurricane Electric.

Side note – I’m not particularly interested in arguments for and against wildcard certs. If you’re reading this, you’ve obviously come to the conclusion that they’re probably fine and you just want them automated like the rest of your Let’s Encrypt certs!

This post assumes the following:

  • You are obtaining wildcard certificates from Let’s Encrypt
  • Your DNS is hosted with Hurricane Electric
  • Your ACME client for Let’s Encrypt is Certbot
  • You have shell access to your server

Obtain a new Let’s Encrypt wildcard certificate

Note: if you’ve already got a wildcard certificate, you can mostly skip this bit – but skim this section to make sure you’ve done everything you need to do!

1. Request wildcard certificate

Here’s the command-line incantation to request a new wildcard certificate:

sudo certbot certonly --cert-name example.com-wildcard -d '*.example.com' --manual --preferred-challenges dns 

Couple things to note here:

  • I’m not including the base domain in the certificate here – I wouldn’t be able to automate it if I did, as I’d end up needing two TXT records for the same hostname. My solution was to separate the base domain out into its own certificate, which works perfectly for me.
  • I’m specifying a certificate name. This is important, as otherwise it ends up trying to name it the same as the base domain – which is no good if you have a cert for the base domain as well.

Run the Certbot wizard – it will soon ask you to create a TXT record!

2. Create TXT record

The Certbot wizard will ask you to create a record, something like the following:

Please deploy a DNS TXT record under the name
_acme-challenge.example.com with the following value:

qwertyuiop-1234567890

Log into the Hurricane Electric DNS console, select your domain and create a new TXT record with the following settings:

Name_acme-challenge.example.com
Text dataqwertyuiop-1234567890
TTL (Time to live)5 minutes (300)

Don’t check the Enable entry for dynamic dns box yet! Save the record and complete the Certbot wizard. Your certificate should now be issued, and you can configure your Apache / Nginx / etc server as appropriate.

Set up renewal scripts

1. Create DDNS TXT record key

As of current writing, the various Hurricane Electric DNS plugins for Certbot that I’ve seen all log into the actual account – which is horrendous from a security perspective. You don’t want a script to have full control over all of your domain records!

Thankfully, Hurricane Electric now allow TXT records to be updated with a key that only has control over just that one record.

Go back to the _acme-challenge TXT record for your domain, check the box to enable the entry for dynamic dns and Update. This enables the DDNS feature – you should now see an “arrow circle” symbol for that record:

Hurricane Electric DDNS record

Click the arrow circle symbol to generate a new DDNS key (save this key somewhere – this is the last time you’ll see it in the Hurricane Electric interface!)

For the purposes of this example, I’ll assume that the generated key looks something like ‘qwertyuiop123456’.

2. Add manual authentication script

Copy and paste the following into /etc/letsencrypt/he-dns-update.sh:

#!/bin/bash

# Do we have everything we need?
if [[ -z "$CERTBOT_DOMAIN" ]] || [[ -z "$CERTBOT_VALIDATION" ]]; then
    echo '$CERTBOT_DOMAIN and $CERTBOT_VALIDATION environment variables required.'
    exit 1
fi

# Add all HE TXT record DDNS keys to the txt_key object
# Remember to protect this script file - chmod 700!
declare -A txt_key
txt_key['_acme-challenge.example.com']='qwertyuiop123456'

# Create a FQDN based on $CERTBOT_DOMAIN
HE_DOMAIN="_acme-challenge.$CERTBOT_DOMAIN"

# Update HE DNS record
curl -s -X POST "https://dyn.dns.he.net/nic/update" -d "hostname=$HE_DOMAIN" -d "password=${txt_key[$HE_DOMAIN]}" -d "txt=$CERTBOT_VALIDATION"

# Sleep to make sure the change has time to propagate over to DNS
sleep 30

As the comment suggests – protect this file from prying eyes by using chmod 700. If you have multiple wildcard certificates, you can add in extra entries to the txt_key object.

3. Add post deployment script

As much as you can manually restart services after the new certificate has been issued, you can automate that as well. Copy and paste the following into /etc/letsencrypt/renewal-hooks/deploy/restart-services.sh:

#!/bin/sh

set -e

for domain in $RENEWED_DOMAINS; do
  case $domain in
    *.example.com)
      systemctl restart apache2
      ;;
    *.example.com.au)
      systemctl restart nginx
      systemctl restart postfix
      ;;
    *)
  esac
done

Update the script to restart specific services for particular domains as required. I recommend chmod’ing this file to 755 – unlike the previous script, there’s nothing sensitive here!

Request Let’s Encrypt wildcard renewal

We’re going to force a renewal of the certificate here – this will test the two scripts above, plus update the renewal config so that the certificate will automatically renew in the future. If your certificate is due for renewal already, you don’t need to include the –force-renewal flag. Run the following command:

sudo certbot renew --cert-name example.com-wildcard --manual --manual-auth-hook /etc/letsencrypt/he-dns-update.sh --preferred-challenges dns --force-renewal

Check that:

  • There are no errors in the Certbot logs,
  • The certificate renewed successfully, and
  • All services restarted appropriately

If everything worked – your Let’s Encrypt wildcard certificates should now renew automagically!

UniFi Controller 5.11, Let’s Encrypt SSL and Docker

A slight change of plans from earlier posts on the topic of UniFi Controllers! Here’s how to get a UniFi Controller running inside a Docker container, along with a trusted Let’s Encrypt SSL certificate.

Note: this guide assumes you’re configuring things on a server or VM with public Internet access. You’ll also need a fixed public IP and functional DNS to get an SSL certificate.

Here we go:

Firewall

UniFi needs a bunch of inbound ports open. Here’s the official list – it differs slightly to what I use:

PortDescription
UDP/3478STUN – required for device communication with the controller
TCP/8080Inform – required to adopt devices
TCP/8443GUI – required even if you use the Cloud Controller access
TCP/8880Captive Portal – HTTP – only needed if you use the captive portal feature
TCP/8843Captive Portal – HTTPS – only needed if you use the captive portal feature
TCP/6789Speed Test – only needed if you use the speed test feature

Let’s Encrypt also needs a port open:

PortDescription
TCP/80HTTP – required for the HTTP-01 challenge type

I use ufw to configure iptables – first, set up an application definition for the UniFi Controller – in /etc/ufw/applications.d/unifi:

[unifi]
title=unifi
description=UniFi Controller
ports=6789,8080,8880,8443,8843/tcp|3478/udp

Run the following four commands to configure and enable the firewall. I’ve made some assumptions about what’s needed – you may need to customise things a little more:

sudo ufw allow ssh
sudo ufw allow http
sudo ufw allow unifi
sudo ufw enable

User Account

UniFi probably shouldn’t be run as root – this is generally a good idea, plus it may also become a requirement for the Docker image I’m using in the future. This will also affect what ports you can configure the controller to use – the default ports work fine for any user, but changing any of the ports to <1024 requires root.

Create the unifi user and group accounts:

sudo adduser unifi --system --group --no-create-home

Pay attention to the UID and GID that get created; you need them in the Docker Compose file below.

Docker

Here’s the tl;dr version of the installation instructions, but if you want to read the full version with all the details – check the Docker website.

Configure the Docker repository – it contains a more up-to-date version:

sudo apt-get update && sudo apt-get upgrade
sudo apt-get install apt-transport-https ca-certificates curl gnupg2 software-properties-common
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"

Install Docker and related tools:

sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io

UniFi Controller

There are a number of UniFi Docker images out there, but I like the one by jacobalberty as it’s kept up to date – plus it exposes a volume for adding trusted certificates. His Docker Compose file isn’t quite to my taste, so I’ve adjusted things. Create the file /opt/unifi/docker-compose.yml:

version: '2.2'
services:
  mongo:
    image: 'mongo:3.4'
    restart: always
    volumes:
      - db:/data/db
  controller:
    image: 'jacobalberty/unifi:${TAG:-latest}'
    depends_on:
      - mongo
    init: true
    restart: always
    volumes:
      - data:/unifi/data
      - log:/unifi/log
      - cert:/unifi/cert
      - init:/unifi/init.d
    environment:
      RUNAS_UID0: 'false'
      UNIFI_UID: 100
      UNIFI_GID: 100
      JVM_MAX_THREAD_STACK_SIZE: 1280k
      DB_URI: mongodb://mongo/unifi
      STATDB_URI: mongodb://mongo/unifi_stat
      DB_NAME: unifi
    ports:
      - '3478:3478/udp'
      - '6789:6789/tcp'
      - '8080:8080/tcp'
      - '8443:8443/tcp'
      - '8880:8880/tcp'
      - '8843:8843/tcp'
  logs:
    image: bash
    depends_on:
      - controller
    command: bash -c 'tail -F /unifi/log/*.log'
    restart: always
    volumes:
      - log:/unifi/log

volumes:
  db:
  data:
  log:
  cert:
  init:

Note: if you’re going to change the location of this file, it should be in a directory called ‘unifi’. Bring the stack up like so (it will take a fair while first time around):

sudo docker-compose up -d

Install SSL

This part requires a few sections that need to be completed in order – first you need a script to load the SSL certificate into the UniFi Docker cert volume, then you need to run a certbot command to obtain the certificate.

If you use a provider other than Let’s Encrypt for SSL certificates, these instructions will need to be adjusted.

UniFi SSL Deploy Script

It may seem backwards, but the deploy script needs to exist before obtaining the certificate. Read through this script carefully and adjust any domains and directories as needed. Create the file /opt/unifi/unifi-ssl-deploy.sh:

#!/bin/sh

set -e

for domain in $RENEWED_DOMAINS; do
  case $domain in
  unifi.example.com)
    # Where does the Docker cert data volume live?
    cert_root=/var/lib/docker/volumes/unifi_cert/_data
    # Where is the Docker Compose file?
    compose_file=/opt/unifi/docker-compose.yml

    # Make sure the certificate and private key files are
    # never world readable, even just for an instant while
    # we're copying them into cert_root.
    umask 077

    cp "$RENEWED_LINEAGE/cert.pem" "$cert_root/cert.pem"
    cp "$RENEWED_LINEAGE/privkey.pem" "$cert_root/privkey.pem"
    cp "$RENEWED_LINEAGE/chain.pem" "$cert_root/chain.pem"

    # Apply the proper file permissions
    # Files can be owned by root
    chmod 400 "$cert_root/cert.pem" \
      "$cert_root/privkey.pem" \
      "$cert_root/chain.pem"

    # Restart the Docker container
    docker-compose -p unifi -f $compose_file stop
    docker-compose -p unifi -f $compose_file start
    ;;
  esac
done

Now make the file executable:

sudo chmod a+x unifi-ssl-deploy.sh

Obtain SSL with Certbot

Conveniently, Certbot has its own mechanism for obtaining an SSL certificate without using a webserver. If you have a webserver configured, you will want to adjust these instructions accordingly.

As above, adjust the following to suit your domain:

sudo apt-get install certbot
sudo certbot certonly --standalone --domain unifi.example.com --deploy-hook /opt/unifi/unifi-ssl-deploy.sh

The command to obtain the certificate will ask a few questions – you may also see an error from the deploy script, but it’s not actually an error per se.

Note: After the deploy script has run, you need to wait up to 5 minutes for the UniFi Controller to fully start back up again. If you don’t, you’re likely to get an SSL error (PR_END_OF_FILE_ERROR) in the browser!

We’re all done – your UniFi Controller should now be available via: https://unifi.example.com:8443

Reverse Proxy

I’ve opted to not configure a reverse proxy, as I don’t believe one is needed. If port 8443 is blocked on your network, you can configure cloud access via https://unifi.ui.com.

If you want to configure a reverse proxy, note you’ll need something that handles websockets gracefully – Nginx and Traefik are probably your best options.

Exchange Server 2007 – troubleshooting SSL

Some points to troubleshoot an Exchange Server 2007 SSL configuration. For god knows what reason, Microsoft decided that everything needed to be command line, so it’s pretty easy to get stuck.

Starting point – find what certificates are installed

Run the following cmdlet from the Exchange Shell:

Get-ExchangeCertificate

You should see output similar to:

Thumbprint                Services Subject
----------                -------- -------
1B5667CCB803BC4AD13E7E51A .IP.W    CN=mail.example.com....
103F3F32814A48D2416ECC5DB S....    CN=exch-07
43C6A1548782A25ABA425B471 ....W    CN=exch-07.example....

The Thumbprint is the identifier used in other cmdlets when referring to a specific certificate. The Services are what the certificate is enabled for; each letter indicates what service(s) are configured:

LetterService
SSMTP – outbound e-mail secured with TLS
IIMAP – inbound e-mail
PPOP3 – inbound e-mail
UUnified Messaging – I believe this is for Outlook Anywhere
WOutlook Web Access / IIS – webmail

You can also get more detailed information about a specific certificate with the following command:

Get-ExchangeCertificate [thumbprint] | fl

Are the certificates enabled for the right things? Are multiple certificates enabled for the same service (as per the example above)? Is the right certificate installed at all? Has the certificate expired? Does the certificate have incorrect or misspelt details in the DN? Etc..

Handy commands:

Turn a specific service on (Outlook Web Access in this example):

Enable-ExchangeCertificate -Thumbprint [thumbprint] -Services IIS

Disable a certificate:

Enable-ExchangeCertificate -Thumbprint [thumbprint] -Services None

Remove a certificate:
This command does what it says on the tin – there is no undo!

Remove-ExchangeCertificate -Thumbprint [thumbprint]

Aside from using the above to fix obvious problems, the Event Viewer contains very useful error codes and explanatory messages in well-formed English (which is just about a first for any Microsoft product, I think). Combination that + Google will provide fixes for most problems.

One thing I’ve seen once or twice is a certificate that the customer swears black-and-blue has been installed and it’s just not showing up in the Get-ExchangeCertificate output. If you look in (the Certificates snap-in in) MMC, it’s there. What’s happened? The customer requested the certificate in Exchange, but imported the certificate response in to MMC directly. Ergo, public and private keys not matched up and certificate not available to Exchange. Delete certificate from MMC, import in to Exchange instead.