Padlock with Chain Header

Let’s Encrypt wildcard certificates with Hurricane Electric DNS

Renewing Let’s Encrypt wildcard certificates is generally a massive pain. You need to be able to automatically update DNS records for the domain – which is fine if you use a DNS provider that has an official Let’s Encrypt DNS plugin, but less so if you use a DNS provider that doesn’t – such as Hurricane Electric.

Side note – I’m not particularly interested in arguments for and against wildcard certs. If you’re reading this, you’ve obviously come to the conclusion that they’re probably fine and you just want them automated like the rest of your Let’s Encrypt certs!

This post assumes the following:

  • You are obtaining wildcard certificates from Let’s Encrypt
  • Your DNS is hosted with Hurricane Electric
  • Your ACME client for Let’s Encrypt is Certbot
  • You have shell access to your server

Obtain a new Let’s Encrypt wildcard certificate

Note: if you’ve already got a wildcard certificate, you can mostly skip this bit – but skim this section to make sure you’ve done everything you need to do!

1. Request wildcard certificate

Here’s the command-line incantation to request a new wildcard certificate:

sudo certbot certonly --cert-name example.com-wildcard -d '*.example.com' --manual --preferred-challenges dns 

Couple things to note here:

  • I’m not including the base domain in the certificate here – I wouldn’t be able to automate it if I did, as I’d end up needing two TXT records for the same hostname. My solution was to separate the base domain out into its own certificate, which works perfectly for me.
  • I’m specifying a certificate name. This is important, as otherwise it ends up trying to name it the same as the base domain – which is no good if you have a cert for the base domain as well.

Run the Certbot wizard – it will soon ask you to create a TXT record!

2. Create TXT record

The Certbot wizard will ask you to create a record, something like the following:

Please deploy a DNS TXT record under the name
_acme-challenge.example.com with the following value:

qwertyuiop-1234567890

Log into the Hurricane Electric DNS console, select your domain and create a new TXT record with the following settings:

Name_acme-challenge.example.com
Text dataqwertyuiop-1234567890
TTL (Time to live)5 minutes (300)

Don’t check the Enable entry for dynamic dns box yet! Save the record and complete the Certbot wizard. Your certificate should now be issued, and you can configure your Apache / Nginx / etc server as appropriate.

Set up renewal scripts

1. Create DDNS TXT record key

As of current writing, the various Hurricane Electric DNS plugins for Certbot that I’ve seen all log into the actual account – which is horrendous from a security perspective. You don’t want a script to have full control over all of your domain records!

Thankfully, Hurricane Electric now allow TXT records to be updated with a key that only has control over just that one record.

Go back to the _acme-challenge TXT record for your domain, check the box to enable the entry for dynamic dns and Update. This enables the DDNS feature – you should now see an “arrow circle” symbol for that record:

Hurricane Electric DDNS record

Click the arrow circle symbol to generate a new DDNS key (save this key somewhere – this is the last time you’ll see it in the Hurricane Electric interface!)

For the purposes of this example, I’ll assume that the generated key looks something like ‘qwertyuiop123456’.

2. Add manual authentication script

Copy and paste the following into /etc/letsencrypt/he-dns-update.sh:

#!/bin/bash

# Do we have everything we need?
if [[ -z "$CERTBOT_DOMAIN" ]] || [[ -z "$CERTBOT_VALIDATION" ]]; then
    echo '$CERTBOT_DOMAIN and $CERTBOT_VALIDATION environment variables required.'
    exit 1
fi

# Add all HE TXT record DDNS keys to the txt_key object
# Remember to protect this script file - chmod 700!
declare -A txt_key
txt_key['_acme-challenge.example.com']='qwertyuiop123456'

# Create a FQDN based on $CERTBOT_DOMAIN
HE_DOMAIN="_acme-challenge.$CERTBOT_DOMAIN"

# Update HE DNS record
curl -s -X POST "https://dyn.dns.he.net/nic/update" -d "hostname=$HE_DOMAIN" -d "password=${txt_key[$HE_DOMAIN]}" -d "txt=$CERTBOT_VALIDATION"

# Sleep to make sure the change has time to propagate over to DNS
sleep 30

As the comment suggests – protect this file from prying eyes by using chmod 700. If you have multiple wildcard certificates, you can add in extra entries to the txt_key object.

3. Add post deployment script

As much as you can manually restart services after the new certificate has been issued, you can automate that as well. Copy and paste the following into /etc/letsencrypt/renewal-hooks/deploy/restart-services.sh:

#!/bin/sh

set -e

for domain in $RENEWED_DOMAINS; do
  case $domain in
    *.example.com)
      systemctl restart apache2
      ;;
    *.example.com.au)
      systemctl restart nginx
      systemctl restart postfix
      ;;
    *)
  esac
done

Update the script to restart specific services for particular domains as required. I recommend chmod’ing this file to 755 – unlike the previous script, there’s nothing sensitive here!

Request Let’s Encrypt wildcard renewal

We’re going to force a renewal of the certificate here – this will test the two scripts above, plus update the renewal config so that the certificate will automatically renew in the future. If your certificate is due for renewal already, you don’t need to include the –force-renewal flag. Run the following command:

sudo certbot renew --cert-name example.com-wildcard --manual --manual-auth-hook /etc/letsencrypt/he-dns-update.sh --preferred-challenges dns --force-renewal

Check that:

  • There are no errors in the Certbot logs,
  • The certificate renewed successfully, and
  • All services restarted appropriately

If everything worked – your Let’s Encrypt wildcard certificates should now renew automagically!

Fibre Optics Header

Access your modem’s status page behind a UniFi USG router

With a full UniFi network setup (a UniFi USG router, switches, access points and so on), the UniFi router is separate from any modem needed to connect to the Internet. Your setup might look a little like this:

Because your computer isn’t directly connected to the modem, the modem is effectively “hidden” by the UniFi USG. You can still get online, but you can’t see your connection statistics and logs.

Here’s how to fix it:

Pre-requisites

First thing’s first, you need to know the IP address of your modem, and the network range for the UniFi network. The IP address of the modem must also be different to the UniFi network range. In my case, the modem IP address is 192.168.0.1 and the UniFi network range is 192.168.1.0/24.

So the instructions below will work, enter your modem IP address, the network mask in CIDR notation (without the /), enter an unused firewall rule index (the example is probably fine for a USG, but may not be for other Ubiquiti devices), and finally choose the port your modem is connected to.

Once complete, click Update:

Configuration

The remaining steps assume you have a UniFi Network Controller online somewhere, and in that controller you have Advanced Features turned on.

Log into the controller, navigate to the site where your UniFi USG is configured and go to Settings. At the bottom of the screen is the Device Authentication section – copy the SSH authentication details, then SSH into the IP address of the UniFi USG.

Enter the following commands, one line at a time:

configure
set interfaces pseudo-ethernet pREPLACEME4 link REPLACEME4
set interfaces pseudo-ethernet pREPLACEME4 address REPLACEME1/REPLACEME2
set interfaces pseudo-ethernet pREPLACEME4 description "Access to modem"
set service nat rule REPLACEME3 type masquerade
set service nat rule REPLACEME3 destination address REPLACEME0
set service nat rule REPLACEME3 outbound-interface pREPLACEME4
commit
save
exit

You should now be able to access your modem’s status page at http://REPLACEME0/.

Assuming that worked, you need to make the changes permanent with a configuration file for your UniFi Network Controller. Save the following into a text file called config.gateway.json:

{
    "interfaces": {
        "pseudo-ethernet": {
            "pREPLACEME4": {
                "address": ["REPLACEME1/REPLACEME2"],
                "description": "Access to modem",
                "link": ["REPLACEME4"]
            }
        }
    },
    "service": {
        "nat": {
            "rule": {
                "REPLACEME3": {
                    "destination": {
                        "address": ["REPLACEME0"]
                    },
                    "outbound-interface": ["pREPLACEME4"],
                    "type": "masquerade"
                }
            }
        }
    }
}

Upload this file into the UniFi Network Controller. The directory you need to upload to will depend on which site you’re configuring; the UI support article for this topic is quite detailed and I’d recommend reading it.

TL;DR: I uploaded mine to /unifi/data/sites/default/ – but read the support article for yourself; don’t assume this is the correct directory for your Network Controller!

Extra: NBN modems

For readers from Australia – there are a few things to know about the modems (or NCD / NTDs) supplied by NBNCo. Some you can access, some you can’t:

NBN Connection TypeAccess to modem?
FTTPn/a
HFCYes*
FTTCNo
FTTN/FTTBYes

The two other connection types (Fixed Wireless and Satellite) I have no idea about, so I haven’t included them in this table.

HFC

For HFC, the modem makes it difficult to access the connection statistics page. Thankfully, it’s fairly easy to get in – follow the instructions above (the modem IP is 192.168.0.1), then hard reset the modem by inserting a pin in the reset hole for around 10 seconds.

The reset is complete when the green lights go out and a blue light flashes a couple times – once you see that, release the pin and load http://192.168.0.1/main.html in a browser. Chances are very good you’ll need to refresh several times before the status page displays.

If the modem restarts for any reason (power outage, NBN outage etc), you’ll need to do another hard reset to get back in.

FTTN/FTTB

For these connection types, you supply your own modem – no special instructions required. 🙂 Refer to your modem’s documentation to understand what the default IP address and access details are.

References

I found the following sites helpful in writing this post:

https://owennelson.co.uk/accessing-a-modem-through-a-ubiquiti-usg/
https://forums.whirlpool.net.au/archive/90ym1z23

Pink White Black Purple Blue Textile Web Scripts

Hash PII Data with Google Tag Manager

I thought Google Tag Manager (GTM) would have a built-in method to hash Personally Identifiable Information (PII) – but it doesn’t, and I couldn’t for the life of me find a simple guide on how to do it.

Plenty of guides on how to obfuscate PII in URLs and query strings, even a couple guides on encrypting – but nothing on hashing.

Isn’t encrypting and hashing the same?

In short: no. Encrypting implies that there is a method to decrypt the data again, where hashing is a one-way operation. Keep in mind that with GTM, any data manipulation happens client-side using JavaScript. Because it’s happening client-side – you have to hand over the encryption mechanism and key to the end user. Not exactly fantastic for security…

Why would I want to do this?

Any time you want to send PII data to a 3rd party vendor, it should be hashed using an agreed-upon hash algorithm. As an example: Facebook needs a hashed version of a user’s email address to do advanced matching. The Facebook Pixel tag will hash the email address for you – but if you’re using the pixel image instead, you need to hash that yourself.

Note: Google Analytics does not allow the use of PII data, hashed or otherwise. This is probably not a rule you want to break.

How do I do this?

There are three steps involved:

  1. Declare the incoming data as a Data Layer Variable
  2. Load a library for the hash algorithm you want to use
  3. Create a Custom JavaScript Variable to hash the data

Declare a Data Layer Variable

Your CMS will need to surface the PII data (such as an email address) into the dataLayer object somehow. How that happens is outside the scope of this article, but the Google Tag Manager for WordPress plugin happily adds the email of the logged-in user in a data point called visitorEmail.

Once the PII data is in the Data Layer, it needs to be declared in GTM:

  • Go to the container for your site and select Variables on the left
  • Create a new User-Defined Variable
  • Select Data Layer Variable from the list of variable types
  • For Data Layer Variable Name, enter the name of the variable exactly as it appears in the dataLayer object

For PII with alpha characters (such as email addresses), you also need to lower-case the string. While you’re in the variable configuration screen:

  • Open the Format Value section
  • Check the box for Change Case to…
  • Select Lowercase in the dropdown

Load a JS Library to Hash Data

Note: Do not write your own library unless you are a cryptographer with years of experience. Do validate that the library you choose does what you expect.

I have minimally tested the js-sha256 library and have confirmed it correctly hashes data. It doesn’t appear to do anything nefarious, but I am not a programmer – so you should do your own testing and code review!

Next, whatever you’re using needs to be hosted somewhere – hosting it yourself is an option, but I prefer using an established CDN. The js-sha256 library is available on Cloudflare’s cdnjs CDN:

https://cdnjs.cloudflare.com/ajax/libs/js-sha256/0.9.0/sha256.min.js

If you’re using a CDN-hosted library, you should make sure you’re using a specific version of the code and that there’s an SRI Hash in your tag configuration to make sure someone doesn’t tamper with the code.

Here’s how to load a library in GTM:

  • Go to the container for your site and select Tags on the left
  • Create a new Custom HTML tag
  • Go to the SRI Hash tool linked above and create an SRI Hash for your library
  • Edit the output for use in a Custom HTML tag (see second code block below)

For the js-sha256 library hosted by cdnjs, the SRI-hashed version looks something like:

<script src="https://cdnjs.cloudflare.com/ajax/libs/js-sha256/0.9.0/sha256.min.js" integrity="sha384-2epjwyVj8M4n8AweIsY7SKPSJmqBBBkmksXvkmtYORfxPS1I4NZE/+Ttk/9gCELG" crossorigin="anonymous"></script>

To use this in a Custom HTML tag, you have to use JavaScript to construct the script tag (for some reason, GTM thinks the above isn’t valid HTML).

Here’s what the result looks like – note how the src, integrity and crossorigin attributes are set:

<script>
  (function() {
  var script = document.createElement('script');
  script.type = 'text/javascript';
  script.src = 'https://cdnjs.cloudflare.com/ajax/libs/js-sha256/0.9.0/sha256.min.js';  
  script.setAttribute('integrity','sha384-2epjwyVj8M4n8AweIsY7SKPSJmqBBBkmksXvkmtYORfxPS1I4NZE/+Ttk/9gCELG');
  script.setAttribute('crossorigin','anonymous');
  document.getElementsByTagName('head')[0].appendChild(script);
  })();
</script>

You can optionally set this tag to load once per page – so long as the function is available, it should work fine for subsequent events.

Create a Custom JavaScript Variable

Last but not least – hash the data. When hashing data, it’s critically important to make sure that the source data is what you expect before hashing. After all, there’s no way to check after it’s been hashed!

In my example of hashing an email address in the visitorEmail variable, I want to run some very basic tests on the string to make sure it looks like an email address. In the code block below, you can see the test called out – you should do something similar for other types of data.

For the final piece – set up a Custom JavaScript variable:

  • Go to the container for your site and select Variables on the left
  • Create a new User-Defined Variable
  • Select Custom JavaScript from the list of variable types
  • In the Custom JavaScript box, enter the following:
function() {
  // Test email address first
  function emailIsValid (email) {
    return /\S+@\S+\.\S+/.test(email)
  }
  // If email address is valid, hash it
  if (emailIsValid({{visitorEmail}})) {
    var hash = sha256({{visitorEmail}});
    return hash;
  } else {
    return undefined;
  }
}

Next steps

Test things! Use the Preview function to make sure the variable you created actually contains a hash of the PII data you want to send to the vendor. Test with known bad data – does the code in the Custom JavaScript variable catch it?

Once you’re satisfied that the code is actually working properly, map it to the vendor in your tag configuration – safe in the knowledge you’re not sending PII data in the clear! 🙂