Software RAID on Ubuntu Linux

Preamble

I recently purchased myself a HP ProLiant MicroServer with a stack of Western Digital Red hard drives with the intention of building a media server. After stumbling through a few tutorials that didn’t work very well, a colleague took pity on me and gave me some really good advice on how to get things running – so I figured a post on how to build a software-based RAID using Ubuntu Linux might be a reasonably useful thing. 🙂

Ingredients

  • Server hardware and dedicated hard drives for the RAID array
    (shouldn’t really matter what specific hardware gets used)
  • Ubuntu Server 13.04 or later
  • Patience

Recipe

First up, install the server edition of Ubuntu Linux. All the default install options are probably fine; there are enough guides on the Internets to cover setting this up if you get lost.

Once Linux is up and running and all of the hard drives have been detected appropriately (use lsblk to confirm device names), use parted to configure the partition table and create a partition on each disk..:

sudo parted -a optimal
select /dev/sda
mklabel gpt
mkpart
  -> raid_d1
  -> ext4
  -> 1MiB
  -> 3TiB
quit

Rinse and repeat for each drive – you’ll need to change the device being selected each time (use the output from lsblk as a guide).

For the mkpart section, the question ‘Partition name?’ is mostly irrelevant (I went with “raid_d1”, “raid_d2” etc) .. and for the question ‘End?’, use the advertised size of the drive (in my case, 3 Terabytes).

In hindsight, because I’m using the full drive for the array (vs. some for the array and some for other partitions), I’m not even sure that creating partitions using parted > mkpart is entirely necessary. It doesn’t take long though, so I guess it doesn’t hurt. 😉

Time to build the array (this is where having patience comes in)..:

sudo mdadm --create /dev/md0 --chunk=256 --level=5 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sdd

My colleague suggested a chunk size of 256k (vs. the default of 512k) for better performance. Other than that, the command should make plenty of sense: you’re creating a RAID 5 array at /dev/md0 with 4 devices (as listed at the end of the command).

In case it complains mdadm isn’t installed..:

sudo aptitude install mdadm

This will take ages .. ~18 hours for me. Monitor the progress with..:

watch -n5 cat /proc/mdstat

Once the array has completed building, you need to..:

  1. Update /etc/mdadm/mdadm.conf to list the hard drives that form part of the array (DON’T list the array device itself)
  2. Update initrd
  3. Reboot

I customised the following two lines in mdadm.conf (no need to change anything else from defaults)..:

DEVICE /dev/sda /dev/sdb /dev/sdc /dev/sdd
MAILADDR some@email.address

then

sudo update-initramfs -u
sudo reboot

After the reboot, your RAID array should turn up at /dev/md0. Confirm with the following..:

sudo mdadm --detail /dev/md0

If it’s not there, try..:

sudo mdadm --detail /dev/md127

If the array turns up at /dev/md127, it could mean that you have problems with mdadm.conf – see this Ubuntu Forums thread for further reading.

Now the array is configured, time to format it. Choice of filesystem is up to you, but I chose XFS (it plays nice with RAID and large file sizes). First, install XFS as it’s not a part of the standard Ubuntu Server image, then create the filesystem..:

sudo aptitude install xfsprogs
sudo mkfs.xfs -L data /dev/md0

Finally, get the UUID of the file system and load that in to /etc/fstab. Use blkid to find the UUID that belongs to /dev/md0..:

sudo blkid
/dev/md0: LABEL="data" UUID="3d3cf1c1-6015-4b5d-ac08-e38832fa29d6" TYPE="xfs"

Now, add that to /etc/fstab (I’m using /data as my mount point, but whatever works best for you)..:

# RAID array
UUID=3d3cf1c1-6015-4b5d-ac08-e38832fa29d6 /data xfs defaults 0 0

Reboot once more for good measure. Hopefully you get something like this as your df -h output..:

Filesystem Size Used Avail Use% Mounted on
/dev/sde1 28G 1.4G 25G 6% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 929M 8.0K 929M 1% /dev
tmpfs 188M 284K 188M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 939M 4.0K 939M 1% /run/shm
none 100M 0 100M 0% /run/user
/dev/md0 8.2T 15G 8.2T 1% /data

😀

Add a VPN to a FRITZ!Box router

FRITZ!Box modem routers are able to run IPSec VPN endpoints, and even come with a clever VPN configuration utility .. that’s only available for Windows. If you don’t run Windows, you’re mostly out of luck (at least, as far as the official documentation goes), because the assumption is that everyone is running Windows (!)

If you’re not running Windows, you’ll need to create the VPN configuration file and upload it to the FRITZ!Box router by hand. It’s actually not that tricky, but the instructions are near impossible to find on the Internets – so here’s my own version. I’ve borrowed liberally from Marius van Witzenburg’s blog post on the same topic.

Copy the following in to a plain text file and save it as vpn.cfg:

vpncfg {
        connections {
                enabled = yes;
                conn_type = conntype_user;
                name = "NAME";
                always_renew = no;
                reject_not_encrypted = no;
                dont_filter_netbios = yes;
                localip = 0.0.0.0;
                local_virtualip = 0.0.0.0;
                remoteip = 0.0.0.0;
                remote_virtualip = VPN_CLIENT_IP;
                remoteid {
                        key_id = "NAME";
                }
                mode = phase1_mode_aggressive;
                phase1ss = "all/all/all";
                keytype = connkeytype_pre_shared;
                key = "SHARED_SECRET";
                cert_do_server_auth = no;
                use_nat_t = yes;
                use_xauth = yes;
                use_cfgmode = no;
                xauth {
                        valid = yes;
                        username = "USERNAME";
                        passwd = "PASSWORD";
                }
                phase2localid {
                        ipnet {
                                ipaddr = 0.0.0.0;
                                mask = 0.0.0.0;
                        }
                }
                phase2remoteid {
                        ipaddr = VPN_CLIENT_IP;
                }
                phase2ss = "esp-all-all/ah-none/comp-all/no-pfs";
                accesslist = "permit ip any VPN_CLIENT_IP 255.255.255.255";
        }
        ike_forward_rules = "udp 0.0.0.0:500 0.0.0.0:500", "udp 0.0.0.0:4500 0.0.0.0:4500";
}

A few changes are needed — highlighted above and explained below:

  • NAME – replace these (x2) with an alphanumeric name – if you’re using an Android device, it’s important that you do NOT use special characters, symbols or any punctuation (if you do, the connection will likely fail without any sort of useful error message)
  • VPN_CLIENT_IP – replace these (x3) with an IP address from the internal network range being used by the FRITZ!Box, but outside the DHCP pool range. If you haven’t changed any of the default settings in the Home Network > Network > Network Settings section, then use 192.168.178.201 (if you have made changes, you’ll need to work out what IP address to use for yourself :))
  • SHARED_SECRET – replace this with a long, alphanumeric password – I use a random string about 30 characters long – just bear in mind that you’ll need to enter this string on anything you set the VPN up on 😉
  • USERNAME – replace with a username (only use lowercase letters)
  • PASSWORD – replace with a password (avoid special characters, especially quote marks)

Leave everything else as is and re-save the file. Log in to the FRITZ!Box and go to the VPN configuration section (Internet > Permit Access > VPN), upload the file and you should now see a configured VPN waiting to be used. 🙂

To configure an iDevice, check this Apple Support KB article and use the following settings:

Configuration optionSetting to use
VPN TypeIPSec
DescriptionFritzVPN
ServerThe (static) IP address, hostname or DDNS hostname for the Internet service that your FRITZ!Box is connected to
AccountUSERNAME from above
PasswordLeave set to ‘Ask Every Time’ (or use PASSWORD from above, however consider the security implications of doing so..)
Use CertificateLeave set to ‘Off’
Group NameNAME from above
SecretSHARED_SECRET from above

To configure an Android device, go to Settings, select ‘More…’ under Wireless & Networks, select VPN and then tap the ‘+’ symbol. In the Edit VPN profile pop-up, use the following settings:

Configuration optionSetting to use
NameFritzVPN
TypeIPSec Xauth PSK
Server addressThe (static) IP address, hostname or DDNS hostname for the Internet service that your FRITZ!Box is connected to
IPSec identifierNAME from above
IPSec preshared keySHARED_SECRET from above

The Android instructions were written based on the AOSP release of Android 4.2 Jelly Bean – your mileage may vary with different/customised versions of Android. The earliest version of Android with built-in support for the IPSec VPN that the FRITZ!Box uses is 4.0.4 Ice Cream Sandwich – if you have an earlier version and can’t upgrade, you’ll need to buy VpnCilla from the Google Play store instead.

Setting up VPN access from a computer or other device may be slightly tricker but should follow the outline above – the actual instructions will depend on operating system version etc (Google will be your best friend here).

If multiple concurrent VPN connections are required, simply edit all of the fields per the above instructions to something else (except maybe leave SHARED_SECRET the same), save and upload as a second VPN connection.

Download Brightcove videos

2016 update: Turns out this is the most popular article on this site – with nearly 4 years of changes to Google Chrome since it was posted, the instructions needed a little tweaking..!


Brightcove typically delivers videos via a flash stream – making it almost impossible to download (as it is a stream, there is no file to save). A video capture of the stream might be possible, but there’s an easier way if the video has a .mp4 “fallback option” for iDevices.

Ingredients

  • Video that you want, delivered using a Brightcove stream
  • Google Chrome (tested with version 50.0.2661.102)

Recipe

First, you need to disable the Flash plugin in Google Chrome: open Google Chrome, go to chrome://plugins/ and one of the first plugins you should see listed is Flash. Click the Disable link and it ought to go grey.

Open a new tab and go to the web address for the video – you should see this:

If the video starts playing instead of seeing the image above, you haven’t disabled Flash properly – try the above steps again.

Open the View menu, select Developer and then select Developer Tools. Ensure the user agent dropdown at the top of the page shows “iPad”:

chrome-developer-tools

Reload the web page and a video ought to display, with a play icon overlay – that’s the video in .mp4 format. In the Developer Tools section below the website, select the Network tab, then type “mp4” in to the Filter search box. There should be a single item listed in the section at the bottom of the page – this is the video file. Right-click the filename and copy the link address.

Finally, load the copied link address in to the address bar of Google Chrome and load it – et voilà, the video should start playing. Right-click the video and save it to a file.