Encryption at rest

Encryption at rest has two layers in Weik.io. The platform itself seals every value that would let someone sign in or act for it, with AES-256-GCM under the installation’s master key, and every backup archive is sealed whole with the same key. Everything else lies on the hub’s disks, so those disks are encrypted, by the cloud provider or with Linux’s own disk encryption.

What the platform seals#

The platform seals these with the master key:

• the secrets and the secret values of integrations
• the local users’ password hashes and second factors
• the signing keys of the platform’s tokens
• the keys of the platform’s certificates: the ACME account, the platform’s authority and the client certificates
• the forwarding token and the platform’s own secrets, such as a DNS connection’s token
• the sign-in refresh tokens
• every backup archive, whole

The master key is a file on each backend host, readable only by the user the images run as, and it’s never in the message server’s storage or in a backup.

What the disks keep in clear#

Everything else the hub keeps lies in clear on its disks, and disk encryption protects it against a disk that’s lost, discarded or copied:

A site keeps in clear what it runs with: its credentials, each integration’s settings with the values of its secrets filled in, and the site gateway’s route tables. The same disk rule applies to every site.

What every hub needs#

Docker’s data folders, /var/lib/docker and /var/lib/containerd, and the home, /opt/weikio, sit on an encrypted disk, set up before Docker is installed. The stack’s stores are Docker volumes, so a home on an encrypted disk alone leaves them in clear. Swap space is off or encrypted.

The keys follow four rules:

• The key that opens a disk is never on the disk it opens.
• Each server has its own key, so one leaked key opens one server.
• A hub opens its disks without a person, from a key file on another disk, from key servers on your network or by the provider’s own encryption. A passphrase typed at every start would leave the hub down after a power cut until someone types it.
• Docker waits for the encrypted disks, because Docker started before they’re open creates empty folders in their place. With systemd, the line RequiresMountsFor=/var/lib/docker /var/lib/containerd /opt/weikio in a drop-in for both docker.service and containerd.service does it.

Keep these off the servers and apart from the backup copies: the copy of the master key, every earlier master key for as long as archives sealed with it are kept, the operator seed and each server’s disk recovery passphrase.

Providers#

Azure and Google encrypt their servers’ disks by default. AWS encrypts them when it’s switched on, for example as the region’s default before the servers are created. On Hetzner and on bare metal, encrypt the disks with LUKS2, as the steps below do. Hetzner’s backups and snapshots of a server don’t include the volumes attached to it, so the hub’s data on such a volume is kept by the platform’s archives.

Copy the archives to a bucket that encrypts what it stores by itself, as AWS S3 and UpCloud Object Storage do by default. The archive is sealed with the master key in any case, so a copy opens nothing without it.

Proof for an auditor#

These commands show that the folders sit on an encrypted disk and that swap space is safe:

$ lsblk -o NAME,TYPE,FSTYPE,MOUNTPOINTS
$ findmnt -T /var/lib/docker
$ findmnt -T /var/lib/containerd
$ findmnt -T /opt/weikio
$ cryptsetup status <name>
$ swapon --show

lsblk shows each mount of the three folders on a device of type crypt, and findmnt gives the device each folder is on. cryptsetup status with that device’s name prints its cipher and that it’s active. swapon --show prints nothing when swap space is off. Where the provider encrypts the disks, its console shows the setting for each disk, and an archive’s manifest.json gives the master key it’s sealed with.

On Hetzner and bare metal#

These steps put Docker’s data folders and the home on one data disk encrypted with LUKS2, which the server opens at every start from a key file on its system disk. They’re the steps of Ubuntu 24.04, run as root on a new server before Docker is installed. A hub that runs already moves onto such a server with a backup and a restore.

Before step 1, check from your own machine that ssh root@<server> true returns without an error, because step 7 copies the recovery passphrase off the server over ssh.

The data disk. The steps format the data disk, so use an empty one. On Hetzner it’s a Volume created for the server with no file system on it:

$ hcloud volume create --name hub-data --size <GB> --server <name of the server>
$ hcloud volume describe hub-data

On bare metal it’s a second disk. On the server, find the disk’s name under /dev/disk/by-id/, set it once as disk and check that the disk is empty. wipefs prints nothing for an empty disk, and when it prints anything, such as ext4 or gpt, the disk has data: stop and check the name.

$ lsblk -o NAME,TYPE,FSTYPE,SIZE,MOUNTPOINTS
$ ls -l /dev/disk/by-id/
$ disk=/dev/disk/by-id/scsi-0HC_Volume_<id>
$ wipefs --noheadings --output TYPE "$disk"

Until step 7, a step that stops part way can be run again from its first line, because the steps format nothing twice and add no line twice.

1. Turn swap space off and keep it off. Both commands print nothing when it’s off. A bare-metal install of Ubuntu makes the swap file /swap.img: turn it off with swapoff /swap.img, remove its line from /etc/fstab and delete the file.

   Terminal

   $ swapon --show
   $ awk '!/^[[:space:]]*#/ && $3 == "swap"' /etc/fstab

2. Install the disk tools, write the key file on the system disk, and draw a recovery passphrase into /run, which is in memory, so the passphrase never lands on a disk of the server. The passphrase is 40 lower-case letters and digits, without those that look alike.

   Terminal

   $ apt-get update && apt-get install -y cryptsetup-bin
   $ mkdir -p -m 0700 /etc/luks /run/weikio-luks
   $ (umask 0277; set -o noclobber; head -c 64 /dev/urandom > /etc/luks/weikio.key)
   $ (umask 0077; head -c 4096 /dev/urandom | LC_ALL=C tr -dc abcdefghjkmnpqrstuvwx23456789 | (set -o noclobber; head -c 40 > /run/weikio-luks/recovery))

3. Format the disk under the key file, add the recovery passphrase in a second key slot, and take the header backup. The key file’s slot opens the disk at once at every start, and the passphrase’s slot keeps the slow key derivation a typed passphrase needs.

   Terminal

   $ [ -z "$(wipefs --noheadings --output TYPE "$disk")" ] && cryptsetup luksFormat --batch-mode --type luks2 --cipher aes-xts-plain64 --key-size 512 --pbkdf pbkdf2 --pbkdf-force-iterations 1000 --key-file /etc/luks/weikio.key "$disk"
   $ cryptsetup luksDump "$disk" | grep -q '^  1: luks2' || cryptsetup luksAddKey --key-file /etc/luks/weikio.key --pbkdf argon2id "$disk" /run/weikio-luks/recovery
   $ [ -e /run/weikio-luks/header.img ] || (umask 0077; cryptsetup luksHeaderBackup "$disk" --header-backup-file /run/weikio-luks/header.img)

4. Add the disk to /etc/crypttab with the key file, so the server opens it at every start without a person, then open it. The first lines keep copies of /etc/fstab and /etc/crypttab as they were.

   Terminal

   $ touch /etc/crypttab
   $ for f in /etc/crypttab /etc/fstab; do [ -e "$f.before-weikio" ] || cp -p "$f" "$f.before-weikio"; done
   $ for f in /etc/crypttab /etc/fstab; do if [ -s "$f" ] && [ "$(tail -c 1 "$f" | od -An -c | tr -d ' ')" != '\n' ]; then printf '\n' >> "$f"; fi; done
   $ grep -q '^weikio ' /etc/crypttab || echo "weikio ${disk:?} /etc/luks/weikio.key luks,nofail,no-read-workqueue,no-write-workqueue" >> /etc/crypttab
   $ systemctl daemon-reload
   $ systemctl start systemd-cryptsetup@weikio.service

   With no-read-workqueue and no-write-workqueue each read and write goes straight to the disk. On a Hetzner Volume a write that waits until it’s on the disk took 5 to 9 ms with them, the same as unencrypted, and about 1 ms more without.

5. Make the file system, mount it on /data, and bind-mount a folder of it on each of the three folders, so one disk and one key cover all three. Then measure the disk’s speed.

   Terminal

   $ [ -z "$(wipefs --noheadings --output TYPE /dev/mapper/weikio)" ] && mkfs.ext4 -L weikio /dev/mapper/weikio
   $ mkdir -p /data
   $ grep -q ' /data ' /etc/fstab || echo '/dev/mapper/weikio /data ext4 defaults,nofail 0 2' >> /etc/fstab
   $ systemctl daemon-reload
   $ mount -a
   $ mkdir -p /data/docker /data/containerd /data/weikio /var/lib/docker /var/lib/containerd /opt/weikio
   $ chmod 0710 /data/docker
   $ chmod 0711 /data/containerd
   $ grep -q ' /var/lib/docker ' /etc/fstab || echo '/data/docker /var/lib/docker none bind,nofail,x-systemd.requires-mounts-for=/data 0 0' >> /etc/fstab
   $ grep -q ' /var/lib/containerd ' /etc/fstab || echo '/data/containerd /var/lib/containerd none bind,nofail,x-systemd.requires-mounts-for=/data 0 0' >> /etc/fstab
   $ grep -q ' /opt/weikio ' /etc/fstab || echo '/data/weikio /opt/weikio none bind,nofail,x-systemd.requires-mounts-for=/data 0 0' >> /etc/fstab
   $ systemctl daemon-reload
   $ mount -a
   $ dd if=/dev/zero of=/data/disk-speed bs=4k count=1000 oflag=dsync
   $ rm /data/disk-speed

6. Write Docker’s drop-ins now, so Docker’s first start after its install already waits for the disk:

   Terminal

   $ mkdir -p /etc/systemd/system/docker.service.d /etc/systemd/system/containerd.service.d
   $ printf '[Unit]\nRequiresMountsFor=/var/lib/docker /var/lib/containerd /opt/weikio\n' > /etc/systemd/system/docker.service.d/override.conf
   $ printf '[Unit]\nRequiresMountsFor=/var/lib/docker /var/lib/containerd /opt/weikio\n' > /etc/systemd/system/containerd.service.d/override.conf
   $ systemctl daemon-reload

7. Check that both keys open the disk. The first command prints “Key slot 0 unlocked.” and the second “Key slot 1 unlocked.”:

   Terminal

   $ cryptsetup open --test-passphrase --verbose --key-file /etc/luks/weikio.key "$disk"
   $ cryptsetup open --test-passphrase --verbose --key-slot 1 --key-file /run/weikio-luks/recovery "$disk"

   Then, from your own machine, copy the recovery passphrase and the header backup off the server into a folder only you can read, and remove them on the server only when the copy matches. Set the server’s address as server and your folder as keys:

   Terminal

   $ server=<address of the server>
   $ keys=<a folder of yours>
   $ mkdir -p -m 0700 "$keys" "$keys/$server"
   $ scp -r "root@$server:/run/weikio-luks" "$keys/$server/"
   $ here="$(cd "$keys/$server/weikio-luks" && sha256sum header.img recovery)"
   $ there="$(ssh "root@$server" 'cd /run/weikio-luks && sha256sum header.img recovery')"
   $ if [ -n "$here" ] && [ "$here" = "$there" ]; then ssh "root@$server" rm -r /run/weikio-luks && echo removed; else echo "the copy differs: copy again"; fi

   You see

   removed

   Go on to step 8 only after it printed removed, because step 8 restarts the server and the restart empties /run. Keep the passphrase where only the people who may open the disk can read it, such as one entry per server in a password manager, with the header backup beside it.

8. Bring the server up to date, install Docker from Docker’s own package repository with the log limits, and restart the server:

   Terminal

   $ apt-get update && DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade -y
   $ install -m 0755 -d /etc/apt/keyrings
   $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
   $ chmod a+r /etc/apt/keyrings/docker.asc
   $ echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" > /etc/apt/sources.list.d/docker.list
   $ apt-get update && apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
   $ mkdir -p /etc/docker
   $ echo '{ "log-driver": "json-file", "log-opts": { "max-size": "50m", "max-file": "3" } }' > /etc/docker/daemon.json
   $ systemctl restart docker
   $ systemctl reboot

   The server comes back by itself within a minute. Log in and check the disk with the auditor’s commands above and cryptsetup status weikio, which prints “/dev/mapper/weikio is active and is in use.” and the cipher aes-xts-plain64. lsblk shows weikio of type crypt mounted on /var/lib/docker, /var/lib/containerd, /opt/weikio and /data.

When the key file is lost#

When the key file is missing at a start, the server still starts and answers over ssh, the disk stays closed and Docker doesn’t start. Open the disk with the recovery passphrase, start Docker, and give the disk a new key file in place of the lost one. cryptsetup open, luksKillSlot and luksAddKey each ask for the passphrase:

$ systemctl stop systemd-cryptsetup@weikio.service
$ cryptsetup open --perf-no_read_workqueue --perf-no_write_workqueue "$disk" weikio
$ mount -a
$ systemctl start containerd docker
$ (umask 0277; head -c 64 /dev/urandom > /etc/luks/weikio.key)
$ cryptsetup luksDump "$disk" | sed -n '/^Keyslots:/,/^Tokens:/p' | grep -E '^  [0-9]+: |PBKDF:'
$ cryptsetup luksKillSlot "$disk" 0
$ cryptsetup luksAddKey --key-slot 0 --pbkdf pbkdf2 --pbkdf-force-iterations 1000 "$disk" /etc/luks/weikio.key
$ cryptsetup config --priority prefer --key-slot 0 "$disk"

The key slots changed, so take a new header backup into /run/weikio-luks/header.img with cryptsetup luksHeaderBackup, copy it off the server as in step 7 and put it in place of the earlier copy. A header backup taken before the change would bring the lost key file’s slot back.

When the header is damaged#

When the start of the disk that keeps the key slots is damaged, no key opens it. On the server, stop Docker and close the disk:

$ systemctl stop docker.socket docker containerd
$ umount /var/lib/docker /var/lib/containerd /opt/weikio /data
$ cryptsetup close weikio

From your own machine, with server and keys set as in step 7, copy the header backup to the server:

$ scp "$keys/$server/weikio-luks/header.img" "root@$server:/run/header.img"

Then write it back on the server and restart it:

$ cryptsetup luksHeaderRestore --batch-mode "$disk" --header-backup-file /run/header.img && rm /run/header.img && systemctl reboot

The restore brings back the key slots as they were when the backup was taken, which is why a new header backup follows every change of a key slot.