Standby hub

A standby hub is a second host, away from the primary hub’s site, that keeps the primary’s newest backup archive restored and ready for disaster recovery. When you activate it, it serves under the primary’s own names, which DNS moves to it, and the sites link to it with the bundles they already have. Only its published leaf address is its own.

The primary’s first init gives that address with --standby-leaf-url, so every site bundle lists the standby’s leaf address after the primary’s. Between refreshes the standby stays down, its leaf port is closed and no site links to it while the primary runs.

A standby is a hub on one server, and it restores the archives of a hub on one server.

What each refresh does#

standby-refresh.sh, which init writes beside the stack of a hub on one server, refreshes the standby. It changes nothing on a stack whose .env has no WEIKIO_STANDBY_FROM, which only a standby sets, and nothing while the stack’s backend runs. Each run:

1. Reads the manifest.json of every archive at the address WEIKIO_STANDBY_FROM gives and picks the newest, downloads it below backups/standby/ and checks it with restore --verify-only, which opens every file with the standby’s master key. An archive that fails the check is deleted, and nothing else changes.
2. Stops the stack and removes the message server’s storage, the gateway’s and the forwarder’s volumes and every file of the keys directory but operator.nk.
3. Restores the archive as a whole restore does, with init between its two runs, using the standby’s own leaf address.
4. Stops the stack again, keeps the seven newest archives and appends one line to standby-refresh.log in the home: the time, the result, the archive, and one sentence per hub name with the certificate it’s served with once the standby is activated and the days that certificate has left.

A sentence starts with Warning: when an uploaded certificate has under 30 days left, when the renewal of a certificate the platform obtained is overdue, or when a name would have only a temporary certificate in an installation that obtains no certificates by itself. A step that fails ends the run with its line in the log and the stack down, and the next run starts again from the bucket. Two runs never overlap, because the script uses flock.

Set up a standby#

Do these steps once, on the standby’s host, with IMG set as for a first start:

1. Create the home and its keys directory, and put the master key and the operator seed in them, as the first step of a whole restore does, without the archive. When the primary’s master key is changed with weikio-backend secrets rotate, put the new key here too.

2. Write the address and the credentials of a key that may only read the bucket into a file only root can read, such as /root/weikio-bucket.env, with one line each for RCLONE_S3_PROVIDER, RCLONE_S3_ENDPOINT, RCLONE_S3_ACCESS_KEY_ID and RCLONE_S3_SECRET_ACCESS_KEY. List the archives and download the newest one:

   Terminal

   $ docker run --rm --env-file /root/weikio-bucket.env --entrypoint rclone $IMG lsf --dirs-only :s3:weikio-backups/acme
   $ docker run --rm -u 1654:1654 -v /opt/weikio:/opt/weikio --env-file /root/weikio-bucket.env --entrypoint rclone $IMG \
     copy :s3:weikio-backups/acme/<newest> /opt/weikio/backups/standby/<newest>

3. Open its keys and write the stack with the standby’s own leaf address. The archive’s install record gives that address as the primary’s standby, so init gets the primary’s leaf address as --standby-leaf-url. init stops with exit code 6, as in a restore:

   Terminal

   $ docker run --rm -u 1654:1654 -v /opt/weikio:/opt/weikio -e WEIKIO_HOME=/opt/weikio -e Weikio__Keys__Dir=/opt/weikio/keys \
     -e WEIKIO_MASTER_KEY_FILE=/opt/weikio/master.key -e Weikio__Nats__Urls__0=nats://nats:4222 \
     $IMG weikio-backend restore --from /opt/weikio/backups/standby/<newest>
   $ docker run --rm -u 1654:1654 -v /opt/weikio:/opt/weikio -e WEIKIO_HOME=/opt/weikio -e Weikio__Keys__Dir=/opt/weikio/keys \
     -e WEIKIO_MASTER_KEY_FILE=/opt/weikio/master.key -e Weikio__Nats__Urls__0=nats://nats:4222 -e WEIKIO_BACKEND_IMAGE=$IMG \
     $IMG weikio-backend init --public-leaf-url wss://nats.standby.acme.example:443 --standby-leaf-url wss://nats.acme.example:443

4. Set .env as the primary has it, put the lines of /root/weikio-bucket.env in place of the primary’s RCLONE_S3_* lines, add the standby’s own two lines and delete the file. WEIKIO_STANDBY_FROM may also be a folder on this host that has the archives, such as a mounted share:

   .env

   WEIKIO_STANDBY_FROM=s3://weikio-backups/acme
   WEIKIO_STANDBY_LEAF_URL=wss://nats.standby.acme.example:443

5. Put no certificate on this host. While the standby is passive, each refresh writes a temporary certificate for each of the hub’s names and its port is closed. Once it’s activated, the admin UI’s address is served the primary’s certificate while that is valid, and the platform obtains one for the standby’s own leaf address about a minute after the start. The certificates of the gateway hostnames come back with the archive.

6. Run the first refresh, which prints its line:

   Terminal

   $ sudo ./standby-refresh.sh

7. Run it every 6 hours with this line in /etc/cron.d/weikio-standby-refresh. The file it writes has the output of the latest run, which says why a failed run failed:

   /etc/cron.d/weikio-standby-refresh

   0 */6 * * * root /opt/weikio/deploy/compose/single/standby-refresh.sh > /var/log/weikio-standby-refresh.last 2>&1

Take over with the standby#

To activate the standby, remove its cron line, start its stack with docker compose up -d in its stack folder, and point the primary’s names in DNS at the standby: the backend’s name and every gateway hostname on the public entrance. The sites link to the standby’s leaf address, which their bundles already list after the primary’s, so nobody visits a site.

Once activated, the standby’s backend writes the allowed networks the primary had when the archive was taken, after its first pass over the restored stores. Until then it accepts callers from every address and /readyz/node returns 503. While the standby serves, every refresh is refused and says so in its line, so keep the cron line out until the standby is passive again.

Keep it on the primary’s release#

When you upgrade the primary, paste the same images.env lines into the standby’s .env, so its next refresh writes the stack of the primary’s release. Upgrades and rollback covers the primary’s upgrade.