Backups and restore

The stack’s backup service takes an archive of the installation when it starts and then at each interval, into the home’s backups folder. Every archive is sealed with the installation’s master key, so a copy of it opens nothing on its own. The service copies each archive to your bucket, and the platform raises a problem when a backup is overdue or a copy failed. A restore brings the whole installation back onto new servers, or one record back into a running hub.

Set the schedule and the copies#

Three installation settings decide what the backup service does:

• backupEvery, how often it takes an archive, from 1h to 365d: every 24h on one server and every 6h on several when you set nothing.
• backupKeep, how many of its archives each server keeps, 7 by default.
• backupCopyTo, where each archive is copied: an s3://<bucket>/<folder> address, reached with the RCLONE_S3_* lines of .env, or an rclone remote such as offsite:backups, whose RCLONE_CONFIG_OFFSITE_* variables you give the backup service in docker-compose.override.yaml. Without it no archive is copied.

The first init sets them from --backup-every, --backup-keep and --backup-copy-to. A platform administrator changes them later:

$ weikio settings set --backup-every 6h --backup-keep 7 --backup-copy-to s3://acme-backups/weikio

On a hub of several servers every server runs the service, and the server that has the backup lease takes the archives.

What an archive contains#

Every file of the archive but its manifest.json is sealed whole with the master key: the documents, the audit trail, the built versions and the entity stores, the list of agents, the keys directory and the install record. manifest.json gives only the release that wrote the archive, when it was taken and the id of the master key it’s sealed with.

The master key and the operator seed are never in the archive. Keep both apart from the copies of the archives, never in the bucket, and keep each earlier master key for as long as archives sealed with it are kept. Without the master key no file of the archive can be read.

Some of what the hub keeps isn’t in an archive on purpose:

Protect the copies#

The hub never deletes a copy. Set the bucket up so it removes the copies itself once they’re old enough, and so no key can delete one sooner, the key on the hub’s servers included:

1. Create the bucket with Object Lock turned on, which only the bucket’s creation can do, with a default retention in COMPLIANCE mode of as many days as the bucket keeps the copies. Until a copy’s retention ends, no key can delete it or shorten its retention, so whoever takes a server and its key can’t remove the copies already made. GOVERNANCE mode doesn’t protect them, because a key with the right to bypass it deletes the copy anyway.
2. Give the bucket a lifecycle rule that expires each copy after the same days and removes it for good the same days after that.
3. Give the hub a key that may put, get and list objects and has no right to delete them or to change the bucket. Where the storage can’t limit its keys that way but accepts bucket policies, deny deletes to the hub’s key in the bucket’s policy.

For example with the aws CLI and your storage’s --endpoint-url, for a bucket acme-backups whose copies are kept 35 days:

$ aws s3api create-bucket --bucket acme-backups --object-lock-enabled-for-bucket
$ aws s3api put-object-lock-configuration --bucket acme-backups --object-lock-configuration file://lock.json
$ aws s3api put-bucket-lifecycle-configuration --bucket acme-backups --lifecycle-configuration file://lifecycle.json
$ aws s3api put-bucket-policy --bucket acme-backups --policy file://policy.json

{ "ObjectLockEnabled": "Enabled", "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE", "Days": 35 } } }

{
  "Rules": [
    { "ID": "expire", "Status": "Enabled", "Prefix": "",
      "Expiration": { "Days": 35 },
      "NoncurrentVersionExpiration": { "NoncurrentDays": 35 },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 } },
    { "ID": "markers", "Status": "Enabled", "Prefix": "",
      "Expiration": { "ExpiredObjectDeleteMarker": true } }
  ]
}

The policy below uses Hetzner Object Storage’s principal for a project’s keys, arn:aws:iam:::user/p<project id>, where p<project id> is the bucket’s owner that aws s3api get-bucket-acl shows. A policy whose principal is "*" can be accepted and still not apply to the key, so name the key the way your storage identifies its principals:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Sid": "NoDeletes", "Effect": "Deny",
      "Principal": { "AWS": ["arn:aws:iam:::user/p<project id>"] },
      "Action": ["s3:DeleteObject", "s3:DeleteObjectVersion"],
      "Resource": ["arn:aws:s3:::acme-backups/*"] }
  ]
}

Watch the backups#

Every archive is in the audit trail as BackupTaken, with its name, its size and its copy. The problem list raises backup-overdue when no archive is newer than twice the interval, and backup-copy-failed while the newest archive that tried a copy failed it. doctor prints the newest archive’s age and its copy.

Take an archive by hand#

Take an archive by hand before a change you want to be able to undo, such as an upgrade. It goes into a folder of its own, which the service never removes:

$ docker compose run --rm infra weikio-backend backup --to /opt/weikio/backups/<name>

Check an archive#

restore --verify-only opens an archive with the master key and checks it file by file without changing anything:

$ docker compose run --rm infra weikio-backend restore --from /opt/weikio/backups/<name> --verify-only

Restore one document#

A running hub restores one document from an archive as the archive’s configuration has it. A record of that name gets the archive’s configuration and keeps its ID, and with none of that name one is created:

$ docker compose run --rm infra weikio-backend restore document --from /opt/weikio/backups/<name> integration/<integration> --project <project>

One document brings back that record alone. A project document brings back nothing that was in the project, so you restore each record with a command of its own once its project is back. The command tells you which project or environment to restore first when one is missing. An integration restored this way has no version until you apply its directory again with weikio apply -f <directory>. A deleted secret isn’t restored, because the archive’s configuration has no secret value, and the command gives the weikio secret set that sets it again.

Restore the whole hub#

A whole restore brings the installation back to the time the archive was taken, onto a fresh host. The restore goes into empty storage and runs before anything else creates it:

1. Create the home and its keys directory, put the master key and the operator seed in them from where they’re kept, and copy the archive below backups/:

   Terminal

   $ install -d -o 1654 -g 1654 -m 0700 /opt/weikio /opt/weikio/backups /opt/weikio/keys
   $ cp -r <archive> /opt/weikio/backups/<name> && cp <master key> /opt/weikio/master.key && cp <operator seed> /opt/weikio/keys/operator.nk
   $ chown -R 1654:1654 /opt/weikio && chmod 0400 /opt/weikio/master.key && chmod 0600 /opt/weikio/keys/operator.nk

   To fetch the archive from the bucket instead, write the address and the credentials of a key that may read the bucket into a file only root can read, such as /root/weikio-bucket.env, then list the archives and copy the newest one by the createdAt of its manifest.json:

   Terminal

   $ docker run --rm --env-file /root/weikio-bucket.env --entrypoint rclone $IMG lsf --dirs-only :s3:acme-backups/weikio
   $ docker run --rm -u 1654:1654 -v /opt/weikio:/opt/weikio --env-file /root/weikio-bucket.env --entrypoint rclone $IMG \
     copy :s3:acme-backups/weikio/<name> /opt/weikio/backups/<name>

2. Open the archive’s keys with the backend image, IMG set as for a first start:

   Terminal

   $ docker run --rm -u 1654:1654 -v /opt/weikio:/opt/weikio -e WEIKIO_HOME=/opt/weikio -e Weikio__Keys__Dir=/opt/weikio/keys \
     -e WEIKIO_MASTER_KEY_FILE=/opt/weikio/master.key -e Weikio__Nats__Urls__0=nats://nats:4222 \
     $IMG weikio-backend restore --from /opt/weikio/backups/<name>

   It writes every key file of the archive into the keys directory and the archive’s install record where the stack goes. It refuses an operator seed of another installation.

3. Run the first start’s docker run ... init with no flags. It writes the stack from the install record, with the old host’s choices and installation id, and stops with cannot connect to nats://nats:4222 and exit code 6, both expected here. A hub restored under new names gets them in --public-base-url and --public-leaf-url. Then set .env as the old host had it.

4. Start the message server and restore the archive into it:

   Terminal

   $ docker compose up -d nats
   $ docker compose run --rm --no-deps infra weikio-backend restore --from /opt/weikio/backups/<name>

5. Start the stack with docker compose up -d. While the old hub still runs, as in a recovery test, start it with docker compose up -d --scale backup=0 instead, give the restored hub a copy target of its own with weikio settings set --backup-copy-to s3://<bucket>/<folder>, and then run docker compose up -d. That keeps its archives out of the old hub’s folder.

The restore is in the audit trail as RestoreRun. Saved links to records keep working: a record that existed at the backup’s time opens, and one made after it says it isn’t found. When the restored hub is reached on another leaf address than the old one, weikio-backend agents repoint --hub wss://nats.new.example:443 writes a new bundle for every agent, which you install at each site with its setup.sh.

A hub on several servers restores onto as many new servers as it had, behind a new load balancer, or onto one server. The README.md that init writes beside the stack has the steps for both, and a restore onto encrypted disks is how an existing hub moves onto them.