Upgrades and rollback

Every image in the stack is pinned by tag and digest, so an installation changes only when you upgrade it. A release publishes images.env, one line per image, and upgrade.sh in the stack folder moves the stack to that release. The integrations on your sites keep running while the hub upgrades. From the first published release on, a release only adds to what the platform saves and sends and never removes or renames anything, so an installation upgrades in place and keeps everything it has.

Upgrade a hub on one server#

Give upgrade.sh the release’s images.env by its path or its https:// address:

$ sudo sh ./upgrade.sh https://weik.io/downloads/releases/2026.2.1/images.env

The script does these steps in order:

1. It takes a backup of the installation into backups/upgrade-<release>-<time in UTC>, named after the release the stack runs.
2. It pulls the new release’s backend image and runs its weikio-backend upgrade, which writes the platform’s files of the stack folder again from weikio-install.json with the release’s templates and writes the release’s image lines into .env. Every other line of .env and every file of yours stays as it is, and what it replaces goes into .upgrade/<release>/.
3. It pulls the release’s images and brings the stack up on them.
4. It waits up to three minutes each for the message server and the backend to respond, and ends with doctor.

The API is unavailable from the moment the backend container is recreated until it answers again, and the script’s last line gives that time in seconds. Running the upgrade again to the release the stack already runs changes no file and exits 0, so a run that stopped part way is run again from the start.

Afterwards docker compose logs init shows one line per step the release applied, upgrade step <id> applied: <sentence>, and weikio audit query --action UpgradeStepApplied lists them in the audit trail.

See what would change#

$ sudo sh ./upgrade.sh <images.env> --check

--check shows what the upgrade would change and writes nothing. A platform file that was edited by hand stops the upgrade before the stack changes and gives the file’s name. Move the edit into docker-compose.override.yaml or .env and run the upgrade again, or give --replace-edited <file> to have the release’s copy take its place. Pasting image lines into .env by hand isn’t an upgrade, because the platform’s files stay as the previous release wrote them.

Roll back#

To go back to the release the last upgrade moved the stack from:

$ sudo sh ./upgrade.sh --rollback

The rollback moves back every file the upgrade kept in .upgrade/<release>/, .env and weikio-install.json among them, brings the stack up on that release’s images, waits and ends with doctor. It keeps every record stored since the upgrade and the keys directory, which the previous release uses as they are. The allowed networks stay as they’re set, changes made since the upgrade included.

When the upgrade moved the message server to another minor version, the rollback changes nothing and gives the backup the upgrade took, because a message server can’t use storage that a later minor version wrote. Restore that backup as Backups and restore says.

Upgrade a hub on several servers#

A hub on several servers upgrades one server at a time while the load balancer sends callers and site links to the others. doctor on any server gives the order: the standard servers first, then the data servers that don’t lead the storage, and the data server that leads it last, never two data servers at once.

$ docker compose run --rm infra weikio-backend doctor

Its servers row shows on 3 of 3 for three servers while every server responds, and its upgrade order row lists the servers in the order to walk. Then, for each server in that order:

1. From the second server on, copy the keys directory of the first server you upgraded onto this one, so it has the key files the release added:

   Terminal

   $ ssh <first server> 'tar -C / -cpf - opt/weikio/keys' | ssh <server> 'tar -C / -xpf -'

2. In the server’s own node directory, /opt/weikio/deploy/compose/cluster/node-<n>, run the upgrade:

   Terminal

   $ sudo sh ./upgrade.sh <images.env>

   It takes the server out of the load balancer’s pool before any of its services restarts and brings the stack up. It then waits for the message server, the backend and two of the balancer’s checks, so the server is back in the pool before doctor runs.

3. Go on to the next server only when upgrade.sh ended with doctor exiting 0 and the servers row lists every server.

deploy/drills/rolling-upgrade-drill.sh in the release’s source walks a hub through these steps from a machine whose secure shell reaches every server, printing the time of each step, and stops at the first step that fails, with the server and the rollback command. To go back, walk the servers in the same order and run sudo sh ./upgrade.sh --rollback in each node directory.

To pull the images from a registry of your own that keeps the upstream paths, set WEIKIO_REGISTRY on its line near the top of .env, above the image lines.

Upgrade the sites#

Then upgrade each site with the same release’s images.env. In the folder the site’s setup.sh installed the agent in, run:

$ sudo sh ./setup.sh --upgrade <images.env>

It pulls the images, replaces the site’s .env with the file and waits until the agent is online again. An agent on an older release keeps working, shows as behind, and weikio problems lists it with the release to upgrade to. Every site’s agent runs on a Linux host, so every site upgrades this way.

A standby hub follows the primary when you paste the same images.env lines into its .env, so its next refresh writes the stack of the primary’s release.