Install a hub on three or five servers

A hub on several servers keeps working when a server fails. Three or five data servers each keep a copy of the hub’s storage and choose its leader by a majority. Every server runs the same stack: the terminator, the backend, its own instance of the API gateway and a message server. Any backend answers any request, so there are no leader elections in the backend. A TCP load balancer in front takes a server out of its pool within seconds when it stops being ready.

The tested shape#

The tested hub has three servers of 4 virtual CPUs and 8 GB of memory, all three of them data servers (--servers 3 --replicas 3). Each has a disk of 100 GB under its message store and --disk-budget 74GiB, which has room for the installation’s own stores, three environments and 26 projects of the default size. On servers with their own solid-state disks this hub stored about 600 events a second, and about 70 to 100 a second on network-attached disks.

Each server is sized by its role:

A standard server keeps no storage and passes every storage request to the data servers. Node 1 also runs the telemetry forwarder, the log, metric and trace stores and the telemetry door, through which the other servers reach them.

Before you start#

Prepare each server as Install a hub on one server says under “Before you start”: Ubuntu 24.04 LTS on x86-64, encrypted disks, Docker with the Compose plugin and the server up to date. Put the servers on one private network and create the load balancer on that network, with no server in its pool yet. Point the backend’s name and the leaf address’s name at the load balancer.

Open these ports between the servers only, with the network firewall of your provider:

• TCP 6222 between all servers, for their message servers’ links.
• TCP 9428, 8428, 10428 and 14319 from the other servers to node 1, for the stores and the forwarder. A hub written with --no-observability needs only 14319.

Close each server’s port 443 to everything but the load balancer. Docker publishes a port past a host firewall such as ufw, so the provider’s firewall is the one that closes it.

Write the servers’ directories#

On one host, create the home and the master key and download and check the release, as on one server. Then run init with the number of servers:

$ install -d -o 1654 -g 1654 -m 0700 /opt/weikio
$ head -c 32 /dev/urandom > /opt/weikio/master.key && chown 1654:1654 /opt/weikio/master.key && chmod 0400 /opt/weikio/master.key
$ docker run --rm -u 1654:1654 -v /opt/weikio:/opt/weikio -e WEIKIO_HOME=/opt/weikio -e Weikio__Keys__Dir=/opt/weikio/keys \
  -e WEIKIO_MASTER_KEY_FILE=/opt/weikio/master.key -e Weikio__Nats__Urls__0=nats://nats:4222 -e WEIKIO_BACKEND_IMAGE=$IMG \
  $IMG weikio-backend init --servers 3 --replicas 3 --public-base-url https://acme.example --public-leaf-url wss://nats.acme.example:443 --disk-budget 74GiB

init writes one directory per server, node-1 to node-3, under /opt/weikio/deploy/compose/cluster, and prints one line per server with its role and zone. It also prints the installation’s operator seed once. Save it and a copy of master.key outside the servers and apart from the backup archives.

• --servers is the number of servers, 3 or more.
• --replicas is how many of them keep a copy of the storage, 3 or 5, and 3 when you leave it out. A majority of them chooses the storage’s leader, which one server can’t form, so a hub on several servers keeps 3 copies or 5.
• --zones a,b,c lists the zones the servers stand in, given to the servers in turn, so no two copies of a piece of storage are kept in one zone. It needs at least as many zones as copies.

Start each server#

For each server, copy the home to it at the same path, keeping its owner. The DNS token and any certificate of your own that init took stay on node 1, so the copy leaves them out:

$ tar -C / --exclude=opt/weikio/import --exclude=opt/weikio/keys/dns-connection.json --exclude=opt/weikio/keys/hub-certificates.json --exclude-tag-all=agent.creds -cpf - opt/weikio | ssh <server> 'tar -C / -xpf -'

In each server’s own node directory, /opt/weikio/deploy/compose/cluster/node-<n>, set the lines of .env that give every server’s private address and the load balancer’s private address. They’re the same on every server:

WEIKIO_HUB_1_ADDRESS=10.0.0.11
WEIKIO_HUB_2_ADDRESS=10.0.0.12
WEIKIO_HUB_3_ADDRESS=10.0.0.13
WEIKIO_BALANCER_ADDRESSES=10.0.0.5

Start the message server on every server first:

$ docker compose up -d nats

The hub’s storage responds once a majority of the data servers run and have chosen their leader. Then start node 1’s stack and save the first platform-admin API key there, as on one server. Start every other server’s stack the same way. Their init prints no key, because the documents already exist.

$ docker compose up -d
$ docker compose ps -a                  # nats healthy; infra and init exited 0; backend, apim, splitter, terminator and backup up; on node 1 also the forwarder, the telemetry door and the three stores
$ docker compose logs init              # on node 1: the first platform-admin API key and the set-password link, shown once
$ docker compose rm -f init
$ docker compose run --rm infra weikio-backend doctor

Delete /opt/weikio/import on node 1 once the hub has started, because the platform keeps the DNS token and your certificates in its sealed store.

Set up the load balancer#

Add every server to the load balancer’s pool with these settings, which are the same for any number of servers:

• The pool has every server, each with equal weight, and no stickiness.
• The listener is TCP 443 only, passed through to port 443 of each server. A site links over WebSocket on 443, so the balancer needs no other port.
• The PROXY protocol is on, so each server’s terminator gets the caller’s address from its header. The terminator accepts the header only on connections from the addresses in WEIKIO_BALANCER_ADDRESSES, and the audit trail records each caller’s own address.
• The health check is HTTPS on port 443 of each server, GET /readyz/node, with the backend’s name as the server name and as the Host header, without verifying the certificate, every 5 s with a timeout of 4 s, 2xx as healthy.
• Out and back: a server leaves the pool after one failed check and comes back after one passed check.

/readyz/node returns 200 while the server’s backend, its gateway instance and its message server’s listener for site links are all ready, and 503 otherwise. A server whose gateway or message server stops leaves the pool within 5 seconds, and its callers and site links go to the other servers. A server told to stop, as upgrade.sh tells it, returns 503 at once and keeps passing new connections for 10 seconds, so it leaves the pool before it stops taking connections and no call is closed unanswered.

The check doesn’t verify the certificate on purpose. Until a name’s first certificate is issued, each server serves a temporary one. The certificate authority checks the name through the balancer, so a verifying check would keep every server out and the first certificate would never come. The hub checks the certificate each server serves itself, and the problem list says when a server serves a temporary or an old one.

Set up a check from outside as well, on https://acme.example/readyz/node through the balancer, as Health and problems describes.

Add and remove servers#

A hub grows by standard servers. On the host init first ran on, run init again from the backend image of the release the hub runs, with a larger --servers:

$ docker run --rm -u 1654:1654 -v /opt/weikio:/opt/weikio -e WEIKIO_HOME=/opt/weikio -e Weikio__Keys__Dir=/opt/weikio/keys \
  -e WEIKIO_MASTER_KEY_FILE=/opt/weikio/master.key -e Weikio__Nats__Urls__0=nats://nats:4222 -e WEIKIO_BACKEND_IMAGE=$IMG \
  $IMG weikio-backend init --servers 5

This writes node-4 and node-5 beside the others, each a standard server, and changes no file of the servers the hub has. Copy the keys directory and each new server’s directory to that server, set the address lines of its .env for every server, start it with docker compose up -d and add it to the balancer’s pool. The other servers learn its name through the data servers. Every server keeps four links to every other server, about 0.65 MB each, so a hub runs up to a few hundred servers.

To take a standard server out, remove it from the balancer’s pool, stop its stack with docker compose down, and mark it retired where init first ran:

$ docker run --rm -u 1654:1654 -v /opt/weikio:/opt/weikio -e WEIKIO_HOME=/opt/weikio -e Weikio__Keys__Dir=/opt/weikio/keys \
  -e WEIKIO_MASTER_KEY_FILE=/opt/weikio/master.key $IMG weikio-backend init --remove-node 5

A retired number is never given to another server. The data servers are chosen when the hub is installed and stay, and a backup restores onto as many new servers as the hub had.

A server cut off#

A server whose private network to the others is cut keeps running, and the load balancer still reaches it, but it can store nothing. One of two things happens, and you can’t tell which beforehand:

• The balancer takes the server out of its pool within about 15 seconds, because its backend returns 503 about 10 seconds after the cut. New calls and site links go to the other servers.
• The server stays in the pool. Calls that land on it read the hub’s records as they were when the cut began, and a change sent through it fails with 503 and a Retry-After header.

About a minute after the cut, in both cases, the problem list shows the server with the error hub-server-cut-off, and the Health page lists its message server as offline. The check from outside passes either way, so watch the problem list. Run doctor on each server without waiting for its message server:

$ docker compose run --rm --no-deps infra weikio-backend doctor

When one server is cut off, every other server’s servers row shows that one, such as on 2 of 3: weikio-hub-1 not responding. When the network between all of them is cut, each server shows only itself. To take one cut-off server out of the pool for certain, and close the connections it still has, stop its splitter in its stack directory:

$ docker compose stop splitter

Take out at most one server this way. Once its network is back and docker compose ps shows nats healthy, start it again with docker compose up -d. The server is back when doctor shows on 3 of 3.

A data server whose disk fills turns its storage off and keeps it off until it restarts, and the problem list shows it as message-storage-off a minute later. Free disk space on that server, then restart its message server while the others keep running:

$ docker compose restart nats

Upgrades and rollback shows the rolling upgrade, one server at a time, and Storage and its budget shows how to raise each data server’s budget.