Testing 2026.2.0 for 24 hours on three servers
Before 2026.2.0 was published, its release candidate ran for 24 hours under steady load on a hub of three servers, from 23 October at 10:16 UTC to 24 October at 10:21 UTC. During the run, three drills broke parts of the platform on purpose. This post describes the setup, the drills and the results.
The setup
The hub ran on three servers with 4 vCPUs and 8 GB of memory each. Every server kept a copy of the storage and ran a backend, a gateway and a message server, behind a TCP load balancer. Sites with agents ran 250 deployments, publishers sent 34 events a second through the channels, and a probe called an API through the gateway for the whole 24 hours. The load was about half of what the same hub had handled at its measured ceiling.
The run had to meet these bars: every probe call answers with 200, no deployment fails, no reader of a channel stops, no problem appears that a drill doesn't explain, and the stores keep exactly what the sites sent, counted for every hour.
The drills
| When | What the drill did | What happened |
|---|---|---|
| +4 hours | Stopped the log, metric and trace stores of one hub server for 15 minutes | No telemetry was lost, and the hourly counts of stored and sent records stayed equal |
| +6 hours | Paused the message server that led the storage for 30 seconds | Calls during the pause were answered with 200 or with 503 and a Retry-After header, never with 500, and every call after it with 200 |
| +8 hours | Cut one site off from the hub for 15 minutes | The integrations at the site kept running, and the site sent the telemetry it had kept with the original times when the link was back |
The results
Of the 243,468 probe calls, the only failures came during the 30-second pause the test caused on purpose, and the stores received every telemetry record the sites sent. The hub discarded no record, and the counts of stored and sent records were equal for every hour of the run, with nothing lost and nothing stored twice.
The run failed one bar as it was written: the lowest point of the backend's memory use on one hub server rose slowly through the day. The finding was rated medium, and its fix shipped in 2026.2.1 on 28 October 2026 and limits the memory arenas of the backend's runtime with MALLOC_ARENA_MAX=2.
The security test
Adafy's security team, which is separate from the development team, then tested the release candidate from the internet and found nothing rated critical or high. Of its 17 findings, 2 were rated medium, 10 low and 5 informational. Only port 443 answered, on one load balancer, every name had its own public certificate, and no request without a valid sign-in read or wrote anything. The medium finding in the gateway, which passed a caller's cookies on to the upstream, is fixed in 2026.2.1, released on 28 October 2026.
Running your own
The three-server shape of this test is the one the guide Install a hub on three or five servers sets up. On the same servers with local SSD the hub stored about 600 events a second, and on network-attached disks about 70 to 100, because the message server confirms every write only after the disk has it.