Storage and its budget

A hub keeps two kinds of data on disk. The message server keeps the platform’s records, the events, the audit trail and the built versions of your integrations, within the disk budget you give init. Three stores beside it keep the log lines, metrics and traces, each within its own caps and for its own number of days. The Health page shows both, with what each budget reserves, what it has and how much room is left.

The disk budget#

--disk-budget caps the message server’s storage at a size of at most 80 % of the disk under it. The rest of the disk is for the message server’s own files and the margin the Health page watches, so the disk needs the budget and a quarter of it more. The message server reserves the full size of every store against the budget when it creates the store:

Each entity type a project declares in an environment reserves its bucket too, at the size its policy gives, 128 MiB when the policy gives none, and an environment’s entity types together stay within its entity-store budget. So on three data servers with --disk-budget 74GiB, three environments leave room for 26 projects of the default size (74 minus 10.2 minus 3 times 12.5 is 26.3), and 149 GiB has room for 101 projects beside three environments.

A platform administrator gives a project another store size with weikio project create or weikio project set and --store-bytes, and an environment other budgets with weikio env create or weikio env set and --events-bytes, --entity-events-bytes, --dead-letter-bytes and --idempotency-bytes. weikio platform health gives a line for each data server, such as:

data server 1   budget 74 GiB, 48 GiB reserved, 1.2 GiB stored, room for 26 more projects of 1 GiB, 71 GiB free of 93 GiB on the message store disk

Raise the budget#

Grow the disk first, by what the budget grows and a quarter of it more. Raising the --disk-budget 40GiB of a single server to 80 GiB grows the tested disk of 160 GB to about 214 GB. Then run the raise in the stack’s folder:

$ sudo sh ./raise-budget.sh 80GiB

Server 1 raised to 80 GiB. The hub's storage is 80 GiB.

The message server applies the new budget while it runs. A size above 80 % of the disk is refused with the largest budget the disk can take, and a budget is never lowered.

On a hub of several servers the storage follows the smallest budget among its data servers, so grow each data server’s disk and run the raise in each data server’s node directory. On the first servers the script says the hub waits for the others:

$ sudo sh ./raise-budget.sh 149GiB

Server 1 raised to 149 GiB. The hub's storage stays 74 GiB per server until servers 2 and 3 are raised.

On the last one it gives the new size:

Server 3 raised to 149 GiB. The hub's storage is 149 GiB per server.

The log, metric and trace stores#

The three stores run beside the backend, on node 1 of a hub of several servers. Each is bounded on its own:

A log line takes about 50 to 90 bytes of the log store’s disk, so 30 days at 100 lines a second take about 15 to 25 GB. At that size 30 GiB keeps 30 days up to about 140 to 250 lines a second. Integrations that log whole message bodies write far more. A call through the gateway into an integration takes about 600 bytes of the trace store’s disk, so 20 GiB keeps 7 days up to about 59 such calls a second. With their default caps the stores take up to about 75 GB, and the metric store’s 90 days take about 5 GB for 50 integrations and 20 GB for 250.

Once the log or the trace store reaches its cap it deletes the oldest days first, and the Health page and weikio platform health say how many days it keeps. To change a cap, set its line in .env, such as WEIKIO_LOGS_MAX_DISK_SPACE_USAGE_BYTES=64424509440 for 60 GiB or 0 for no cap, and run docker compose up -d. Past the series limits the metric store discards the samples of new series, and doctor warns.

Set how long records stay#

The log store keeps log lines for 30 days. How long the metric and trace stores keep their records is a line of the stack’s .env for each, a whole number followed by h, d, w or y:

WEIKIO_METRICS_RETENTION=90d
WEIKIO_TRACES_RETENTION=14d

Run docker compose up -d after a change, so the store and the backend both take the new value. On a hub of several servers, set the same lines in the .env of every server, because the backend of each server uses them to know what node 1’s stores keep.

Memory of the stores#

Each store has a memory limit, so a heavy read, such as a week of traces on a busy hub, never uses the memory the hub’s own services run in. init sizes the limits from the server’s memory: a quarter of it for WEIKIO_TRACES_MEMORY_LIMIT and an eighth for each of WEIKIO_LOGS_MEMORY_LIMIT and WEIKIO_METRICS_MEMORY_LIMIT, at least 512m each. A read that would need more is refused, and the Traces page and weikio traces say to narrow its time range. Raise a limit in .env, such as WEIKIO_TRACES_MEMORY_LIMIT=4g, and run docker compose up -d.

Telemetry waiting at the hub#

While a store doesn’t take what the hub is sent, such as while it restarts, the backend keeps the log lines, metrics and spans on its data volume and writes them to the store as soon as it accepts them, with the times they arrived. It keeps up to 8 GiB of them and never so much that less than a tenth of the disk is free, so leave 8 GiB on the disk for it beyond that tenth. Past that it refuses new telemetry and the problem telemetry-waiting turns to an error. doctor says what waits for each store.

The three store volumes aren’t in the backup archives, because what they keep is telemetry and not the installation’s state. A restored hub on a new host starts them empty.

Docker’s own logs#

Docker keeps the output of each container apart from the stores, within the limits of /etc/docker/daemon.json, at most three files of 50 MB per container as Install a hub on one server sets them, and the stores’ retention doesn’t change that. An integration whose own Log4j 2 or Logback configuration writes a log file keeps writing it. A file at a relative path lands in the deployment’s run folder at the site, which the agent deletes when the deployment leaves the site, and a file at another path stays until you remove it.