A store is filling up

The row’s figure gives the numbers in a few words, such as 34 GiB of 40 GiB reserved.

What it means#

One of the hub’s stores is close to its limit: the message storage that has every record and event, the disk under it or under the log, metric or trace store, or the log store’s own disk cap.

What raises it#

Each backend node checks the stores from its own readings, so the message storage needs no metric store. The title says which store it is:

• Message storage is a warning once its budget has no room for another project of the default size, 1 GiB, and an error once its stores reserve the whole budget. Each store in it is a warning from 85 % of its own size and an error from 95 %, but a store that deletes its oldest messages to make room, such as an environment’s events, is never an error.
• The disk under a data server’s message store or under a telemetry store is a warning below a fifth free and an error below a tenth, and a data server whose budget is too large for its disk is a warning.
• The log store’s disk cap, 30 GiB on a hub by default, is a warning from 85 % and never an error, because a store at its cap deletes its oldest days and keeps taking new lines. Once the cap limits the log store to fewer than its 30 days, the title is The log store keeps fewer days than set.

What to check#

See the stores and the environments’ streams with their usage:

$ weikio platform health
$ weikio get streams --env <environment>

On a hub server, in the stack’s folder, run the checks, which add up the stores against the budget:

$ docker compose run --rm infra weikio-backend doctor

How to fix it#

For message storage, grow the disk under the message server and raise the budget in the stack’s folder:

$ sudo sh ./raise-budget.sh 80GiB

A budget is never lowered. You can also lower what the environments keep, with smaller event budgets in weikio env set. For a telemetry store, free or grow its disk.