Health and problems

Weik.io monitors the integrations you run and the platform that runs them. Every 15 seconds the backend raises a problem for anything that needs attention, with a few words of figures, the time it began and the command that shows more. The same problems show on Home under Needs attention, on the Observability Overview, on the page of the integration or API they concern and in weikio problems, so everyone who looks sees the same list.

Problems the platform raises#

For the integrations and APIs you run:

• an integration whose messages fail, at least 5 and 5 % of them in 15 minutes
• an integration that keeps restarting, with how many restarts in how long and “out of memory” when that was why
• an integration whose memory keeps growing day after day
• an integration that has stopped handling messages for much longer than its usual gaps
• an API failing on the server side, 5 % of its calls in 15 minutes, with the callers’ own mistakes left out
• a file transfer whose last run counted errors, with the first error line and the command that shows the whole run
• a deployment no agent runs, with the reason and what a platform administrator does about it

For a platform administrator, the platform also raises problems about itself:

• an agent that is offline, on an older build or low on memory or disk
• message storage or a telemetry store filling up
• telemetry the hub lost
• a hub server cut off from the others
• a certificate near its expiry
• no backup in twice the backup interval, or a backup copy that failed
• two-factor sign-in turned off for administrators

Each problem has its own page in the docs, such as A backup is overdue, with what raised it, the threshold and the commands that fix it. The Problems reference lists them all.

Every problem also writes one log line on the backend when it’s raised, when its severity changes and when it clears, and counts in the metrics weikio.problem.raised and weikio.problem.active. Every problem Home shows is therefore also in the log and metric stores, and it reaches your own monitoring through Telemetry forwarding.

List the problems#

$ weikio problems --env production

Each row has the severity, the kind, the environment and project, what it’s about, since when, a FIGURE column with the numbers in a few words, such as 42 of 310 failed in 15 min or 6 restarts in 40 min, out of memory, and the problem’s title. On a terminal the detail follows under each row: the problem’s sentence and, after Read it with:, the command that shows what the row is about, such as weikio status finance/erp-invoice-export --env production. Into a pipe or a file the detail is the last column, so a script gets one row per line. With nothing to report the command prints No problems you may read.

The Health page#

Settings › Operations › Health is for platform administrators. It shows every part of the platform with its state, version, figures and what to do: each backend node, each gateway instance, the message servers, the log, metric and trace stores, the forwarder and every agent. Its tabs cover the parts, a live view, storage, the gateway instances and the telemetry the hub received, and Platform logs has the lines the backend, the gateway instances and the agents write about themselves.

/settings/health

One rule judges every store and every disk: a part is to watch from 85 % of its budget or below a fifth free, and an error from 95 % or below a tenth free. A store that deletes its oldest records once it’s full, such as an environment’s events or the log store at its cap, is never an error. An agent’s server is to watch when more than 90 % of its memory is in use and an error above 95 %.

Health from the CLI#

weikio platform health asks the backend whether it’s alive and ready, and with a platform administrator’s key lists every part as the Health page does, with its status, version, uptime, CPU, memory and disk, followed by the storage lines:

$ weikio platform health

Server:     https://hub.example
Liveness:   ok
Readiness:  ok
Issuer:     https://hub.example

--watch draws it again every 30 seconds with the platform’s latest warnings and errors under it. The command exits 6 when the hub isn’t ready or a part is offline, so a script can act on it. weikio platform metrics hub shows the figures of message storage over the last hour, and weikio platform logs backend the backend’s own log lines.

Watch the hub from outside#

A hub that stops responding can’t say so itself. Set up a check from a place that shares neither the hub’s network nor its hosting, with a monitoring service or a server of your own:

• The check asks for GET https://acme.example/readyz/node on the backend’s name. It needs no sign-in, and it accepts callers from every address, whatever the hub’s allowed networks are.
• The good reply is 200 with the text ready, while the backend, the gateway and the message server’s listener for site links are all ready. Otherwise the hub returns 503 with one sentence for each part that isn’t ready.
• The check runs once a minute with a timeout of 10 seconds and alerts after two failed checks in a row, so one lost request alerts nobody and a hub that stopped responding is known within about two minutes.

Let the check verify the certificate, so it also fails on an expired certificate or on one the weikio CLI and the sites would refuse. Turn it on once the backend’s name has its first certificate, and pause it while you upgrade a hub on one server, because the API is away for the seconds the upgrade’s last line gives.

When the check fails, the reply’s sentences name the part that isn’t ready, docker compose ps -a shows which service is down, and docker compose run --rm infra weikio-backend doctor checks the rest. On a hub of several servers the check goes through the load balancer. A server cut off from the others passes it, so watch the problem list as well.

Prometheus pages#

Each process of the platform also serves its metrics in Prometheus format on its internal listener, for monitoring of your own: the backend at http://<backend>:8080/metrics inside the hub’s network, and the agent at http://127.0.0.1:6006/metrics inside the agent’s container at the site. The hub’s public names answer /metrics with 404, so these pages stay inside.