Logs, metrics and traces

Every Camel integration you deploy sends three things to the hub: its log lines, the traces of the messages it handles and its metrics. You don’t add a Java agent or a library for it. Each line, span and figure carries the integration, version, project, environment, agent and Camel version, so you filter by any of them. The hub stamps the project, environment, integration and agent from the sender’s own credential, so code in one integration can’t file numbers under another.

What every integration sends#

• Log lines come through Camel’s own log4j2 logging, which is untouched, so log: steps and logger.info work as they always have. An integration’s own log4j2.properties still decides what its console shows while every line also reaches the hub, and lines written through java.util.logging or System.Logger arrive too.
• Traces follow one message’s path through the integration: the route, each endpoint it called, the kamelets and a hop into another integration through the platform’s events. A call through the API gateway is traced into the integration behind it.
• Metrics include Camel’s counts and timings per route, the JVM’s memory and garbage collection, and your own counters, written the way the Camel documentation writes them. micrometer:counter:invoices.processed and opentelemetry-metrics:counter:invoices.rejected both count, as Custom metrics shows.

The agent also measures each deployment from outside the JVM every 30 seconds: messages handled and failed, in flight, the last message, memory, CPU, threads and restarts. When a process exits without the agent asking, the agent writes one line into the deployment’s own logs, because a JVM’s own last words can be lost when it dies:

The process of deployment 'production/hr/payroll-sync (01k5xb7q2d9m4r8t6w3y0hcnfe)' exited with code 3; the agent restarts it in 10 s (restart 2 in 7 min).

Where it’s kept#

Three stores run at the hub beside the backend: VictoriaLogs for log lines, VictoriaMetrics for metrics and VictoriaTraces for traces. Only the backend reaches them. Log lines stay 30 days, metrics 90 days and traces 7 days, the log lines and traces within their store’s disk cap. WEIKIO_METRICS_RETENTION and WEIKIO_TRACES_RETENTION in the hub’s .env set other times for metrics and traces, as Storage and its budget shows.

The admin UI and the CLI never talk to a store. They ask the backend, which adds your permissions to every read, so you see the logs, metrics and traces of the projects and environments your grant covers. A grant narrowed to some integrations shows only what those integrations sent, and a trace that crosses into a project you can’t read shows the steps you may see and how many it hides.

Logs#

$ weikio logs finance/erp-invoice-export --env production -f

-f follows new lines until Ctrl-C. Without it the command shows a window, and the filters narrow it:

$ weikio logs finance/erp-invoice-export --env production --since 15m --level warn --limit 2

14:31:02.118 UTC WARN  fetch-invoices  agent-fi-02  HTTP 429 from https://erp.example.internal/api/invoices/pending, retrying in 30 s
14:31:32.204 UTC WARN  fetch-invoices  agent-fi-02  HTTP 429 from https://erp.example.internal/api/invoices/pending, retrying in 60 s

To read older lines, run:
  weikio logs finance/erp-invoice-export --env production --since 15m --level warn --limit 2 --before GNf5pJG7hYAAyvg9zpCA

Each line has the time, the level, the route that wrote it, the agent and the message. --histogram counts the lines per interval and level, which shows when warnings began:

$ weikio logs finance/erp-invoice-export --env production --since 6h --histogram --interval 1h

HOUR        INFO   WARN   ERROR
09:00 UTC   612    0      0
10:00 UTC   598    0      0
11:00 UTC   601    0      0
12:00 UTC   604    2      0
13:00 UTC   590    14     1
14:00 UTC   211    38     0

--top-errors lists the most frequent errors, --correlation-id the lines of one message or one transfer run, and --query searches the text. The agent’s own lines are in weikio agent logs <agent>, and the hub’s in weikio platform logs backend.

Metrics#

$ weikio metrics finance/erp-invoice-export --env production

The command prints the integration’s numbers over the window, one per line: messages, failed with its share, processing time typical and for the slowest 5 %, memory against its memory setting, restarts, the last message and failure, in flight, CPU and threads. Under them come your own counters with the route that counts each, then the routes, the failing first and then the busiest. --since 24h widens the window, and --name messages --by agent draws one metric split by agent.

The same command shows an API’s numbers with --api finance/invoices@v1, and weikio metrics --env production prints the Observability › Metrics tables of the whole environment, the most failed first.

/integrations/finance/erp-invoice-export/metrics

Traces#

$ weikio traces finance/erp-invoice-export --env production --errors

The list has one row per message, newest first, with when it started, how long it took, its status, where it came from, what it went through and, for a failed trace, where it failed and why, such as http erp.example/api/invoices: Read timed out. Open one trace by its id, by a traceparent a caller quotes or by a message id:

$ weikio trace <trace-id|traceparent|message-id>

The trace prints as a waterfall of its steps, each under the step it ran in, and a failed trace ends with Failed at: and the step’s error. The command then gives the weikio logs ... --correlation-id line that shows the trace’s log lines. Every response of the API gateway carries a traceresponse header, so a caller who quotes it is found by pasting it into Traces.

/observability/traces

In the admin UI#

Observability has four pages for what you run: the Overview, Metrics, Traces and Logs. The Overview says what runs in the environment now, such as 15 integrations and 5 APIs run in production, lists what needs attention, and shows four headline numbers with a chart and a list, each linking to its full page.

Each integration’s page has its own Observability group. Metrics opens on six numbers: messages, failed, processing time, memory, restarts and the last message. Under them are two charts, messages over time with the moments a version was deployed and failures began, and memory per agent over up to 90 days. Traces lists the integration’s messages and shows retries as numbered attempts. Logs follows the live tail, and a line that belongs to a trace opens it.

Keep fewer traces#

A busy installation has two settings that keep fewer traces and change nothing else. WEIKIO_APIM_TRACE_SAMPLE_RATIO in the hub’s .env, from 0 to 1, keeps that share of the calls through the gateway. The integration behind a call follows the gateway’s choice, so a trace is whole or absent. jvm.traceSampling in an integration’s integration.yaml keeps a share of its own traces. jvm.telemetry: off turns an integration’s traces and metrics off and keeps its log lines. A change in integration.yaml takes effect at the deployment’s next start, and a change in .env once docker compose up -d has run.

When a site is cut off#

A site keeps its integrations’ log lines, metrics and spans on its own disk while it can’t reach the hub, and sends them with their original times later. A running weikio logs -f prints such lines within about a minute of their arrival, marked late after the level, and lines that arrived before the follow started are in a read of their window with --from and --to.