An integration is failing

The row’s figure gives the numbers in a few words, such as 42 of 310 failed in 15 min.

What it means#

Messages fail in a deployment’s routes and no error handler of the route handles them. A message that fails here didn’t reach where the route sends it, so the work it carries, such as an invoice or an order, wasn’t done.

What raises it#

The hub checks the deployment’s message counts every 15 seconds. The problem is raised when at least 5 messages and at least 5 % of the deployment’s messages failed in the last 15 minutes, and it becomes an error when half of them failed. The detail gives the route with the most failures. A deployment that started less than 15 minutes ago is judged over the time since it was deployed, so a broken release shows up within the first minutes. The problem needs the hub’s metric store, which every installation has unless it was set up without one.

What to check#

See where the integration runs and in what state:

$ weikio status <project>/<integration> --env <environment>

List the errors of the last 15 minutes, grouped by exception and route:

$ weikio logs <project>/<integration> --env <environment> --since 15m --top-errors

Open the failed traces, which show the step that failed and every retry:

$ weikio traces <project>/<integration> --env <environment> --errors --since 15m

How to fix it#

When the cause is on the other side, such as an upstream that answers with errors or a server that refuses the connection, fix it there, and the problem clears once the share of failures falls below the threshold. When a new version broke the route, roll back to the version that ran before while you fix it:

$ weikio rollback <project>/<integration> --env <environment>

An error handler that catches a failure, retries it or sends it to a dead letter channel stops the failure from counting here, because the route then handles it.