Memory grows day after day

The row’s figure gives the numbers in a few words, such as growing 38 MB a day.

What it means#

A copy of the deployment uses more memory every day, which usually means something in its routes keeps objects it no longer needs. Left alone, the process runs out of memory and restarts.

What raises it#

Each copy is judged over the latest 3 to 7 whole days since it last started. The first two hours after a start don’t count, so a cache that fills on its first day raises nothing. The problem is raised when the lowest memory of each day grows by 20 % or more over those days, or when the heap that stays in use after garbage collection is on course to reach its limit within 7 days. The detail says which of the two rules fired. The problem needs the hub’s metric store, which every installation has unless it was set up without one.

What to check#

Chart the memory over a week:

$ weikio metrics <project>/<integration> --env <environment> --name memory --since 7d

See which agents run the copies and when each started:

$ weikio describe deployment <project>/<integration> --env <environment>

How to fix it#

Look for what grows in the routes: an aggregation that never completes, an idempotent repository or cache without a size limit, or a collection a processor keeps adding to. Fix it and publish a new version. A restart frees the memory for a while and buys time:

$ weikio restart <project>/<integration> --env <environment>