The entity store

Many systems can’t say what changed since yesterday. They have no change timestamps, so an integration polls the whole list of customers or products and must work out itself which records are new or different. The entity store does that work. An integration writes each record it reads, the store skips writes that change nothing, and every real change becomes a change event that other routes react to.

Declare a record type#

A record type exists in an environment once its project declares it, which creates its store with a size limit and a number of revisions to keep:

$ weikio entitystore policy set finance/invoice --env production --max-bytes 512MiB --history 2 -y

Target: https://hub.example, project finance (argument), environment production (--env)
entity-store-policy production/finance/invoice created (maxBytes 512 MiB, history 2, revision 57)
production entity store budget: 512 MiB of 2 GiB allocated

Each environment has an entity store budget, which a platform administrator sets on the environment. A declaration that would go over it is refused, so a write never is. The admin UI’s Entity store page has a form for the same, and in Git it is an entity-store-policy document. A route that writes into a type its project hasn’t declared doesn’t start, and its error gives the command that declares it.

Write records#

The entitystore-sink kamelet writes a record of a type, with an ID the route gives it:

- route:
    from:
      uri: timer:poll?period=300000
      steps:
        - to: "{{erp-url}}/api/products"
        - split:
            jsonpath: "$.items[*]"
            steps:
              - setHeader: { name: entityId, simple: "${jsonpath($.sku)}" }
              - to: "kamelet:entitystore-sink?entityType=product"

The ID comes from the entityId header unless entityIdExpression says otherwise. The route never gives a project: the kamelet gets it from the deployment it runs in, so the same YAML works in every project. A write that changes nothing is skipped, and every real change is published as a change event.

React to changes#

The entitystore-source kamelet starts a route from the changes of a type:

- route:
    from:
      uri: "kamelet:entitystore-source?entityType=product&operations=created,updated"
      steps:
        - to: "{{shop-url}}/api/products"

operations is created,updated,deleted by default. The copies of a deployment share the changes, so a deployment on two agents handles each change once. Only the products that really changed reach the shop.

Read the records back#

$ weikio entities list finance/invoice --env production --id-prefix 2026-
$ weikio entities get finance/invoice INV-1001 --env production --revisions
$ weikio get entity-store-policies --env production

entities list pages through the IDs in order with their update time and size. entities get --revisions prints a record with the revisions the store still keeps, newest first, so you can see how it changed. get entity-store-policies lists each type with its limits, how many records it has and how much of the budget it uses. The admin UI’s Entity store page shows each type’s records and its Changes over the time you pick.

Who can reach it#

The store keeps one bucket per environment, project and record type, and the project’s ID is part of its name. An agent’s credential reaches only the projects the agent may run, so an integration can’t read another project’s records. Queries go through the platform’s API, with the same roles as everything else.

The entity store’s writes and changes have an event stream of their own in each environment, beside the channels’ stream, so a large import can’t push a channel’s backlog out.