React to a run

When the morning’s invoices have landed, the invoice processing should start, without anyone polling the storage account. A transfer makes that possible because every run publishes its summary into the environment’s event stream, where any integration of the project can read it.

The run summary#

Each run publishes one event when it ends. It has the run’s ID, its start and end, the files, the bytes, the errors and the first error rclone wrote. You list them with the CLI:

$ weikio events list --env production --family mft --integration finance/supplier-invoices --since 24h --with-body

Read it in an integration#

An integration reads the transfer’s records with the platform’s weikio-event-source kamelet, with the family mft and the transfer’s name as the channel:

- route:
    from:
      uri: "kamelet:weikio-event-source?family=mft&channel=supplier-invoices"
      steps:
        - log: "Run of supplier-invoices ended: ${body}"
        - to: direct:process-batch

The integration receives each run summary as soon as the hub has stored it. Like every reader of stored events, it gets the summaries published since its first deploy, also those published while it was stopped, and a summary its route fails on is delivered again after the delays of its delivery settings. Delivery and CloudEvents explains the delivery, and Dead letters what happens after the last attempt.

An event for every file#

The summary is one small event per run, which is what most readers need and what stays small inside the environment’s event budget. A transfer whose files each need their own reaction sets emitPerFile: true and publishes both, the summary and one event per file:

apiVersion: weik.io/v1
kind: mft
metadata: { name: supplier-invoices, project: finance }
spec:
  schedule: "0 0 5 * * ?"
  source:      { remote: supplier-ftp,      path: /outgoing }
  destination: { remote: invoice-archive,   path: invoices/incoming }
  operation: move
  filters: ["*.pdf"]
  emitPerFile: true

Every file still has its own log line whether emitPerFile is on or off. The events are for integrations that react, and the log lines are the record of what moved.

One transfer, two agents#

A transfer that two agents match still runs once. Both agents start a copy, the copies elect a leader through a lock the hub keeps, and only the leader runs the schedule. When the leader’s agent goes away, the other copy takes over, so the morning’s run happens even when one site’s host is down. The readers of the run summaries see one summary per run either way.

One copy at a time shows how to see which copy leads.