Telemetry forwarding

Weik.io 2026.2

One telemetry-forwarding document sends the platform's logs, metrics, traces and audit records to Azure Monitor, Splunk or any system that accepts OTLP.

On this page

How forwarding works
Store the secrets
Start from an example
Apply it
Audit records
What can still lose a record
A receiver of your own

Every hub runs a telemetry forwarder: an OpenTelemetry collector that runs the configuration a platform administrator applies as one telemetry-forwarding document. You write the configuration in the collector’s own format and keep it in Git beside your other documents. The backends send the forwarder what that configuration selects of the log lines, metrics and traces the hub received. The forwarder reads the audit trail itself, so an audit record waits in the trail and on the forwarder’s disk until your system has accepted it.

How forwarding works#

The installation has one telemetry-forwarding document or none. Its spec.collector is a complete OpenTelemetry collector configuration, kept as you wrote it, and spec.audit.exporters lists the exporters that receive the audit records. Before the platform stores a revision, it checks the file with its own rules and with the collector the platform ships. A configuration the collector can’t start is refused with the collector’s own sentence, and the forwarder keeps running the revision it has.

A pipeline receives the platform’s telemetry when it lists the receiver otlp/weikio. Every log line, metric point and span reaches it stamped with the identity its sender’s credential proved, so a deployment of one environment can’t pass itself off as another’s. Each resource says what sent it:

Attribute	Value
`service.namespace`	`weikio`
`service.name`	`project/integration` for an integration, such as `finance/erp-invoice-export`, or `weikio-backend`, `weikio-apim`, `weikio-agent` and `weikio-forwarder` for the platform’s own processes
`deployment.environment.name`	the environment of an integration
`weikio.project.name`	the project of an integration
`weikio.api.name`	the API of a call through the gateway
`weikio.installation`	the installation, on everything it sends
`weikio.environment.uid`, `weikio.project.uid`, `weikio.integration.uid`	the IDs, which stay the same when a record is renamed

To send each environment’s data to its own place, route on deployment.environment.name. The hub’s own processes send no environment, so a filter on the environment spares weikio.ingest.sender == "hub" when you want the platform’s own lines too. Group an integration’s telemetry by weikio.integration.uid and weikio.environment.uid, because names are labels and a renamed integration keeps its IDs.

Store the secrets#

Store every credential the configuration needs as a platform secret, and write it in the file as ${secret:<name>}. The command asks for the value without echo:

Terminal

$ weikio platform secret set siem-token

The platform keeps the value sealed and sends it to the forwarder with each revision that uses it. A secret the document uses can’t be deleted until you remove it from the configuration and apply it again. Only platform administrators set the values and apply the document.

Start from an example#

Three configurations ship with the platform, in deploy/forwarding/examples/ of the Weik.io source. Each opens with a comment that says where every signal and the audit records go and which secrets to store, and each passes the platform’s checks as written.

A SIEM that accepts OTLP#

otlp-siem.yaml

# Telemetry forwarding to a SIEM that receives OTLP over HTTP.
#
#   logs     to the SIEM over OTLP/HTTP
#   metrics  to the SIEM over OTLP/HTTP
#   traces   to the SIEM over OTLP/HTTP
#   audit    to the SIEM over OTLP/HTTP, queued on the forwarder's disk until the SIEM accepts them
#
# Every signal waits on the forwarder's disk and is retried until the SIEM takes it. Write the SIEM's address below,
# then store the bearer token this file reads:
#
#   weikio platform secret set siem-token
#
# and apply it with 'weikio apply -f otlp-siem.yaml'.
apiVersion: weik.io/v1
kind: telemetry-forwarding
spec:
  audit:
    exporters: [otlp_http/siem]
  collector:
    extensions:
      file_storage/siem:
        directory: /var/lib/otelcol/file_storage
        create_directory: true
        fsync: true
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 55
        spike_limit_percentage: 15
    exporters:
      otlp_http/siem:
        endpoint: https://siem.example.com:4318
        headers:
          Authorization: "Bearer ${secret:siem-token}"
        sending_queue:
          storage: file_storage/siem
          sizer: bytes
          queue_size: 268435456
          batch:
            flush_timeout: 1s
            min_size: 65536
            max_size: 4194304
        retry_on_failure:
          max_elapsed_time: 0
    service:
      extensions: [file_storage/siem]
      pipelines:
        logs:
          receivers: [otlp/weikio]
          processors: [memory_limiter]
          exporters: [otlp_http/siem]
        metrics:
          receivers: [otlp/weikio]
          processors: [memory_limiter]
          exporters: [otlp_http/siem]
        traces:
          receivers: [otlp/weikio]
          processors: [memory_limiter]
          exporters: [otlp_http/siem]

Splunk HEC#

This one sends the log lines of production deployments and of the hub’s own services, and the audit records, to Splunk’s HTTP Event Collector:

splunk-hec.yaml

# Telemetry forwarding to Splunk through its HTTP Event Collector.
#
#   logs     to Splunk HEC: the logs of production deployments and of the hub's own services
#   metrics  not forwarded
#   traces   not forwarded
#   audit    to Splunk HEC, queued on the forwarder's disk until Splunk accepts them
#
# Log lines and audit records share one queue on the forwarder's disk, retried until Splunk takes it. Write the HEC
# address below, then store the HEC token this file reads:
#
#   weikio platform secret set splunk-hec-token
#
# and apply it with 'weikio apply -f splunk-hec.yaml'.
apiVersion: weik.io/v1
kind: telemetry-forwarding
spec:
  audit:
    exporters: [splunk_hec]
  collector:
    extensions:
      file_storage/splunk:
        directory: /var/lib/otelcol/file_storage
        create_directory: true
        fsync: true
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 55
        spike_limit_percentage: 15
      # Keeps the log lines of production deployments and of the hub; the platform writes both attributes from what the
      # sender's credential proves, so a deployment elsewhere cannot pass by claiming production. The environment's name
      # is a label: group a deployment's lines by weikio.environment.uid and weikio.integration.uid, which a rename leaves.
      filter/production:
        error_mode: ignore
        log_conditions:
          - resource.attributes["deployment.environment.name"] != "production" and resource.attributes["weikio.ingest.sender"] != "hub"
    exporters:
      splunk_hec:
        endpoint: https://splunk.example.com:8088/services/collector
        token: ${secret:splunk-hec-token}
        sending_queue:
          storage: file_storage/splunk
          sizer: bytes
          queue_size: 268435456
          batch:
            flush_timeout: 1s
            min_size: 65536
            max_size: 4194304
        retry_on_failure:
          max_elapsed_time: 0
    service:
      extensions: [file_storage/splunk]
      pipelines:
        logs:
          receivers: [otlp/weikio]
          processors: [memory_limiter, filter/production]
          exporters: [splunk_hec]

Azure Monitor#

This one sends metrics and traces to Azure Monitor through its OTLP ingestion and the audit records to a SIEM. Azure Monitor accepts the data from a service principal that has the Monitoring Metrics Publisher role on the data collection rule:

azure-monitor.yaml

# Telemetry forwarding to Azure Monitor, with the audit trail to a SIEM.
#
#   logs     not forwarded
#   metrics  to Azure Monitor through its OTLP ingestion, as deltas
#   traces   to Azure Monitor through its OTLP ingestion
#   audit    to a SIEM over OTLP/HTTP, queued on the forwarder's disk until the SIEM accepts them
#
# Azure Monitor accepts the data from a service principal that holds the Monitoring Metrics Publisher role on the data
# collection rule. Write its tenant and client ids, the two ingestion endpoints of the rule and the SIEM's address below,
# then store the two secrets this file reads:
#
#   weikio platform secret set azure-monitor-secret
#   weikio platform secret set siem-token
#
# and apply it with 'weikio apply -f azure-monitor.yaml'.
apiVersion: weik.io/v1
kind: telemetry-forwarding
spec:
  audit:
    exporters: [otlp_http/siem]
    processors: [attributes/audit]
  collector:
    extensions:
      azure_auth:
        service_principal:
          tenant_id: 00000000-0000-0000-0000-000000000000
          client_id: 00000000-0000-0000-0000-000000000000
          client_secret: ${secret:azure-monitor-secret}
        scopes:
          - https://monitor.azure.com/.default
      file_storage/siem:
        directory: /var/lib/otelcol/file_storage
        create_directory: true
        fsync: true
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 55
        spike_limit_percentage: 15
      # Azure Monitor takes metrics as deltas, and the platform sends cumulative ones.
      cumulative_to_delta: {}
      # Audit records reach the SIEM without the browser's user agent.
      attributes/audit:
        actions:
          - key: user_agent.original
            action: delete
    exporters:
      otlp_http/azure:
        traces_endpoint: https://example-dce.westeurope-1.ingest.monitor.azure.com/datacollectionRules/dcr-00000000000000000000000000000000/streams/Microsoft-OTLP-Traces/otlp/v1/traces
        metrics_endpoint: https://example-dce.westeurope-1.metrics.ingest.monitor.azure.com/datacollectionRules/dcr-00000000000000000000000000000000/streams/Custom-Metrics-Otel/otlp/v1/metrics
        auth:
          authenticator: azure_auth
        sending_queue:
          sizer: bytes
          queue_size: 67108864
          batch:
            flush_timeout: 200ms
            min_size: 1048576
            max_size: 8388608
      otlp_http/siem:
        endpoint: https://siem.example.com:4318
        headers:
          Authorization: "Bearer ${secret:siem-token}"
        sending_queue:
          storage: file_storage/siem
          sizer: bytes
          queue_size: 268435456
          batch:
            flush_timeout: 1s
            min_size: 65536
            max_size: 4194304
        retry_on_failure:
          max_elapsed_time: 0
    service:
      extensions: [azure_auth, file_storage/siem]
      pipelines:
        metrics:
          receivers: [otlp/weikio]
          processors: [memory_limiter, cumulative_to_delta]
          exporters: [otlp_http/azure]
        traces:
          receivers: [otlp/weikio]
          processors: [memory_limiter]
          exporters: [otlp_http/azure]

The collector’s azure_monitor exporter cuts each attribute value to 8,192 bytes, the length Application Insights keeps, and logs a warning that weikio forwarding status --lines shows.

Apply it#

Terminal

$ weikio apply -f otlp-siem.yaml

The apply prints the revision and the collector that checked it, such as revision 9, checked by otelcol-contrib 0.160.0, starting now. Then see what the forwarder sends:

Terminal

$ weikio forwarding status

The status shows the revision each forwarder runs, where each signal and the audit records go, and what each exporter sent and failed to send in the last hour. --lines adds the collector’s last warnings and errors, and --watch follows it. The admin UI shows the same under Settings › Operations › Telemetry forwarding. weikio forwarding delete stops forwarding: the forwarder stops its collector and the backends stop sending it copies.

When an apply makes a signal kept that no store at the hub keeps, the deployments start sending it from their next start, and the apply’s row says so. Nothing is restarted for it.

Audit records#

The forwarder reads every audit record from the trail through a place of its own in it. An audit exporter keeps the records on the forwarder’s disk until the destination accepts them: its sending_queue.storage uses a file_storage extension that service.extensions starts, and its retry_on_failure.max_elapsed_time is 0. The platform refuses an audit exporter without both. While a destination is away, its records fill its queue on disk and then wait in the trail, which keeps them for 400 days, up to 8 GiB, and they all follow when it returns.

Each record arrives as one log record with the event name weikio.audit.record, with attributes such as weikio.audit.action, weikio.audit.outcome, weikio.audit.actor.display, weikio.audit.target.name, weikio.audit.reason, weikio.audit.origin.commit and weikio.audit.diff, beside client.address, the caller’s address. Records arrive at least once and not always in order, so drop a second record with the same log.record.uid in your system and sort by the record’s time. Don’t drop repeats by weikio.audit.seq, because after a restore the trail goes on from the backup’s last record.

The first apply sends the records written from then on. weikio forwarding audit-from <time> sends older ones and first says how many, and weikio forwarding audit-from now skips what waits.

Azure Monitor’s own exporter keeps records in memory and gives up on a record after a few failed tries, so the Azure Monitor example sends the audit records to a SIEM with an exporter that queues them on disk. The platform’s own trail keeps every record either way, and weikio audit query shows it.

What can still lose a record#

A destination that answers that a record is invalid refuses it for good, and the collector discards it. The discarded record counts as failed and raises the problem forwarding-exporter-failing.
Renaming an audit exporter or its storage starts an empty queue, so what the old queue had isn’t sent, though it stays in the trail.
A host that stops without warning can lose the last records written to the disk queue unless the storage writes each one through to the disk, which the shipped examples do with fsync: true.

At a stop of the forwarder, the collector can log Exporting failed. Dropping data. with interrupted due to shutdown. The audit records that waited in the disk queue are still there when the forwarder starts again and go then.

A receiver of your own#

A receiver you add to spec.collector listens inside the hub, where the backend, the gateway and an integration running at the hub reach it, and nothing outside the hub does. Give such a receiver authentication.