Remotes and credentials

A remote is one end of a transfer: an FTP or SFTP server, an S3 bucket, an Azure storage account, a folder on the agent’s host, or anything else rclone has a backend for. The transfer refers to it by name, or carries its definition inline.

A remote in a variable#

A variable whose value is the remote’s rclone definition can be shared by several transfers, and each environment can give it a value of its own:

apiVersion: weik.io/v1
kind: variable
metadata: { name: supplier-sftp, project: finance }
spec:
  value: |
    type = sftp
    host = sftp.supplier.example
    user = weikio
    pass = {{supplier-sftp-password}}

The keys are rclone’s own, as rclone config writes them for that backend, so every rclone backend works the same way. A transfer uses the remote by the variable’s name:

source: { remote: supplier-sftp, path: /outgoing }

Credentials#

A password or a key inside a remote is a {{placeholder}} for a secret, so the rest of the definition stays readable in Git and reviewable in a pull request:

$ weikio secret set finance/supplier-sftp-password --env production

At deploy, the agent unseals the secrets and writes the remote into the deployment’s own rclone.conf, which only the agent’s user can read, with the passwords obscured as rclone expects. A deploy whose remote needs a secret with no value in the environment fails at once and says which secret is missing.

When the server’s password changes, set the secret again. The transfer gets the new value at its next start, and --restart restarts it at once:

$ weikio secret set finance/supplier-sftp-password --env production --restart

Everything in one file#

Three documents are worth having when several transfers share a server, or when devtest and production use different servers. A one-off transfer doesn’t need them, and there are two shorter ways to write it.

apply accepts multi-document YAML, so the two variables and the transfer can sit in one supplier-invoices.yaml, separated by ---. Or write the remotes inline, with remote as the rclone settings themselves:

apiVersion: weik.io/v1
kind: mft
metadata: { name: supplier-invoices, project: finance }
spec:
  schedule: "0 0 5 * * ?"
  source:
    remote: { type: ftp, host: ftp.supplier.example, user: weikio, pass: "{{supplier-ftp-password}}" }
    path: /outgoing
  destination:
    remote: { type: azureblob, account: financearchive, key: "{{invoice-archive-key}}" }
    path: invoices/incoming
  operation: move
  filters: ["*.pdf"]

Here too the credentials are placeholders for secrets, and the passwords stay out of Git.

Inline or named#

An inline remote is the same in every environment, and changing it is a new version of the transfer. A placeholder such as host: "{{supplier-ftp-host}}" gives each environment its own host, but the kind of remote stays fixed in the version, so devtest can’t be a local folder while production is an FTP server.

Use inline remotes for a transfer that lives in one environment. Keep the remotes in variables as soon as the transfer is promoted, or a second transfer needs the same server, as To production shows.

How the admin UI shows a remote#

The transfer’s page shows each end in words. An inline remote shows as its type and place, such as SFTP sftp.supplier.example:/outgoing, Azure Blob invoices/incoming or local folder /var/lib/weikio/mft/outgoing, and a host that comes from a value says so, as in SFTP /outgoing (host set by supplier-sftp-host). A remote kept in a variable or a secret shows as its name and path, such as bank-sftp:/outgoing.