Managed file transfer
A transfer moves files on a schedule between any two places rclone can reach, such as a supplier's SFTP server and an Azure storage account, or an FTP server and a folder at your plant. You write it as a short YAML file or create it in the admin UI, and each file it moves is logged with the transfer's version, environment and agent. Every run ends with a summary of the files, bytes and errors, and other integrations can react to that summary as an event.
apiVersion: weik.io/v1
kind: mft
metadata: { name: supplier-invoices, project: finance }
spec:
schedule: "0 0 5 * * ?" # every day at 05:00 UTC
source:
remote: supplier-ftp
path: /outgoing
destination:
remote: invoice-archive
path: invoices/incoming
operation: move
filters: ["*.pdf"]
retries: 3
retriesSleep: 30sA transfer in YAML
The transfer finance/supplier-invoices moves the supplier's invoice PDFs into the archive every morning at 05:00 UTC. Its two ends are variables that contain rclone remote definitions, which lets devtest point at a local folder while production points at the supplier's FTP server. Passwords stay out of Git as secrets, set with weikio secret set.
apiVersion: weik.io/v1
kind: variable
metadata: { name: supplier-ftp, project: finance }
spec:
value: |
type = ftp
host = ftp.supplier.example
user = weikio
pass = {{supplier-ftp-password}}$ weikio apply -f ./finance/
Target: https://hub.example, project finance (from finance/project.yaml)
variable finance/supplier-ftp created
variable finance/invoice-archive created
mft finance/supplier-invoices created (version 0.1.0, built in 14.2 s)
Nothing new is running yet. To run it:
weikio deploy -f ./finance/ --env devtest
$ weikio deploy finance/supplier-invoices --env production
Target: https://hub.example, project finance (context), environment production (--env)
This will deploy 'finance/supplier-invoices' 0.1.0 to production environment 'production'.
Continue? [y/N] y
Deploying finance/supplier-invoices 0.1.0 (latest) to production...
deployment production/finance/supplier-invoices created (version 0.1.0)
Agents: 1 of 1 running after 3.6 sWhat a transfer does
Any storage at either end
SFTP, FTP, SMB, S3, Azure Blob, Google Cloud Storage, WebDAV and local folders work as a source or a destination, with their credentials kept as secrets. rclone is part of the agent image, and a site installs nothing for it.
Docs: Remotes and credentialsOperations and filters
A transfer copies, moves or syncs files, or runs another rclone operation such as check. Patterns such as *.pdf choose the files that move, and retries repeat a failed attempt within the same run.
Different systems per environment
Devtest can use a local folder while production uses the supplier's FTP server. Promotion moves the same version to production, and each environment resolves the remote to its own system.
Docs: To productionSchedules
Schedules are cron expressions in UTC, and the admin UI shows each one in words beside its expression.
Docs: Your first transferA record of every file
Each file a run moves is a log line with the transfer, version, environment and agent, and Transferred files in the admin UI lists them. Every change to a transfer and every deploy writes an audit record with who made it and what changed, and the audit log keeps them for 400 days.
Docs: Runs and transferred filesRuns as events
Each run publishes a summary with its files, bytes and errors. An integration that starts from kamelet:weikio-event-source?family=mft&channel=supplier-invoices gets the summary when the batch has landed and processes the files from there.
When a run fails
A run that ends with errors counts as failed and raises a problem in the admin UI. The problem links to the run and gives the weikio logs command that shows its lines.
Transfers on two agents
When two agents match the deployment, the transfer still runs once, and the second agent takes over when the first one goes away.
Docs: React to a run
One run, start to end
Every morning the agent at your site moves the PDF files from the supplier's FTP server into the invoice archive in Azure Blob storage. The run's summary goes into the channel supplier-invoices as an event, and an integration that starts from that channel posts the invoices to the ERP.
A transfer runs on the agent at your site. Every morning at 05:00 UTC it moves the PDF files in the outgoing folder of the supplier's FTP server to the invoices/incoming folder of the invoice archive in Azure Blob storage, with retries, using rclone inside the agent. After each run the run summary goes to the channel supplier-invoices as an event, and an integration receives it and posts the invoices to the ERP's REST interface.
Every file on record
A run's page lists the files it moved, and Transferred files lists every file your transfers have moved. On the command line, weikio logs prints one line per file and the run's totals.
$ weikio logs finance/supplier-invoices --env production --since 24h
05:00:01.412 UTC INFO mft-log agent-fi-01 {"level":"info","msg":"Copied (new)","object":"invoice-1043.pdf","size":48213}
05:00:01.415 UTC INFO mft-log agent-fi-01 {"level":"info","msg":"Deleted","object":"invoice-1043.pdf"}
05:00:01.630 UTC INFO mft-log agent-fi-01 {"level":"info","msg":"Copied (new)","object":"invoice-1044.pdf","size":51870}
05:00:01.633 UTC INFO mft-log agent-fi-01 {"level":"info","msg":"Deleted","object":"invoice-1044.pdf"}
05:00:01.702 UTC INFO mft-log agent-fi-01 {"level":"info","msg":"Transferred: 2 / 2, 100%","stats":{"bytes":100083,"errors":0,"transfers":2}}Partners need an SFTP server. Your partners can upload to an SFTP server in Weik.io Cloud, hosted in Helsinki, and a transfer picks up their files through the remote that comes with the server. SFTP servers in Weik.io Cloud
Managed file transfer track
Five short videos take one transfer from its first file to production, each with the commands to follow along.
Questions
Which storage systems work?
Every storage system rclone supports works at either end, including SFTP, FTP, SMB, S3, Azure Blob and Google Cloud Storage.
Does the site need rclone installed?
It doesn't, because rclone is part of the agent image that runs at the site.
How do I see which files a run moved?
Open the run in the admin UI, which lists its files, or run weikio logs for the transfer, which prints one line for each file.
What time zone do schedules use?
Schedules use UTC, and the admin UI shows each schedule in words beside its cron expression.
Can partners push files to me?
Partners can upload to an SFTP server in Weik.io Cloud or to one of your own, and a transfer picks up the files from there.
Is Weik.io open source?
Weik.io is open source under the Apache License 2.0 and runs on your own servers, in your cloud or in Weik.io Cloud.
Read the transfer docs
Your first transfer needs one file and two commands.