Backups and restore

Backing up Flux's event log, artifacts, and encryption key — restore procedure, drill cadence, and what's NOT recoverable from a snapshot.

A Flux backup has three planes. Miss any one and your restore fails — silently in the case of secrets, loudly in the case of artifacts that the event log still references.

What to back up

Plane A: event log and catalog

This is your database. SQLite (sqlite:///.flux/flux.db by default) for dev, Postgres for prod. Everything Flux remembers lives here: workflow source, every state transition, schedules, principals, API keys, encrypted secrets, and worker registration data.

The schema is Alembic-managed (flux/migrations/); Flux migrates the database to the latest revision automatically on first connect, and flux db current shows the revision a backup was taken at. The tables:

executions, execution_events — the event log; the source of truth for replay
workflows — registered workflow source (base64-encoded), versions, metadata
schedules — cron/interval/once schedules and their run history
secrets, configs — key/value store; secrets.value is AES-GCM encrypted
agents — AI agent definitions
workers, worker_runtimes, worker_resources, worker_resources_gpus, worker_packages — worker registration snapshot
services — workflow-as-service exposures
roles, api_keys, principals, principal_roles — auth/RBAC
alembic_version — the schema revision marker; back it up with everything else so a restored database migrates forward correctly

If you back up the database, you back up all of this in one consistent snapshot. Take a fresh snapshot before every Flux upgrade — migrations modify schema in place, and the backup is the only rollback path (see Upgrades and migrations).

Plane B: output storage / artifacts

Depends on which OutputStorage you’ve configured. The default is InlineOutputStorage — task outputs are pickled into the execution_events.value column. If you’re on the default, Plane A backs up your artifacts too. No separate artifact store exists.

If you’ve switched to LocalFileStorage, outputs live as files under <home>/<local_storage_path> — .flux/.data/ by default. Flux 0.56.0 ships only InlineOutputStorage (the default) and LocalFileStorage. S3 storage requires shipping your own OutputStorage subclass. If you’ve added one, back up its bucket separately.

Check OutputStorageReference.storage_type on a recent execution if you’re not sure: inline means Plane A is sufficient; local_file (or a custom storage_type you’ve implemented yourself) means you need a separate plane.

Plane C: secrets encryption key

FLUX_SECURITY__ENCRYPTION__ENCRYPTION_KEY — the value that derives the AES-GCM key for the secrets table. The same key also HMAC-signs the runtime pickle columns — execution input/output, event values, schedule input — so a database restored under a different key fails integrity checks on that data, not just on secrets. Back the key up separately from the database. Compromising Plane A and Plane C together means your secrets are plaintext to anyone holding both.

Out-of-band channels for this: a cloud KMS, a sealed envelope, a corporate password manager. Not a file next to the dump.

What NOT to back up

Worker process state, in-flight memory, SSE connections, the module-source cache. Workers re-register on restart, claim work fresh, and recompile workflow modules from the catalog. None of it is durable, none of it needs to be.

Postgres backup

Two paths depending on volume and RPO:

Logical (low-volume installs):

pg_dump --format=custom --file=flux-$(date +%F).dump $DATABASE_URL

Restore with pg_restore --clean --if-exists --dbname=$DATABASE_URL flux-2026-05-14.dump. Simple, slow on large stores, gives you a point-in-time snapshot only as fresh as the last dump.

Physical (high-volume or tight RPO): enable WAL archiving and take a periodic pg_basebackup. This gives you point-in-time recovery (PITR) to any moment between the base backup and the latest archived WAL segment. Required if your RPO is measured in minutes.

SQLite backup

The server keeps SQLite in WAL mode (set via PRAGMA journal_mode=WAL in SQLiteRepository). Do not copy flux.db while the server is running — you’ll get a database without its WAL tail. Use the online backup API instead:

sqlite3 .flux/flux.db ".backup '/backups/flux-$(date +%F).db'"

This handles the WAL correctly and produces a consistent file.

Artifact backup

For LocalFileStorage, rsync the configured directory:

rsync -a --delete .flux/.data/ /backups/flux-data/

Run it after the DB snapshot — if a new execution writes an artifact between the two backups, you want the file present, not missing.

For custom S3-backed storage (not shipped by Flux 0.56.0 — only relevant if you’ve subclassed OutputStorage yourself), lean on the platform: enable versioning on the bucket and cross-region replication. Flux writes objects under deterministic reference IDs, so versioning gives you point-in-time semantics for free.

Restore procedure

Order matters. Restore inconsistencies (DB references an artifact you haven’t restored yet, or a key you don’t have) surface as decoding errors at runtime, not at startup.

Stop the server and all workers. No new writes during restore.
Restore the database to a known consistent state (the latest dump, or a PITR target).
Restore artifacts (if Plane B applies). DB references must resolve.
Restore the encryption key into the environment (FLUX_SECURITY__ENCRYPTION__ENCRYPTION_KEY) or config. It must be the original key: without it every read of the secrets table raises, and the HMAC-signed pickle columns (execution values, event payloads) fail integrity checks.
Start the server. It migrates the restored database to the current schema revision automatically if the backup predates the running Flux version. Hit /ready and confirm it returns 200.
Start workers. Watch them register and pick up the SSE stream.

What is NOT recoverable

Executions that were in-flight at the moment the backup ran. The event log is consistent up to the snapshot, but a task that was mid-execution when the dump completed has no resume point — the worker that owned it is gone, its in-memory state is gone.

The replay model salvages most of this: tasks that completed (TASK_COMPLETED event present) before the snapshot replay deterministically on resume; tasks that were running (TASK_STARTED without a matching TASK_COMPLETED) re-run from the start. That’s safe for idempotent tasks and unsafe for tasks with external side effects. If your workflow writes to a non-idempotent system, audit recently-resumed executions after a restore.

Restore drills

Monthly, against a non-prod environment. The drill is not the dump — it’s the restore. Test:

Database restores cleanly into an empty Postgres / fresh SQLite file
An execution that referenced a file or S3 artifact can read it after restore
A workflow that uses a secret runs without raising ValueError: Failed to decrypt value
Workers re-register against the restored server

A backup that has never been restored is a hypothesis, not a backup.

What can go wrong

Encryption key lost. Secrets are unrecoverable; the secrets column is AES-GCM with a PBKDF2-derived key and no escape hatch. The HMAC-signed execution payloads become unreadable too. Mitigation: out-of-band key escrow with at least two independent custodians.
Backup-restore lag exceeds RPO. Nightly logical dumps give you 24-hour RPO at worst. Tighten it with WAL archiving (Postgres) or more frequent .backup runs (SQLite).
Plane B drift. DB and artifact store are backed up at different times, so the DB references files that don’t exist in the artifact snapshot. Mitigation: snapshot DB first, then rsync; or pin both to a quiet window.