Disaster recovery

A DR plan is a runbook plus a set of targets. The Flux-specific wrinkles: everything coordinates through PostgreSQL (so the database is the availability floor and the thing you replicate), the encryption key lives outside the database (a DB-only backup can’t decrypt secrets), and replay re-runs in-flight tasks (idempotency is part of your DR contract).

Failure-mode taxonomy

Six categories. Match the incident to one and follow its recovery path.

1. Worker crash. A single worker process dies. No DR needed beyond restart automation — the server reassigns the execution and the new worker replays the event log up to the last checkpoint.
2. Server crash. A single server replica dies. If you run multiple replicas behind a load balancer, traffic routes elsewhere, another replica’s reaper reclaims executions from workers that were attached to the dead replica, and the scheduler advisory lock releases automatically so a survivor picks up the next cycle. If you run a single server, scheduled work pauses (each missed schedule fires once on recovery); in-flight executions on workers continue, and checkpoints queue locally until the server is reachable again.
3. Database unavailability. Server stops accepting writes. Workers can’t checkpoint. Recovery is a Postgres failover or a restore. See Backups and restore.
4. Region failure. A full data center is out. Recovery requires a cross-region replica or a restore from off-site backup. The RTO depends on whether you’ve pre-provisioned the standby.
5. Data corruption. Rows are wrong because of a bug, a bad migration, or an accidental DELETE. Recovery is restore-from-backup (point-in-time with WAL archiving, otherwise the last daily snapshot).
6. Encryption key loss. The FLUX_SECURITY__ENCRYPTION__ENCRYPTION_KEY is gone with no copy. Secrets in the database are unrecoverable AES blobs. Mitigation is out-of-band key escrow: KMS, a sealed-secret manager, or a printout in a safe. There is no recovery without it.

RTO and RPO targets

Pick numbers, then engineer to them. Conservative defaults for a single-region install:

• RTO 1 hour. Provision infrastructure, restore from backup, restart everything, validate.
• RPO 24 hours. Daily backup default. Tighter targets need WAL archiving (Postgres) or per-hour snapshots (SQLite).

Cross-region: RTO depends on whether the standby is hot (replica streaming, server replicas warm) or cold (restore from snapshot). Hot standby gets you to single-digit minutes; cold restore is the single-region number plus DNS propagation. RPO depends on Postgres replication lag — async is typically sub-second; sync gives RPO≈0 at the cost of write latency.

Replication options

Three pieces to replicate: the database, the artifact store, and the encryption key.

• Postgres streaming replication. Async by default, optionally sync. Sync gives RPO≈0 at the cost of every commit waiting for the standby. For most Flux workloads, async is the right trade.
• Postgres logical replication. More flexible, slower to fail over, and not supported by Flux out of the box — Flux assumes a single writer.
• Artifact store replication. With the built-in LocalFileStorage, replicate the storage directory (rsync, S3 sync from a sidecar, or a replicated filesystem). The 0.56.0 release ships only InlineOutputStorage and LocalFileStorage — there is no built-in S3 storage backend, despite “s3” appearing as a docstring example in output_storage.py. For S3-native storage, ship your own OutputStorage subclass.
• Encryption key replication. Replicate the key into the DR region through your key manager. Without it, the DB restores but secrets stay opaque.

Hot standby topology

Two regions, primary and standby. Postgres replica streams from primary. Server replicas in the standby region stay cold (scaled to zero, or held out of the DNS cutover) and start against the promoted database at failover — Flux servers write on startup (migrations, heartbeats), so they can’t idle against a read-only replica. Workers in both regions, with standby workers idle until promotion. Encryption key replicated. Artifact storage replicated.

Scheduler behavior during failover

Flux’s scheduler runs inside the server process, but replicas coordinate through PostgreSQL: each scheduler cycle is guarded by a session-scoped pg_try_advisory_lock, so exactly one replica dispatches due schedules per cycle no matter how many servers point at the database. If the lock holder dies mid-cycle, its connection drops and PostgreSQL releases the lock automatically — a surviving replica takes the next cycle. Scheduler run state (next_run_at, last_run_at) is persisted per fire, so a schedule neither double-fires across replicas nor re-fires when a restarted server replays the cycle.

Two failover consequences worth planning for anyway:

• Both sides of a region failover must not run against diverged databases. The advisory lock protects replicas sharing one database; it cannot protect a split where the old primary keeps writing to the old database while the standby writes to the promoted one. Fence the old region’s database (or stop its servers) before promoting — this is standard Postgres-failover hygiene, not a Flux-specific mechanism.
• In-flight cycles are at-least-once at the boundary. A dispatch that committed just before the failover is a fire; workers’ idempotency contract covers the edge.

Within a single region, multi-replica servers are a supported availability topology, not a risk — see High availability.

Restore-from-backup procedure

To bring up the standby from scratch:

1. Provision infrastructure in the surviving region (database, server, workers).
2. Restore Postgres from the latest backup, or promote the replica snapshot.
3. Restore artifact storage (S3 restore, rsync from secondary, or remount the replicated volume).
4. Set FLUX_SECURITY__ENCRYPTION__ENCRYPTION_KEY from your key manager. With auth enabled, the server refuses to start without it (and without FLUX_SECURITY__EXECUTION_TOKEN_SECRET).
5. Set FLUX_WORKERS__BOOTSTRAP_TOKEN to the same value as the primary (otherwise workers can’t re-register).
6. Run flux db upgrade against the restored database if the restoring binaries are newer than the backup’s schema (migrations also run automatically on first connect, advisory-lock-guarded).
7. Start the server (one replica first, then scale out), then workers.
8. Validate GET /health returns {"status":"healthy","database":true} and GET /ready returns 200.
9. Run a synthetic canary workflow — a one-task no-op — and confirm it completes end-to-end.

Cut DNS or load-balancer traffic over only after the canary passes.

What survives, what doesn’t

Survives a restore from the most recent snapshot: every ExecutionEvent, artifact, schedule, and encrypted secret recorded before the snapshot (provided the key is available).

Does not survive: in-flight tasks that started after the snapshot’s checkpoint. On resume, Flux re-executes them by replaying the event log. Non-idempotent side effects (email, payment, external API call) happen twice. Idempotency is part of your DR contract.

Drill cadence

Quarterly minimum. Restore the most recent backup onto a non-production replica, run the canary, time the restore end-to-end, and update the runbook with whatever surprised you.

What can go wrong

Three failure modes worth pre-mortem-ing:

• Encryption key not in the DR region. The DB restores, the server starts, but every secret read returns a decrypt error. Replicate the key before you need it.
• The old region keeps writing after promotion. Advisory locks coordinate replicas on one database; a region split with two live databases is a Postgres split-brain, and schedules can fire on both sides. Fence the old database before promoting the standby.
• Workers in the DR region can’t reach the DR server. DNS, load balancers, and security groups aren’t always in failover scope. Test the full path — worker registration, SSE stream, checkpoint POST — not just /health. If the DR server runs multiple replicas, the DR load balancer needs the same sticky routing for /workers/{name}/connect as the primary.