Production checklist

This is the gate-check list. If you can’t say yes to every item below, the deployment is not production-ready. Each section links into Operate for the depth-of-detail walkthrough.

Storage backend

• PostgreSQL, not SQLite. SQLite is single-node only — one server plus one colocated worker, or inline workflow.run(). SELECT … FOR UPDATE SKIP LOCKED claim safety and the advisory locks that make multi-node/multi-replica safe are silently unenforced on SQLite, and the server logs a warning when a second worker registers against it. PostgreSQL 14+ with the psycopg v3 driver (pip install 'flux-core[postgresql]').
• FLUX_DATABASE_URL set, not the default. The default sqlite:///.flux/flux.db will quietly work in production — that’s the trap. An explicit env var prevents the “we forgot to set it” outage.
• Pool and executor sized together. Per-replica defaults: database_pool_size = 20, database_max_overflow = 20, database_executor_threads = 16. Keep executor threads ≤ pool size so DB threads never block waiting for a connection, and give PostgreSQL max_connections ≥ replicas × (pool_size + max_overflow) + LISTEN connections + headroom. Watch pg_stat_activity under load.
• flux db upgrade in the deploy pipeline. The schema is Alembic-managed and auto-migrates on connect (advisory-lock-guarded), but running the migration explicitly before rolling replicas keeps deploys deterministic. See Upgrades and migrations.
• Retention enabled. [flux.retention] enabled defaults to false, and every task is a persisted event row — execution history grows without bound otherwise. Set FLUX_RETENTION__ENABLED=true and a retention_days that matches your audit needs. See Retention.
• Automated backups. Daily full + WAL archiving for point-in-time recovery. Most managed Postgres offerings do this by default — verify it’s on.

See Storage backends.

Authentication

• At least one auth provider enabled. oidc.enabled = true or api_keys.enabled = true. With auth off, every request resolves to the built-in admin principal.
• execution_token_secret set. With auth enabled, the server refuses to start unless both FLUX_SECURITY__EXECUTION_TOKEN_SECRET and FLUX_SECURITY__ENCRYPTION__ENCRYPTION_KEY are set — a deliberate fail-fast so the misconfiguration surfaces at deploy time instead of mid-traffic. Generate 32+ random bytes for each.
• OIDC for human users, API keys for service accounts. Mixing both is supported; pick the right tool per principal type. Worker API keys are minted with a TTL (worker_key_ttl, default 7 days) and self-rotate: on the first 401 after expiry the worker re-registers and gets a fresh key.
• No long-lived admin API keys for humans. If your CLI users have personal API keys, scope them to the smallest role that does the job and set an expiration. Human admins should go through OIDC.

See Authentication overview.

Bootstrap token

• Generated explicitly, not auto-generated. Run python -c 'import secrets; print(secrets.token_hex(32))' once and store the output in your secret manager.
• Distributed to workers via env var. FLUX_WORKERS__BOOTSTRAP_TOKEN injected from the secret store; never baked into images or checked into VCS.
• Identical across server and every worker. A mismatch produces silent 401 Unauthorized on POST /workers/register and workers that never come online.
• Rotation runbook exists. flux server bootstrap-token --rotate writes a fresh token to the persisted file, but the running server keeps its in-memory copy until restart (and an env/config override always wins over the file). Plan the overlap window: rotate, distribute to workers, restart the server, then roll the workers.
• Registration rate limit fits your fleet. POST /workers/register is limited to 30/minute per client IP by default ([flux.workers] register_rate_limit, slowapi syntax, "" disables). A large fleet restarting behind one NAT trips it — raise the limit or make sure your proxy forwards real client IPs.

See Bootstrap tokens.

Encryption key

• FLUX_SECURITY__ENCRYPTION__ENCRYPTION_KEY set before any data lands. Encrypted columns (the secrets table, OAuth refresh tokens) use this key. Setting it after data is written leaves orphaned ciphertext.
• Backed up out of band. The key must survive every other piece of state being lost. Store it in your KMS with replication to a different region or account; back up an offline copy somewhere a single ransomware event can’t reach.
• Not the same as any other secret. Distinct value, distinct rotation cadence, distinct access controls.

Without the encryption key, the database backup is useless for any column that touches secrets.

Workers

• At least 2 replicas per pool. A single-replica pool is a single point of failure; one pod restart drains every in-flight execution for that pool until a new replica registers.
• Pools labeled for routing. Pass --label key=value (repeatable) to flux start worker — --label pool=default, --label pool=gpu, --label region=eu-west-1, whatever your routing dimensions are. Workflows then target with @workflow(workers=...).
• Capacity slots sized. Workers advertise FLUX_WORKERS__MAX_CONCURRENT_EXECUTIONS (default 16, 0 = unlimited) at registration; the server never assigns beyond a worker’s free slots. With the default subprocess runner each concurrent execution is its own process — size against memory, not just CPU.
• Resource requests and limits set. Without them, the cluster’s eviction logic will OOM-kill workers under memory pressure with no warning.
• Graceful shutdown wired. On SIGTERM the worker drains: it stops accepting work, finishes running executions (up to drain_timeout, default 60s), flushes terminal checkpoints, and exits; a second signal aborts the drain. Set the platform’s termination grace period to drain_timeout + 30s. See Worker capacity and drain.

See Running workers.

Server topology

• Replica count is a decision, not an accident. One replica is fine (restarts are brief; workers buffer checkpoints). Two or more replicas give you zero-downtime deploys and pod-loss tolerance; they coordinate through PostgreSQL — advisory-lock scheduler singleton, SKIP LOCKED dispatch — with no leader election to configure. Multi-replica requires PostgreSQL, not SQLite. See High availability.
• Sticky routing for worker connections at replicas ≥ 2. A worker’s SSE stream and dispatch queue live on the replica it connected to. Configure source-IP or cookie affinity on the load balancer for /workers/{name}/connect; round-robin is fine for everything else.
• Dispatch mode chosen deliberately. The default [flux.dispatch] mode = "poll" degrades superlinearly with fleet size; mode = "event" (PostgreSQL LISTEN/NOTIFY + batch claims + a 15s fallback tick) is the scalable mode for production fleets. See Dispatch modes.
• flux db upgrade runs before rolling replicas. Migrations are advisory-lock-guarded so racing replicas can’t corrupt the schema, but running the upgrade explicitly first keeps old and new replicas from disagreeing mid-roll.

Observability

• Prometheus scrape configured. /metrics requires the admin:metrics:read permission; create a dedicated scrape API key with that single permission. Don’t reuse the human-admin key.
• OTLP traces exported. Configure FLUX_OBSERVABILITY__OTLP_ENDPOINT to your collector. Workflow-level spans tie a request to its checkpoint sequence and any sub-workflow fanout.
• Structured logs flowing to your aggregator. Flux logs are stdlib logging with structured fields; ship them to your platform’s log sink (CloudWatch, Stackdriver, Loki).
• Dashboards for the four golden signals. Server request rate / latency / errors, scheduler dispatch lag, worker queue depth, DB connection saturation.
• Alerts on the things that page. Server 5xx rate, scheduler lag > threshold, worker pool fully drained, DB connections at limit.

See Metrics and Tracing.

Backups validated

• Restore drill executed. A backup you’ve never restored is theatre. Restore to a non-prod target at least once per quarter; time the operation; confirm the restored Flux can execute a workflow.
• Encryption key replicated separately. Backed up out of band — see the Encryption key section above. A DB backup without the key is unreadable for any encrypted column.
• Backup retention matches your RPO/RTO. Daily backups give a 24h worst-case RPO; if you need tighter, enable WAL archiving and document the recovery procedure.

See Backups and restore.

Capacity planned

• Starting numbers documented. “We sized this for X workflows/hour with Y workers.” Conservative is fine — over-provisioning at launch is cheaper than under-provisioning.
• Load test ran against representative workflows. Synthetic load that mirrors your real shape (CPU-bound vs I/O-bound vs sub-workflow-heavy). Capture latency percentiles, not averages.
• Scaling triggers identified. “When worker queue depth exceeds N for 5 minutes, scale up.” HPA on metrics, not on time-of-day.

See Capacity planning.

TLS

• Terminated at a reverse proxy. Flux’s flux start server has no --ssl-keyfile or --ssl-certfile flags. Sit it behind nginx, an ALB, a Cloud Load Balancer, an ingress controller, or a service mesh — anywhere TLS terminates and plain HTTP is forwarded to Flux on a private network.
• Workers connect over TLS too. FLUX_WORKERS__SERVER_URL=https://flux.example.com (or --server-url). The SSE stream is long-lived; an aggressive intermediate timeout severs it. Set proxy_read_timeout 3600s (or your proxy’s equivalent) on the route.

See Running the server.

Health checks

• Liveness vs readiness wired separately. Liveness is GET /health — restart the process when it fails. Readiness is GET /ready — it performs a database round-trip and returns 503 when the DB is unreachable; wire it to load-balancer membership, not restarts, so a DB blip drains traffic instead of bouncing pods.
• Probes use the status code. Both endpoints return HTTP 503 on failure and 200 on success; a plain httpGet probe is sufficient. The JSON bodies (status, database) are there for diagnostics.
• Probe timeout absorbs cold pool acquisitions. 2–3 seconds is fine; under 1 second flaps on cold connections.

See Health checks and readiness.

Failure-mode drills

• Worker crash + restart. Kill a worker pod; confirm in-flight executions resume on another worker after the heartbeat reaper evicts the dead registration.
• Server crash + restart. Kill the server pod; confirm workers reconnect with backoff and in-flight checkpoints are buffered and replayed.
• DB failover. Trigger a planned failover on your Postgres (RDS / Cloud SQL / patroni); confirm the server reconnects and execution resumes. Measure the outage window.
• Scheduler failover. Kill the server pod (or, at multiple replicas, the one currently holding the scheduler advisory lock) while a cron is due; confirm exactly one execution happens once a replica picks the cycle back up. Scheduler run state is persisted, so the fire must be exactly one — a duplicate is a bug, not an accepted window.

See Disaster recovery.

What can go wrong

The most common production miss isn’t on this list — it’s checking the boxes once at launch and never again. Re-walk the list quarterly; treat any failed item as a P2 ticket.

Specific traps:

• “We turned on auth in staging.” Production is not staging. Confirm the CRITICAL - Authentication is DISABLED line is absent from the production server’s startup log.
• “Backups are configured.” Configured is not validated. Configured-and-restored is validated. Drill.
• “The encryption key is in the secret store.” And? Is it backed up to a second region? Is the recovery procedure documented? Has someone other than the original engineer tested it?

Next

• Cloud recipes for managed-platform shapes.
• Operate for the ongoing operational concerns once the deployment is live.