Capacity planning

Flux 0.56.0 ships some measured numbers — per-runner execution overheads and dispatch-mode stress results are quoted where they exist (single-machine measurements from the Flux repository’s production-readiness work). Everything else below is a conservative starting point derived from reading the source. Treat estimates as the floor of your envelope, not the ceiling, and run your own load tests before committing to a capacity plan.

Workers

A worker is a single Python process that holds an SSE connection to the server and runs multiple workflow executions concurrently. It is not a one-workflow-at-a-time worker, and its concurrency is a configured slot count: the worker advertises max_concurrent_executions at registration (default 16, 0 = unlimited legacy behavior) and the server never dispatches beyond its free slots. See Capacity and drain.

The capacity unit to plan around is the slot, and what a slot costs depends on the runner:

Starting points: with the default subprocess runner, size slots × (100 MB + workflow working set) against host memory and slots ≈ cores for CPU-heavy work (child processes sidestep the GIL across executions). For I/O-bound workflows, raise the slot count on fewer workers rather than multiplying worker processes. Watch resident memory, schedule-to-start latency, and DB connection counts as you ramp — per-execution memory is what blows you up.

Server

The server is a single FastAPI process — roughly 4,800 lines in flux/server.py plus the SSE dispatch loop and SQLAlchemy pool. Per execution it does a handful of small writes (claim, checkpoint, completion), an SSE push to the assigned worker, and several reads. Most routes also run a permission check.

A single server replica should comfortably handle dozens-to-hundreds of workers concurrently — if you switch dispatch modes first. The default poll dispatch degrades superlinearly with fleet size (measured: ~500 workers saturates the DB executor and submissions take minutes to accept); FLUX_DISPATCH__MODE=event replaces the per-worker query loops with one batching dispatcher per replica and is the prerequisite for large fleets. See Dispatch modes. Two ways to scale:

• Vertical. Bigger box, more memory, larger DB pool and executor. Easiest first move.
• Horizontal. Multiple server replicas behind a load balancer, with Postgres as the shared store. Replicas coordinate through the database — event-mode dispatchers batch-claim with FOR UPDATE SKIP LOCKED (never double-assigning), and the scheduler and retention job act as fleet-wide singletons per cycle via PostgreSQL advisory locks, so running every replica with all subsystems enabled is safe. Worker SSE streams are sticky to whichever replica accepted the connection, so the load balancer needs session affinity for /workers/{name}/connect; plain round-robin works for everything else.

Storage growth

A back-of-envelope formula for event-log growth, per execution:

~1 KB execution metadata
+ N events × ~500 bytes each       (N is typically 5–50)
+ task output payload sizes        (inline by default)

Worked example: 1,000 executions per day, ~10 events each, ~500 bytes per event = ~5 MB/day of event-log rows before output payloads. Output payloads dominate as soon as workflows return non-trivial JSON. For large structures or binary artifacts, route them through configured output storage (S3, GCS, filesystem) rather than letting them land inline.

Two levers bound this growth in 0.56.0. The retention job ([flux.retention], off by default) deletes terminal executions and their events past retention_days — enable it, or the tables grow without bound. And transient workflows persist only the outer lifecycle (~4 event rows measured on an 8-task workflow versus ~19.9 durable) — the right mode for high-frequency mesh traffic whose history you would only delete anyway.

Postgres sizing

The connection pool defaults live in flux/config.py:

• database_pool_size = 20
• database_max_overflow = 20
• database_executor_threads = 16
• database_pool_timeout = 30s
• database_pool_recycle = 3600s

So one server replica peaks at 40 concurrent Postgres connections, driven by a 16-thread executor for blocking DB calls — keep database_executor_threads at or below database_pool_size so threads never block waiting for a connection. For a 3-replica horizontal deployment: 3 × 40 = 120 connections from Flux alone, plus one LISTEN connection per replica in event dispatch mode, plus your console, metrics scrapers, ad-hoc psql sessions, and any backup tooling. Set Postgres max_connections ≥ replicas × (pool_size + max_overflow) with headroom.

Workers connect to the server over HTTP/SSE, not directly to Postgres. They do not consume DB connections.

Scaling triggers

Drive scaling decisions from observability signals, not gut feeling. The relevant Prometheus metrics live in flux/observability/metrics.py:

• Scale workers when flux_execution_queue_depth trends upward over minutes. New executions are landing faster than they are being claimed.
• Watch schedule-to-start latency. If flux_execution_schedule_to_start_seconds p95 climbs above a few seconds, dispatch is starved — usually workers (all slots busy), sometimes poll-mode dispatch at fleet scale (switch to event mode), occasionally the server.
• Scale Postgres when you see pool-exhaustion errors in the server logs (QueuePool limit ... overflow ... reached) or sustained DB CPU above 70%. Either raise database_pool_size and database_max_overflow, raise Postgres max_connections, or both.
• Resume queue — flux_resume_queue_depth and flux_resume_schedule_to_start_seconds are the same signals for paused-and-resuming workflows. They behave like the execution queue but are usually a much smaller volume.

What we don’t know yet

Be honest with yourself about the gaps:

• Per-worker throughput is measured only for trivial workflows on a single machine (~4.6–5.0 exec/s at 8 concurrent under the subprocess runner; thousands/s in-process). Your task bodies dominate in practice.
• Per-server-replica throughput ceiling beyond the poll-mode stress tests.
• The point at which SQLite stops being viable. (Switch to Postgres early — any deployment with more than one worker should already be on Postgres, and event dispatch requires it.)
• Real-world replay latency for long-running workflows.

Your own load tests are ground truth.

Failure modes to plan for

Three things go wrong in capacity-bound deployments. All three are visible in the metrics above before they take the system down.

• Worker overcommit. The slot cap (max_concurrent_executions, default 16) bounds how many executions a worker runs, and subprocess_memory_limit can bound each child’s address space — but nothing in Flux caps what one execution’s code does with CPU. Set the slot count against real host memory, and keep OS/container-level limits (cgroups, Kubernetes resources.limits, Docker --memory) as the backstop. A slot count of 0 (unlimited) restores the old OOM-on-burst behavior; don’t ship it.
• DB connection exhaustion. Pool runs out, the server starts returning 500s on writes, the worker retry storm makes it worse. Mitigation: tune database_pool_size + database_executor_threads + Postgres max_connections together; never raise one without the others.
• Unbounded schedule queue. Catchup is disabled by default, so a backed-up scheduler silently skips runs rather than catching up. Mitigation: scale workers and keep flux_execution_queue_depth flat or sawtoothing, not monotonically climbing.

When in doubt, scale workers first, Postgres second, server replicas last. That ordering matches the bottlenecks most deployments actually hit.