Performance and benchmarking

What’s measured

Flux 0.56.0 ships without a regression-tested public benchmark suite, but it is no longer number-free. The Flux repository publishes measured figures for per-execution runner overhead (inprocess ~0.1 ms, subprocess ~0.55–0.7 s, docker ~1.1–1.6 s with a precompiled image), transient same-worker call() hops (~2.3 ms median versus ~526 ms server-relayed), and dispatch-path stress tests (reproducible via scripts/stress_dispatch.py in the repository). The tuning values below are starting points sourced from those measurements and operational use — validate them against your own representative workloads.

A reference micro-benchmark ships in the Flux repository at examples/fibo_benchmark.py and exercises the inline workflow path. It’s a useful smoke test for “is replay catastrophically slow on my hardware” — not a load test for the server or scheduler.

Worker concurrency

Workers advertise a capacity at registration — [flux.workers] max_concurrent_executions, default 16, 0 = unlimited — and the server never assigns beyond a worker’s free slots. How to size it depends on the runner (see Runners):

• subprocess (the default) — each concurrent execution is its own Python process, roughly 50–100 MB baseline plus workflow memory, and each execution pays a process spawn (~0.5–0.7 s, amortized under concurrency). Size against memory as much as CPU.
• inprocess — workflow code shares the worker’s event loop. Size against what the loop can genuinely run: high capacity for I/O-bound workflows (HTTP calls, database queries, LLM completions), low (1–4 per CPU) for CPU-bound ones.
• Mixed workloads — split workloads across worker pools rather than tuning one pool for both. Run two worker fleets and use affinity={"profile": "io"} or affinity={"profile": "cpu"} to dispatch correctly.

On SIGTERM, a worker drains: it stops accepting work and finishes running executions up to drain_timeout (default 60 seconds). Give your orchestrator a termination grace period of drain_timeout plus ~30 seconds so deploys don’t cut executions off mid-drain.

Dispatch mode

The server dispatches executions to workers in one of two modes ([flux.dispatch] mode):

• poll — the default: a per-worker query loop. Fine for small fleets, but the query load grows with worker count and degrades superlinearly at scale.
• event — one dispatcher task per server replica, woken by PostgreSQL LISTEN/NOTIFY and claiming work in batches (batch_size, default 64) with FOR UPDATE SKIP LOCKED. The scalable choice for large worker fleets; a fallback tick (fallback_interval, default 15 seconds) covers missed notifications.

If you run more than a handful of workers on PostgreSQL, set mode = "event".

Storage tuning

Flux’s persistence layer uses SQLAlchemy connection pools. Defaults are reasonable for development and small production:

[flux]
database_url = "postgresql://..."
database_pool_size = 20          # connections (PostgreSQL only)
database_max_overflow = 20
database_executor_threads = 16   # server thread pool for blocking DB calls
database_pool_timeout = 30       # seconds
database_pool_recycle = 3600     # seconds; recycle connections after one hour

Size the pool and the executor together: keep database_executor_threads at or below the pool size so database threads never block waiting for a connection, and give PostgreSQL max_connections ≥ replicas × (pool_size + max_overflow) plus the workers’ LISTEN connections plus headroom. PostgreSQL is the recommended backend; SQLite is fine for development, single-process deployments, and embedded use, but doesn’t support the concurrent writes a busy server demands. The execution event log is the heaviest table — every workflow state transition is a row, so enable [flux.retention] in production or it grows without bound.

If you see a QueuePool limit ... reached error, the pool is saturated. Raise the limits, or scale out to multiple server replicas — supported on PostgreSQL with no scheduler caveat; see High availability.

Scheduler poll interval

The scheduler polls the database for due schedules on a fixed interval. Default is 30 seconds ([flux.scheduling] poll_interval = 30.0). This governs schedule firing only — dispatch of executions to workers is the separate path covered under “Dispatch mode” above, and in a multi-replica deployment each scheduler cycle runs as a fleet-wide singleton via a PostgreSQL advisory lock, so the poll load doesn’t multiply with replicas. Lowering the interval increases schedule precision — a cron schedule for * * * * * (every minute) won’t actually fire on the second with a 30-second poll — at the cost of more database load.

Reasonable values:

• 30.0 — production default. Fine for hourly, daily, and per-N-minutes schedules.
• 5.0 — when you have per-minute schedules and the precision matters.
• 1.0 — when you have per-second schedules. Note that polling once per second multiplies the scheduler’s DB query rate by 30 versus the default.

The cron and interval tolerance settings (schedule_check_tolerance, default 1.0s; once_schedule_tolerance, default 60.0s) interact with the poll interval — increasing the tolerance compensates for low-frequency polling at the cost of less precise firing times.

Trace sample rate

Observability is gated by [flux.observability] enabled = true and the observability extra. When enabled, Flux emits OpenTelemetry spans for workflow lifecycle, task execution, and HTTP calls. Sample rate controls how many traces are exported:

• 1.0 — every trace exported. Use in development and during incidents.
• 0.1 — 10% sampled. Good production starting point.
• 0.01 — 1% sampled. Use for high-throughput production where 10% is too much volume.

Sampling at the application level keeps the exporter and the tracing backend honest; you can also sample at the OpenTelemetry Collector if you prefer.

Heartbeat and eviction

Workers heartbeat every [flux.workers] heartbeat_interval seconds (default 10). The server evicts a worker that misses heartbeats for heartbeat_timeout seconds. Tuning:

• For unstable networks, raise both values. Eviction is recoverable — a worker that comes back can re-register — but mid-flight executions get rescheduled.
• For load balancers that close idle connections aggressively, lower the heartbeat interval so heartbeats arrive faster than the LB’s idle timeout. The LB’s idle timeout should be at least 5× the heartbeat interval.

Reference benchmark

examples/fibo_benchmark.py in the Flux repository exercises the inline path with a recursive Fibonacci workflow. It’s a useful sanity check for “does replay scale on my hardware” and a regression test for catastrophic event-log slowdowns. Adapt it to your own representative workloads — the test runner is intentionally minimal so you can extend it.

Carry-forward

A regression-tested benchmark suite — server throughput at N concurrent workers, scheduler precision at M schedules, replay latency at K events, tracked release over release — is still the gap. The runner-overhead and dispatch stress-test figures in the Flux repository are the first installment; the rest lands here when it’s measured.