Capacity and graceful drain

Two fleet-operations mechanics live on every worker: a capacity limit the server never dispatches past, and a graceful drain that turns SIGTERM into “finish what you’re running, then exit”. Both are configuration-driven — there is no CLI verb for either.

Capacity slots

A worker advertises its maximum concurrency at registration:

[flux.workers]
max_concurrent_executions = 16   # 0 = unlimited (the legacy behavior)

Or per host: FLUX_WORKERS__MAX_CONCURRENT_EXECUTIONS=16.

The server tracks each worker’s free slots and never assigns beyond them on any dispatch path — fresh dispatch, resume, or event-mode batch claims all respect the same limit. When every eligible worker is full, executions wait in SCHEDULED until a slot frees; they are not queued onto a busy worker.

0 disables the limit and restores the pre-0.53 behavior, where concurrency was bounded only by the event loop and host resources. Don’t use 0 in production — an unbounded worker under a dispatch burst is an OOM waiting to happen.

Sizing the slot count

What a slot costs depends on the runner:

• subprocess (the default): each concurrent execution is its own Python process — roughly 50–100 MB baseline plus whatever the workflow allocates. Size against memory first: max_concurrent_executions × (baseline + workflow working set) must fit the host, with headroom for the worker process itself.
• inprocess: workflow code shares the worker’s event loop. Size against what the loop can genuinely run concurrently — I/O-bound workflows tolerate high slot counts; anything CPU-bound serializes on the GIL regardless of the number you pick.
• docker: like subprocess, plus per-container overhead; docker_memory / docker_cpus give each slot an enforced ceiling.

Note that a transient fast-path hop runs inside its parent’s slot — mesh hops do not multiply slot consumption.

Graceful drain

On SIGTERM (or SIGINT), a worker drains instead of dying:

1. It stops accepting new work — no further claims.
2. Running executions continue, up to drain_timeout seconds (default 60; 0 = cancel immediately).
3. Executions still running at the deadline are cancelled.
4. Terminal checkpoints are flushed to the server, then the process exits.

A second SIGTERM aborts the drain and shuts down immediately.

[flux.workers]
drain_timeout = 60

This is signal-driven by design: there is no flux worker drain command, no pause verb, no API call. Whatever supervises the process — systemd, Docker, Kubernetes — already sends SIGTERM on stop, so the drain happens on every ordinary deploy or scale-down without extra tooling.

A worker killed hard before it can drain (SIGKILL, OOM, node loss) is still covered by the server-side path: the reaper evicts it after heartbeat_timeout + eviction_grace_period and re-dispatches its in-flight durable executions, which replay from their last checkpoint. The drain just makes the routine case — deploys — checkpoint-clean and eviction-free.

Unhealthy workers decline work too

Drain is not the only state in which a worker stops accepting work. A worker that detects its own event loop is starved (lag above loop_lag_threshold on three consecutive probes) marks itself unhealthy: it stays connected and finishes what it’s running — like a drain — but instead of exiting, it releases any newly assigned execution back to the server for immediate re-dispatch and advertises the state on its heartbeat pongs, so the server stops sending it work until three clean probes recover it. No operator action is involved; it’s automatic back-pressure. Mechanics in How workers work, triage in Worker observability.

Observing it

• flux execution list — executions sitting in SCHEDULED while workers are healthy usually means every matching worker is at capacity: add workers or raise slots.
• flux_worker_executions_active (Prometheus) — per-worker active-execution gauge; flat at the slot count means the worker is saturated.
• flux_execution_schedule_to_start_seconds — climbing p95 is the earliest capacity signal.
• Worker logs at drain time show the accepted-work stop, the wait, and any cancellations at the deadline.

What’s next

• Running workers — supervision, restart loops, and the shutdown path in context.
• Capacity planning — fleet-level sizing.
• Execution runners — what one slot costs under each runner.
• Dispatch modes — how the server assigns work into free slots.