Worker pools

Scaling Flux workers across heterogeneous machines, routing workflows via labels, and avoiding starvation in mixed fleets.

A pool is more than one worker pointed at the same server. You build one when a single worker can’t absorb the concurrent load, when you need to mix machine types, or when workloads have to pin to a region.

Why a pool

Each worker holds an independent SSE connection (flux/worker.py:225) and claims executions on its own. Capacity scales by adding processes, not by tuning one process.

Per-worker concurrency is not “one task at a time”. When the server pushes an execution_scheduled event, the worker spawns a new asyncio.create_task for it and tracks it in self._running_workflows — a single worker can hold several running workflows at once. Two server-side gates bound it: a hard cap (each worker advertises max_concurrent_executions at registration, default 16, and the server never assigns beyond its free slots on any dispatch path — see Capacity and drain) and a fairness check (work goes to a worker at or below the minimum active load across the pool; ties resolve via round-robin notification).

The consequence: adding workers raises the parallelism ceiling, and the slot cap plus fairness check spread work across the pool. One fat worker hoarding executions is unlikely under normal load, and a saturated pool queues in SCHEDULED rather than overloading anyone.

How the dispatcher sees the pool

There is no queue in front of Flux — no Redis, no Kafka, no broker. The execution row in the database is the queue, and the worker registry is the dispatch surface.

How claimable work meets a worker depends on [flux.dispatch] mode (see Dispatch modes). In the default poll mode, when a new execution lands the server notifies workers round-robin and each candidate’s dispatch path queries for work it can take. In event mode, one dispatcher task per server replica batch-claims work (SELECT ... FOR UPDATE SKIP LOCKED) on wakeups and routes it — the scalable path for large pools on PostgreSQL. Either way, the assignment logic is the same:

  1. The worker must have a free capacity slot and be at or below the minimum active load across the pool.
  2. Executions whose workflow has requests= or affinity= set are walked in order, asking whether the worker satisfies both — plus any runner= the workflow pins.
  3. Unconstrained workflows are the fallback if no constrained match was found.

In event mode there is a third, soft stage after the hard filters: among the eligible workers, one is chosen — least-loaded by default, or ranked by the workflow’s routing=score(...) policy when it declares one. Scoring is a preference, never a constraint, and poll mode ignores it. See Dynamic routing.

That second step is where labels matter. If no connected worker matches a workflow’s affinity, the execution stays in SCHEDULED. There is no eviction, no timeout, no failover to a “default” worker. It waits.

Routing with labels

Labels are arbitrary key=value strings set on a worker at startup. They don’t change while the worker runs.

# GPU workers
flux start worker gpu-1 --label gpu=true --label model-class=h100
flux start worker gpu-2 --label gpu=true --label model-class=h100

# CPU workers
flux start worker cpu-1 --label tier=cpu
flux start worker cpu-2 --label tier=cpu

# Region affinity
flux start worker eu-1 --label region=eu-west-1
flux start worker us-1 --label region=us-east-1

Workflows opt in to a pool slice with affinity:

from flux import workflow, ExecutionContext

@workflow.with_options(affinity={"gpu": "true"})
async def train(ctx: ExecutionContext[str]):
    return ctx.input

Matching is exact on every required key (flux/domain/resource_request.py:127):

return all(worker_labels.get(k) == v for k, v in required_affinity.items())

The worker’s labels must be a superset of the workflow’s affinity dict — every required key present with the same string value. Extra labels on the worker are fine. A workflow asking for {"gpu": "true"} matches a worker with gpu=true, model-class=h100, region=us-east-1. A worker with only tier=cpu will never run it.

For declaring affinity in code, see Worker affinity.

Resource requests

ResourceRequest is the second axis. It expresses what the workflow needs (CPU, memory, disk, GPU count, installed packages) and is compared against the resources each worker published when it registered.

from flux import workflow, ExecutionContext
from flux.domain.resource_request import ResourceRequest

@workflow.with_options(
    affinity={"gpu": "true"},
    requests=ResourceRequest(cpu=4, memory="8Gi", gpu=1),
)
async def train(ctx: ExecutionContext[str]):
    return ctx.input

Worker resources are gathered once at registration (flux/worker.py:772) via psutil and GPUtil. They are a snapshot, not a live reading. Matching is in ResourceRequest.matches_worker (flux/domain/resource_request.py:86).

Common pool topologies

Homogeneous fleet

Every worker identical, no labels needed. Round-robin spreads work. The right starting point — split the pool only once a workload needs it.

CPU + GPU split

GPU workers tagged with gpu=true. The few workflows that need a GPU declare affinity={"gpu": "true"}; everything else stays unlabeled and runs anywhere. CPU workers never touch GPU jobs; GPU workers stay reserved for them.

Multi-region

Region-tagged workers (region=eu-west-1, region=us-east-1). Workflows pin to the nearest region at registration. This is data-gravity routing, not failover — if eu-west-1 workers are all offline, the EU-tagged workflows wait.

Dev / staging / prod isolation

The cleaner option is separate Flux deployments per environment — separate database, separate server, no chance of a dev workflow touching a prod worker. Prefer this unless you have a reason not to.

The lighter alternative is environment labels on a shared pool — env=prod on production workers, env=dev on dev ones, every workflow registered with affinity={"env": "..."}. It works, but a missing affinity dict on any workflow means it can land anywhere. Enforce the discipline at registration time.

Sizing

Rules-of-thumb territory — calibrate against your workload, and remember that each worker’s concurrency is bounded by its max_concurrent_executions slots (default 16) and that under the default subprocess runner each concurrent execution is its own ~50–100 MB Python process.

Failure modes

Starvation: no worker matches

Symptom. A workflow stays in SCHEDULED forever. The server is healthy. Other workflows complete normally.

Cause. No connected worker satisfies the workflow’s affinity or requests. The dispatcher walks the candidates, finds none, moves on. Nothing logs an error.

Fix. Detect it with a synthetic check that alerts when any execution has been in SCHEDULED longer than your dispatch-latency budget — that catches this and a wedged scheduler at the same time. Then fix the cause: connect a matching worker, or relax the workflow’s constraints.

Hot worker

Symptom. One worker holds most of the active executions; others sit idle.

Cause. Usually one of two things. Either the labels are too narrow — affinity={"node-id": "host-7"} instead of affinity={"role": "training"} — and only one worker can match. Or there’s only one worker in the eligible slice and the load-balance check has nothing to balance against.

Fix. Broaden the labels, or add more workers tagged for that slice.

Stale claims after a partition

Symptom. A worker’s log shows stale-claim warnings — Execution ... was reassigned (stale claim); aborting the local copy — usually right after a network partition heals or an eviction races a worker’s recovery.

Cause. The worker went silent long enough to be evicted, its executions were reassigned to healthy pool members, and then it came back and tried to keep checkpointing. Every claim carries a generation, checkpoints carry the generation they claimed under, and the server rejects mismatches with HTTP 409 — so the returning worker aborts its now-orphaned local runs instead of double-writing history.

Fix. Nothing — this is the fencing working. The reassigned execution completed (or is completing) elsewhere with its event log intact. Investigate only if the warnings appear with no corresponding partition or eviction in the server log (grep 'evicted (stale for'), which would suggest heartbeats are being dropped somewhere in the network path.

Cross-region dispatch

Symptom. A workflow ran, but on the wrong continent — claimed by us-east-1 when it should have been eu-west-1.

Cause. The workflow has no affinity, or it has one but the label was missed at registration. The dispatcher placed it on the first matching worker, which happened to be in the wrong region.

Fix. Enforce a region label on every workflow at registration time, in CI or via a wrapper. The matching algorithm has no opinion about regions; it only knows what you told it.

Next