Worker resources

Flux dispatches each workflow to a worker that satisfies its declared ResourceRequest. Workers publish what they have at registration; workflows publish what they need at decoration time; the server walks both on every claim. Matching is a threshold model: worker capacity must be greater than or equal to the request, and the first satisfying worker wins. Match is not enforce: nothing in Flux caps a running workflow at the cores or memory it asked for. That ceiling has to come from the host.

This page covers what gets published, how matching works, and where the model leaks. Label-based routing (environment, region, GPU class) lives on Worker pools and labels.

What a worker publishes at registration

When flux start worker calls POST /workers/register, the payload assembled in flux/worker.py::_get_resources_info includes:

The server persists this across WorkerModel, WorkerRuntimeModel, WorkerResourcesModel, WorkerResourcesGPUModel, and WorkerPackageModel in flux/models.py.

GPU detection

GPU detection is automatic and depends on a single optional dependency: GPUtil, which shells out to nvidia-smi. The relevant code is Worker._get_gpu_info:

try:
    import GPUtil
except (ImportError, ModuleNotFoundError):
    return []

Consequences:

1. GPUtil is not a default Flux dependency. Install it in the worker environment if you want GPUs to be detected.
2. Detection requires a working nvidia-smi on PATH. No driver, no detected GPUs — even when the hardware is physically present.
3. Non-NVIDIA accelerators (ROCm, Intel, Apple Silicon, TPUs) are not detected. Advertise them with labels (--label accelerator=rocm) and route via affinity={"accelerator": "rocm"}. The numeric gpu=N matching is NVIDIA-only.

How workflows request resources

ResourceRequest lives in flux/domain/resource_request.py. Five optional fields:

from flux import ResourceRequest, workflow

@workflow.with_options(
    requests=ResourceRequest(
        cpu=4,
        memory="8Gi",
        disk=10_000_000_000,
        gpu=1,
        packages=["torch>=2.0", "transformers"],
    ),
)
async def train(ctx): ...

Short-form constructors are also available: ResourceRequest.with_cpu(4), .with_memory("8Gi"), .with_gpu(1), .with_disk(...), .with_packages([...]).

Matching semantics

Dispatch evaluates label affinity first and resource requirements second. A worker passes the resource check when every declared dimension clears its threshold:

• cpu: worker.cpu_available >= request.cpu
• memory: worker.memory_available >= parse_to_bytes(request.memory)
• disk: worker.disk_free >= request.disk
• gpu: count of GPUs with memory_available > 0 is >= request.gpu
• packages: every required package is present, satisfying >= or == constraints by dotted-numeric comparison

Every dimension is a >= threshold, not an exact match. When no worker matches, the execution stays in SCHEDULED until a satisfying worker registers (or the workflow is cancelled).

Requests and affinity are hard constraints — they only decide which workers are eligible. Which eligible worker actually gets the execution is a separate, soft stage: by default the least-loaded one, and in event dispatch mode a workflow can rank the eligible workers with a scoring policy (routing=score(...)) over labels, live metrics, and load — see Dynamic routing. A policy can never route to a worker that fails the resource check.

Unconstrained workflows (no requests, no affinity) take the fast path and are dispatched without per-candidate filtering. Constrained workflows iterate candidates and apply the resource check; heavy use of requests on a busy server adds per-claim CPU on the server.

Matching is not enforcement

This is the single most surprising property of the model.

ResourceRequest is routing guidance only. Flux does not derive a cgroup or rlimit ceiling from it. A workflow that declares cpu=4 and spawns a 64-thread pool gets 64 threads; one that declares memory="2Gi" and allocates 32 GiB will blow past its request.

What Flux does offer are runner-level ceilings, configured on the worker rather than derived from the request. With the default subprocess runner, each execution is its own child process, so a crash or OOM kill takes down only that execution — and subprocess_memory_limit (Linux, bytes) bounds each child’s address space. The docker runner adds docker_memory and docker_cpus per container. See Execution runners. These are per-worker settings applying to every execution equally; they do not read the workflow’s requests.

For request-shaped ceilings, set them at the host:

• Kubernetes: declare resources.requests and resources.limits on the worker pod. The Flux ResourceRequest should mirror — or stay below — the pod limit so the matcher and the kubelet agree.
• Docker / Compose: --cpus, --memory, --device for GPU passthrough.
• systemd: a slice with CPUQuota=, MemoryMax=.
• Bare metal: one worker per machine, sized to the machine, no host enforcement.

The mental model: ResourceRequest answers “which worker should this go to?”. The host answers “what is this worker allowed to do?”. Both are necessary; neither covers the other.

GPU sharing

A worker advertising gpu=2 can be matched against two concurrent workflows with gpu=1 each — both will run on the same worker (separate child processes under the default subprocess runner, but the same physical GPUs) with nothing arbitrating device access unless your code does. The recommended layout is one Flux worker per physical GPU: pin each with CUDA_VISIBLE_DEVICES=0 (or 1, 2, …), label it (--label gpu=true --label gpu_index=0), and dispatch with affinity when you need a specific card. NVIDIA MIG slices follow the same pattern: one worker, one slice.

Custom resource fields

ResourceRequest has a closed schema — cpu, memory, disk, gpu, packages — and the constructor rejects other keyword arguments. There is no extension point in 0.56.0.

For anything else — licensed cores, FPGAs, rack location, dataset cache presence — use labels and match via affinity:

@workflow.with_options(
    affinity={"license": "matlab", "region": "us-east-1"},
)
async def run(ctx): ...

Labels are free-form strings on both sides, do not participate in >= comparison, and cost the matcher only a dict lookup. See Worker pools and labels.

What can go wrong

Resource request unmatched. The execution stays in SCHEDULED. No worker has cpu_available >= request.cpu, or none has GPUs, or every candidate is missing a requested package. Detection: a SCHEDULED-age alert. Fix: register a worker that satisfies the request, or relax the request and re-register the workflow.

Worker overcommit. A workflow declared cpu=4 and consumes 32. The dispatcher does not notice. Symptoms: worker host load, slow checkpoints, OOM kills, adjacent executions on the same worker stalling or crashing. Fix, in order of leverage: lower max_concurrent_executions so fewer executions share the host; set subprocess_memory_limit (or docker-runner limits) so a runaway child dies alone; set a host-level limit; or split workers per workload class so a runaway only takes down its own peers.

GPU detection missing. A worker has a physical NVIDIA GPU but worker.resources.gpus is empty. Almost always one of: GPUtil is not installed; nvidia-smi is not on PATH; the NVIDIA driver is not loaded; or the GPU was hot-attached after the worker registered (resources are sampled once — restart the worker). Non-NVIDIA accelerators will never be detected; use labels.

What’s next

• Worker pools and labels — label-based routing, affinity, environment and region separation
• Capacity and drain — the slot model that gates concurrency per worker
• Execution runners — per-execution isolation and runner-level resource ceilings
• Running workers — CLI, bootstrap token, supervision, reconnect behaviour
• Worker observability — heartbeats, eviction, worker-state metrics