Workers

What a Flux worker is, how to start one, how it registers with the server, and how your workflow code behaves when running on a worker.

This page covers the developer-side perspective on workers (declaring, registering, behavior in code). For production deployment, scaling, and observability, see Operate → Workers.

What is a worker

A Flux worker is a long-running process that picks up queued workflow executions and runs them. The server stores workflow definitions and execution state; workers supply the compute.

Workers and the server are separate processes, often on separate machines. A single server can have many workers connected at once. Each worker registers with the server once on startup, then holds a persistent SSE connection over which it receives dispatch events. When an execution is scheduled, the server first filters to eligible workers — free capacity slot (workers advertise max_concurrent_executions at registration, default 16), matching labels/resources/runner — then picks one. By default that’s the least-loaded eligible worker; in event dispatch mode, a workflow can instead declare a scoring policy that ranks the eligible workers by load, latency, or custom metrics — see Dynamic routing. The chosen worker claims the execution, loads the workflow source, and runs it through a runner — by default a sandboxed subprocess.

Your workflow code behaves the same whether it runs on a worker or inline. The same @workflow and @task decorators, the same ExecutionContext, the same event log. The worker provides the runtime environment; your code does not need to know it is on a worker.

Starting a worker

The minimum command to start a worker:

flux start worker

Without a name argument the worker generates a random name. To give it a deterministic name:

flux start worker my-worker

To connect to a non-default server URL:

flux start worker my-worker --server-url http://server.internal:8000

Bootstrap token (required)

Workers authenticate to the server with a bootstrap token before they can receive work. The server auto-generates a token on first start and persists it. Retrieve it:

flux server bootstrap-token

Pass it to the worker via the environment variable:

export FLUX_WORKERS__BOOTSTRAP_TOKEN="<token>"
flux start worker

Without this token the worker exits immediately with:

RuntimeError: Worker bootstrap token is not configured.

Worker labels

Labels let you route specific workflows to specific workers. A label is a key=value string. Pass one or more --label flags at startup:

flux start worker gpu-worker \
  --label role=training \
  --label gpu=a100 \
  --label env=production

Labels are declared at startup and cannot change while the worker is running. If you need different labels, restart the worker.

What happens at registration

When a worker starts, it registers once with the server. The registration payload includes:

The server responds with a session token the worker uses for all subsequent communication. If the worker disconnects and reconnects, it reuses that token. If the server rejects it (for example, after a bootstrap token rotation or a routine key expiry), the worker falls back to full re-registration automatically.

After registration the worker opens a long-lived SSE connection to the server. Execution dispatch events arrive over this connection. The worker picks up new work without polling.

While connected, the worker answers the server’s heartbeat pings with pongs that carry its self-reported health and its latest advertised metrics — the built-in flux.* set plus anything a configured metrics_provider publishes. Those metrics feed dynamic routing policies and are visible via flux worker show and GET /workers.

How your workflow code runs on a worker

When the server schedules an execution on a worker, it sends the workflow’s source code — the same file you registered with flux workflow register — encoded in the dispatch event. The worker compiles and executes this source in an isolated module namespace, inside whichever runner applies: by default a credential-less subprocess whose environment is sanitized (workflow code never sees the worker’s bootstrap token or security settings). Pin a workflow to a specific runner with @workflow.with_options(runner=...); it then only dispatches to workers advertising that runner.

The workflow file must be self-contained or rely only on packages installed in the worker’s Python environment. Local relative imports that work on your development machine won’t resolve on a remote worker if those modules aren’t installed there.

The worker caches compiled workflow modules (TTL: 300 seconds, LRU-bounded at 64 entries). The cache key includes a hash of the source, so re-registering a workflow — even at the same version — takes effect on the next run rather than serving stale source for the TTL.

Targeting a workflow at specific workers

Use workflow.with_options(affinity=...) to declare that a workflow should only run on workers with specific labels:

from flux import workflow, ExecutionContext

@workflow.with_options(affinity={"role": "training", "gpu": "a100"})
async def train_model(ctx: ExecutionContext[dict]):
    # Only dispatches to workers started with --label role=training --label gpu=a100
    ...

The affinity dict is a map of label keys to label values. A worker matches if it has all of the declared labels; extra labels on the worker are ignored. If no running worker matches at the time of dispatch, the execution stays in SCHEDULED state until a matching worker connects.

Affinity is separate from resource requests. Labels describe capability (what kind of environment), while resource requests describe capacity (CPU, memory, GPU count). For resource requests see Resource requests.

Namespaces

Workflows live in a namespace (default unless overridden). The namespace is declared in workflow.with_options:

@workflow.with_options(namespace="my-team")
async def my_workflow(ctx: ExecutionContext[str]):
    ...

A worker runs workflows from any namespace that arrive over its SSE connection — workers are not scoped to a specific namespace. Namespace isolation is enforced at the server level. For operator-side namespace configuration see Namespaces.

Inspecting running workers

List workers registered with the server:

flux worker list

Show details for a specific worker (labels, resources, status):

flux worker show my-worker

These commands show only workers that have successfully registered. If a worker fails to authenticate, it won’t appear in the list.

What’s next