How workers work

How Flux workers register, claim executions, send heartbeats, and recover from crashes — the operational mechanics of durable execution.

A Flux server stores the catalog and the event log. It does not execute workflows. The actual await fetch_order(...) runs inside a separate process called a worker — usually one or more flux start worker processes sitting on machines that the server can reach. The split is what lets you scale execution capacity independently of the control plane, and what lets a workflow survive a worker crash.

This page is the mechanical walkthrough. What does a worker do when it starts? How does it find work? What keeps the server’s idea of “this worker is alive” honest? And — the question that decides whether durable execution is a marketing claim or a real one — what happens when a worker dies mid-task?

The neighbouring page, System architecture, draws the bigger picture; this page zooms in on the worker.

Registration

A worker boots by running flux start worker [name] --server-url http://server-host:8000. Before it can claim any work, it has to introduce itself.

Authentication uses a bootstrap token — a single shared secret the server prints on first launch (flux server bootstrap-token reveals it) and the worker reads from FLUX_WORKERS__BOOTSTRAP_TOKEN (or [flux.workers] bootstrap_token in flux.toml). The worker POSTs to /workers/register with Authorization: Bearer <bootstrap_token> (see flux/worker.py, Worker._register). The body of that request includes:

The server stores all of this in the workers table (the WorkerModel in flux/models.py) and responds with a session token. From that point on the worker uses the session token, not the bootstrap token, to authenticate. The bootstrap token only sees its one moment of use; if it leaks, you can rotate it without re-issuing the long-lived credentials each worker is holding.

A worker process is a long-running daemon. Once registered, it stays registered. If it disconnects and reconnects later, it tries the existing session token first; only if that’s rejected (HTTP 401/403) does it fall back to full re-registration with the bootstrap token. The retry policy on connection failure is exponential backoff with jitter, capped at reconnect_max_delay seconds (default 60, see WorkersConfig in flux/config.py).

The labels matter. Labels are how affinity routing works — a workflow that wants to land on a GPU-enabled host advertises that requirement, and the server only routes it to workers carrying the matching labels. We get to affinity later in this page.

The claim loop is push, not poll

A worker does not poll the server’s catalog asking “got any work?” The model is closer to the reverse: the worker keeps a long-lived Server-Sent Events stream open to /workers/{name}/connect, and the server pushes an event down the stream whenever there is a workflow to dispatch.

In flux/worker.py, Worker._connect is the relevant block:

async with aconnect_sse(client, "GET", f"{base_url}/connect", headers=headers) as es:
    async for evt in es.aiter_sse():
        if evt.event == "execution_scheduled":
            asyncio.create_task(self._handle_execution_scheduled(...))
        elif evt.event == "execution_resumed":
            asyncio.create_task(self._handle_execution_resumed(...))
        elif evt.event == "execution_cancelled":
            asyncio.create_task(self._handle_execution_cancelled(...))
        elif evt.event == "ping":
            asyncio.create_task(self._send_pong())

When you submit a workflow (CLI, REST, MCP, or schedule fire), the server writes a WORKFLOW_SCHEDULED event to the event log and then calls _notify_next_worker() — a round-robin signal that wakes the next connected worker. Only one worker is signalled at a time, which keeps fan-out cheap; if it can’t take the work, the server falls back to a broadcast.

The signal carries the workflow source (base64-encoded), the execution context (input + current event log), and a short-lived execution token. The worker then has to claim the execution before doing anything else, by POSTing to /workers/{name}/claim/{execution_id}. The claim is what records WORKFLOW_CLAIMED in the event log and locks the row so no other worker can pick it up. If two workers race on the same execution — for instance, the server signalled one and a broadcast caught another — the second one’s claim returns HTTP 409 and the worker quietly drops the dispatch (see the 409 handling around resume claims in _handle_execution_resumed).

After a successful claim, the worker reads the latest context back from the claim response and starts executing.

One refinement to “which worker gets signalled”: when a running workflow relays a child workflow through the server (call() with a string reference, mode="async", or a runner constraint that rules out the in-process fast path), the calling worker tags the submission with an X-Flux-Preferred-Worker header carrying its own name (flux/tasks/call.py). The server stores it on the execution (executions.preferred_worker), and event-mode dispatch prefers that worker — only when it is eligible right now (labels, resources, free slots, healthy). It is a hint, not a pin: an ineligible preferred worker means normal selection, and poll-mode dispatch ignores the hint entirely. The point is module-cache locality — a mesh of agent workflows hopping through call() keeps landing on the worker whose compiled-module cache is already warm. Workflows can also opt the hint into an explicit scoring policy; see Dynamic routing.

Heartbeats and lease enforcement

Once a worker is connected, the server doesn’t trust the SSE connection alone to confirm liveness — a stuck TCP keepalive can keep a half-dead worker “connected” indefinitely. So the heartbeat is application-layer and the server is the one driving it.

Every heartbeat_interval seconds (default 10), the server pushes an SSE event named ping to each connected worker. The worker responds by POSTing to /workers/{name}/pong. The pong is more than an “I’m alive” — it carries an optional JSON payload with the worker’s self-assessed health ({"healthy": bool}) and a snapshot of advertised metrics ({"metrics": {...}}); the metrics feed routing policies and are persisted to the workers table, change-gated so repeated identical snapshots don’t turn the heartbeat rate into a write rate. A reaper task (Server._run_heartbeat_reaper in flux/server.py) tracks the timestamp of each worker’s last pong and runs the eviction state machine:

  1. Healthy. Last pong within heartbeat_timeout seconds (default 30). Nothing happens.
  2. Stale. Last pong is older than heartbeat_timeout. The reaper records the moment the worker went stale and logs a warning. The worker is not yet evicted — there is a grace period to allow for transient network blips.
  3. Recovered. A pong arrives within the grace window. The stale marker is cleared and the worker goes back to healthy.
  4. Evicted. The worker has been stale for longer than eviction_grace_period seconds (default 30). At this point the server disconnects the worker from the connected set, marks its API key revoked, and — critically — runs _unclaim_worker_executions(name).

That last step is where partial work is rescued. _unclaim_worker_executions looks up every execution the dead worker had claimed and runs ContextManager.unclaim(execution_id) against each one. Unclaim moves the execution back to a dispatchable state, clears its worker_name, and the server signals the next available worker. The newly-dispatched execution arrives at a fresh worker with its full event log intact — including any TASK_COMPLETED events the dead worker had managed to checkpoint before going down.

Heartbeats are also persisted, not just tracked in memory. Each pong updates the worker’s last_seen_at column in the workers table — batched, so the reaper flushes the interval’s buffered pongs as one UPDATE per tick per replica rather than one commit per pong. Two things fall out of the persisted timestamp. First, liveness survives replica death: if the server replica a worker was attached to dies, a cross-replica sweep on any surviving replica detects workers nobody has heard from within the stale-plus-grace window and reclaims their executions. Second, GET /workers reads the table, so every replica returns the same fleet view. For workers attached to the local replica, the live in-memory connection state stays authoritative — a locally-disconnected worker reads offline immediately, without waiting out its heartbeat window.

Eviction is fenced against the evicted worker coming back. Every claim carries a claim generation, and every checkpoint the worker sends carries the generation it claimed under. If a partitioned worker is evicted and its execution reassigned, the old worker’s later checkpoints are rejected with HTTP 409 (stale-claim); the worker raises StaleClaimError and aborts its local copy of the run. stale-claim warnings in worker logs after a network partition heals are the fencing working, not a bug.

Defaults summarised (all from WorkersConfig in flux/config.py):

SettingDefaultMeaning
heartbeat_interval10 sHow often the server pings each connected worker
heartbeat_timeout30 sA worker is marked stale after this without a pong
eviction_grace_period30 sStale worker is evicted after this much additional time
reconnect_max_delay60 sCap on the worker’s reconnect backoff
offline_ttl7200 sEvicted workers are kept in the cache for this long
module_cache_ttl300 sCompiled workflow modules are cached for this long
loop_lag_probe_interval1.0 sHow often the worker probes its own event-loop lag
loop_lag_threshold1.0 sLag above this counts as a breach; 0 disables self-health monitoring

The numbers are tunable in flux.toml under [flux.workers] or via FLUX_WORKERS__* environment variables. Default behaviour: a hard worker crash takes between 30 and 60 seconds to be detected, and unfinished work is dispatched to a new worker as soon as the eviction completes.

Self-health: a starving worker steps out of the pool

Eviction handles the worker that goes silent. A different failure mode is the worker that stays responsive enough to answer pings but is too starved to do useful work — a CPU-bound inprocess workflow hogging the event loop, a host being throttled. For that, the worker monitors itself (Worker._monitor_loop_health in flux/worker.py).

Every loop_lag_probe_interval seconds (default 1.0), the worker measures its event-loop scheduling lag — how much later than requested an asyncio.sleep actually returned. A probe that measures lag at or above loop_lag_threshold (default 1.0 s; 0 disables the whole mechanism) counts as a breach. Three consecutive breaches flip the worker unhealthy; three consecutive clean probes recover it. Each transition triggers an immediate pong so the server learns right away instead of waiting for the next ping round-trip.

While unhealthy, the worker:

The mechanism is deliberately advisory, not a replacement for eviction: the probe itself runs on the starved loop, so under total starvation it can’t fire at all — and the server-side heartbeat reaper remains the backstop, since a fully wedged worker stops answering pings too. Transitions are observable as flux_worker_health_transitions_total (labelled state="unhealthy" / state="recovered") and the raw lag as the flux_worker_loop_lag_seconds histogram; see Worker observability.

Execution lifecycle inside the worker

After claiming an execution, the worker hands it to a runner — the pluggable strategy that decides where the workflow code executes. The default subprocess runner spawns a credential-less child process per execution; inprocess runs on the worker’s event loop; docker runs a container. See Execution runners for the trade-offs. Whichever runner is active, the same steps happen:

  1. Load the workflow source. The source arrives as a base64-encoded string in the dispatch payload. The worker decodes it and execs it into a fresh module under a synthetic name (flux_workflow__<namespace>__<name>__v<version>__h<source-hash>). Compiled modules are cached (TTL module_cache_ttl seconds, LRU-bounded by module_cache_max_size) and keyed by a hash of the source, so a frequently-running workflow doesn’t pay the import cost on every claim — and a re-registered workflow recompiles immediately instead of serving stale source from the cache.

  2. Invoke the workflow function. The runner finds the workflow-decorated object inside the loaded module that matches the requested namespace and name, and awaits it with the ExecutionContext from the claim response. From here, the code runs exactly the same way as it would in inline mode — the await task(...) mechanism described in The execution model drives the whole thing.

  3. Checkpoint per event. Each time the workflow function emits an event (every TASK_STARTED, every TASK_COMPLETED, every TASK_FAILED, every workflow-level transition), ctx.checkpoint() is called. The worker sends delta checkpoints through an outbox to /workers/{name}/checkpoint/{execution_id}; the server appends them to the event log and fsyncs. The outbox retries transient send failures with backoff (capped at checkpoint_retry_max_delay, default 30 s), keeps trying a terminal checkpoint for up to terminal_checkpoint_deadline (default 300 s) before leaving the execution to the server reaper, and recovers from a 401 by re-registering. Each checkpoint carries the claim generation — if the claim has been reassigned, the server answers 409 and the worker aborts its local run. The checkpoint POST is what makes durability real — without it, an event is just an in-memory object.

  4. Finalise. When the workflow function returns, WORKFLOW_COMPLETED is recorded. If it raised, WORKFLOW_FAILED is recorded instead. The execution’s state machine moves to a terminal state and the worker is free to pick up something else.

While a workflow is running, the worker can also process execution_cancelled events on the SSE stream — these are how external flux workflow cancel commands reach a running execution. The worker cancels the in-flight asyncio.Task, lets the cancellation propagate, and records the resulting events.

Recovery from crashes

Three scenarios are worth walking through.

The worker dies between tasks. A TASK_COMPLETED has just been fsynced, the workflow function is about to enter the next await task(...), and the worker process is killed by the OOM-killer. The execution is left in state RUNNING with worker_name set to the dead worker. Within heartbeat_timeout + eviction_grace_period (≤60 s with defaults), the reaper evicts the worker and calls unclaim. A fresh worker picks up the execution, replays the event log, finds the TASK_COMPLETED for the last task, returns its cached output, and proceeds to the next await. No task body re-runs.

The worker dies mid-task. A task body is executing — say it’s making an HTTP request — and the worker dies. No TASK_COMPLETED was recorded because the body never returned. After eviction, a fresh worker takes the execution. It replays the log, doesn’t find a TASK_COMPLETED for the in-flight task, and re-runs the task from scratch.

This is the moment where Flux’s guarantees stop and your code’s responsibilities start. The framework cannot make the second execution of the task identical to the first; it can only guarantee that the task will be re-entered with the same arguments. If the task did half its work (charged the card but never recorded a receipt), the re-run will charge the card again unless the task itself is idempotent. See Idempotency for the patterns — typically an idempotency key passed to the downstream API, or a check-then-act guard inside the task.

The whole server is restarted while workers are running. The workers’ SSE connections die and they reconnect with exponential backoff until the server is back. Because the server is the one that runs the reaper, no eviction happens during the downtime — there’s nothing to evict from. When the server comes back up, the workers reconnect with their existing session tokens, the reaper restarts with a fresh sense of “last pong,” and ongoing executions continue without the framework treating the gap as a crash. The event log was on disk the entire time.

In all three cases, the only state that has to be durable is the event log. Workers carry no persistent state of their own.

Affinity routing

Some workflows have hard requirements on where they run. A workflow that calls into a CUDA kernel needs a GPU. A workflow that touches /srv/customer-data needs a worker that has access to that volume. Flux expresses these as label-based affinity.

You start a worker with labels:

flux start worker gpu-worker-1 --label gpu=true --label region=us-east

You define the workflow with an affinity dict:

from flux import ExecutionContext
from flux.workflow import workflow

@workflow.with_options(affinity={"gpu": "true"})
async def train_model(ctx: ExecutionContext):
    ...

When the execution is dispatched, the server’s matcher (_worker_matches_workflow in flux/context_managers.py, calling ResourceRequest.matches_labels) only signals workers whose labels are a superset of the affinity dict. A worker without gpu=true will simply never be offered this workflow. If no matching worker is connected, the execution stays in SCHEDULED until one shows up — it does not fall back to a non-matching worker.

The matching is exact equality on each declared label. There is no glob syntax, no inequality operators, no expression language. If you want gpu=any, you have two label values to handle: gpu=true on the workflow side and gpu=true on every GPU worker. Stick to that shape and the system behaves predictably.

Resource matching (@workflow.with_options(requests=...)) is a separate axis. Where labels are user-defined facts about a worker, resource requests are checks against the worker’s reported CPU/memory/disk/GPU/packages — see flux/domain/resource_request.py. Both apply at dispatch time; a workflow that declares both has to find a worker that satisfies both.

In-process workers (dev mode)

Not every script needs a separate worker process. When you call workflow.run("input") directly from Python — the pattern in tests/examples/ and most “hello world” snippets — Flux runs the workflow in the current process. The implementation in flux/workflow.py is straightforward:

def run(self, *args, **kwargs) -> ExecutionContext:
    ...
    return asyncio.run(self(ctx))

There is no separate worker, no claim loop, no SSE stream, no heartbeats. The same ExecutionContext, the same event log, and the same checkpoint mechanism — but the checkpoint goes straight to the database instead of over HTTP. This is the “inline path” called out in the project’s CLAUDE.md.

The inline path is a real first-class execution mode for tests and notebooks, and it shares almost all of its code with the distributed path — so a workflow that runs correctly inline will run correctly under a worker, modulo anything that relies on per-process state (which would be a determinism violation either way). What inline mode is not suitable for is production: there’s no isolation between the workflow and the caller, no separate process to crash without taking your application down, and no way to scale execution across machines.

Production deployments should always have at least one flux start worker process running against the server.

Worker observability

Everything a worker does — registration, claims, completions, failures, retries, fallbacks, rollbacks — is recorded in the event log. The execution events carry the worker’s name in their metadata, so you can attribute a failure back to the host that produced it.

The CLI gives you two read-only views into the worker registry:

flux worker list                # all registered workers, with labels
flux worker show <name>         # full details: runtime, resources, packages, labels

flux worker list calls GET /workers and prints one line per worker — name, Python version, labels. The endpoint reads the workers table (liveness derived from the persisted last_seen_at heartbeats), so in a multi-replica deployment every replica returns the same fleet view. flux worker show <name> calls GET /workers/{name} and dumps the full record as JSON, including the inventory of installed packages the worker reported at registration. The CLI commands live in flux/cli.py under the @cli.group() def worker(): group; there are no start, stop, or delete commands here — worker lifecycle is managed by whatever runs flux start worker (systemd, Kubernetes, a container orchestrator, your shell).

For deeper introspection you have the standard event-log surface. flux execution show <execution_id> exposes the full event stream for any given execution, including the WORKFLOW_CLAIMED events that identify which worker handled it.

What to remember

Where this shows up