Running workers

You have a Flux server running and reachable. This page is the other half of that deployment: the long-running worker processes that execute workflows. Everything below assumes you have already read Running the server — the bootstrap token, the network reachability constraints, and the SSE-friendly proxy config all originate there.

A worker is a foreground daemon. One worker process per CPU bucket is the usual starting point; you scale up by running more processes, not by giving one process more threads. There is no leader election among workers — every worker is interchangeable, and the server is the one routing work to whichever instance is connected.

The command

flux start worker [name] --server-url http://server-host:8000 [--label key=value ...]

That is the full surface (flux/cli.py::start_worker). One positional argument and two options:

• name — optional. If you omit it, the worker generates one as worker-<6 random hex chars>. In production give it a deterministic name (the hostname is the obvious choice) so you can correlate logs across restarts and so the server’s offline-worker cache reuses the same row.
• --server-url, -surl — defaults to workers.server_url in config (default http://localhost:8000). Use the address the worker can reach, not the public hostname.
• --label, -l — key=value strings, repeatable. Whitespace is stripped, empty keys or values exit with code 1. Labels drive affinity routing.

There are no flags for the heartbeat interval, reconnect backoff, capacity, or module cache TTL. Those are all read from configuration at worker start. Set them in flux.toml under [flux.workers] or as FLUX_WORKERS__* environment variables. Two you should size deliberately before production: max_concurrent_executions (the slot count the server dispatches against, default 16) and drain_timeout (how long a stopping worker finishes running work, default 60 s) — both covered in Capacity and drain.

Bootstrap token

A worker that boots without a bootstrap token raises at startup:

RuntimeError: Worker bootstrap token is not configured. Set
FLUX_WORKERS__BOOTSTRAP_TOKEN or 'bootstrap_token' under [flux.workers] in
flux.toml. Retrieve the server's token by running 'flux server bootstrap-token'
on the server host.

That message comes from flux/worker.py::Worker.__init__ — the check is unconditional. There is no auto-discovery, no anonymous registration path.

The token is a single shared secret that workers POST to /workers/register with Authorization: Bearer <token>. The server responds with a session token (and provisions a service-principal API key behind it); from then on every request the worker makes uses the session token, not the bootstrap token. The bootstrap token is sensitive — leaking it lets anyone register a worker and start receiving dispatched workflows.

Retrieve it from the server host:

flux server bootstrap-token

Distribute it to every worker host. A minimal env file:

# /etc/flux/worker.env
FLUX_WORKERS__SERVER_URL=https://flux.internal:8000
FLUX_WORKERS__BOOTSTRAP_TOKEN=eyJhbGc...   # from `flux server bootstrap-token`

If you later run flux server bootstrap-token --rotate on the server, the running server keeps using its in-memory copy until you restart it, and every worker must then re-register with the new value. Plan rotations during a maintenance window.

How dispatch reaches a worker

A worker does not poll the server. It opens a long-lived Server-Sent Events stream to GET /workers/{name}/connect (via httpx_sse.aconnect_sse in flux/worker.py::Worker._connect) and the server pushes events down that channel:

The 409 on claim matters. In the default poll dispatch mode the server uses a round-robin notify and a broadcast fallback, so two workers can occasionally see the same execution_scheduled event. Only one claim wins; the other gets HTTP 409 Conflict and quietly drops the dispatch. There is no retry storm and no duplicate execution. (How claimable work is found server-side — per-worker polling vs a batching event dispatcher — is a server setting; see Dispatch modes. Workers behave identically under both.)

Once the worker has claimed an execution, it loads the workflow source (delivered base64-encoded in the dispatch payload, cached for module_cache_ttl seconds), runs the workflow function through its configured runner (a sandboxed subprocess by default), and POSTs to /workers/{name}/checkpoint/{execution_id} after every event. The checkpoint POST is what makes durability real — the event log on the server is the source of truth, not anything in the worker’s memory. Checkpoints go through an outbox that retries with backoff (cap checkpoint_retry_max_delay, default 30 s) and keeps pushing a terminal checkpoint for up to terminal_checkpoint_deadline (default 300 s), so a transient server blip doesn’t lose events.

The server also caps how much it sends: a worker advertises max_concurrent_executions at registration (default 16, 0 = unlimited), and no dispatch path assigns beyond its free slots. See Capacity and drain.

Heartbeat mechanics

The server drives the heartbeat. Every heartbeat_interval seconds it pushes a ping event down the SSE stream; the worker responds with POST /workers/{name}/pong. A reaper on the server (Server._run_heartbeat_reaper in flux/server.py) runs the eviction state machine:

After eviction the server runs _unclaim_worker_executions(name), which moves every execution that the dead worker had claimed back to a dispatchable state. The fresh worker that picks it up replays the event log and resumes from the last checkpointed event. Reassignment is fenced by claim generation: if the evicted worker comes back and keeps checkpointing, the server answers 409 and it aborts the stale local run (stale-claim warnings in its log). See How workers work for the full crash-recovery walkthrough.

Heartbeats persist to the workers.last_seen_at column, batched into one UPDATE per reaper tick per replica. That gives multi-replica deployments a shared liveness view: any replica’s reaper can reclaim executions from workers attached to a replica that died, and GET /workers returns the same fleet state no matter which replica answers.

The defaults (from WorkersConfig in flux/config.py):

Default behaviour: a hard worker crash is detected in 30–60 seconds, the worker’s in-flight executions are re-dispatched immediately, and the worker’s row in the offline cache survives for two hours so a quick restart reuses it instead of forcing a re-registration burst.

Reconnect

If the SSE stream drops (network blip, server restart, load balancer eviction), the worker catches the exception in Worker._run, sleeps for an exponentially backing-off delay with jitter, capped at reconnect_max_delay, and tries again. Pseudocode:

backoff = 1
while True:
    try:
        await self._connect()
        backoff = 1
    except Exception:
        delay = min(backoff * (0.5 + random.random()), self._reconnect_max_delay)
        await asyncio.sleep(delay)
        backoff = min(backoff * 2, self._reconnect_max_delay)

On reconnect, the worker tries its existing session token first; only if the server returns 401 or 403 does it fall back to a full re-registration with the bootstrap token. That means a server restart, by itself, does not force every worker to re-register — the session tokens survive a server bounce.

Worker credentials also rotate themselves: API keys minted at registration carry a TTL (worker_key_ttl under [flux.security.auth.api_keys], default 7 days), and the first 401 after expiry triggers an automatic re-registration for a fresh key. No operator action needed. Note that POST /workers/register is rate-limited per client IP (register_rate_limit, default 30/minute) — a large fleet restarting behind one NAT needs it raised.

Labels and affinity

Labels are arbitrary key=value strings attached to a worker at start time. They drive affinity routing: a workflow declares which labels it requires, and the server only dispatches it to workers whose labels match.

flux start worker gpu-worker-1 --label gpu=true --label region=us-east

The workflow side:

@workflow.with_options(affinity={"gpu": "true", "region": "us-east"})
async def train_model(ctx: ExecutionContext):
    ...

Matching is exact-equality on every key (flux/domain/resource_request.py::matches_labels):

return all(worker_labels.get(k) == v for k, v in required_affinity.items())

Extra worker labels are fine. Missing or mismatched required keys are not.

Supervision

A flux start worker process is foreground-by-default. Wrap it in your init system.

systemd

# /etc/systemd/system/flux-worker@.service
[Unit]
Description=Flux worker (%i)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=flux
Group=flux
WorkingDirectory=/var/lib/flux
EnvironmentFile=/etc/flux/worker.env
ExecStart=/usr/local/bin/flux start worker %i
Restart=on-failure
RestartSec=5s
KillSignal=SIGTERM
# Must exceed drain_timeout (default 60s) so the drain can finish
TimeoutStopSec=90s

[Install]
WantedBy=multi-user.target

Enable per-host instances with systemctl enable --now flux-worker@$(hostname).service. The %i template variable becomes the worker name, so logs and the server’s worker registry both key on the hostname.

Docker

docker run -d --name flux-worker-$(hostname) \
  -e FLUX_WORKERS__SERVER_URL="https://flux.internal:8000" \
  -e FLUX_WORKERS__BOOTSTRAP_TOKEN="$(cat /etc/flux/bootstrap-token)" \
  --restart unless-stopped \
  --stop-timeout 90 \
  ghcr.io/edurdias/flux:0.56.0 \
  flux start worker $(hostname)

Workers are stateless. They do not need a volume mount — the workflow source travels with each dispatch, and the module cache is in-memory only.

Kubernetes

A Deployment with a stable per-pod name. The trick is using the pod hostname as the worker name so each pod registers as a distinct worker:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flux-worker
spec:
  replicas: 4
  selector:
    matchLabels:
      app: flux-worker
  template:
    metadata:
      labels:
        app: flux-worker
    spec:
      # Must exceed drain_timeout (default 60s) so a terminating pod can drain
      terminationGracePeriodSeconds: 90
      containers:
        - name: worker
          image: ghcr.io/edurdias/flux:0.56.0
          command: ["sh", "-c"]
          args: ["flux start worker $HOSTNAME --label tier=general"]
          env:
            - name: FLUX_WORKERS__SERVER_URL
              value: http://flux-server:8000
            - name: FLUX_WORKERS__BOOTSTRAP_TOKEN
              valueFrom:
                secretKeyRef:
                  name: flux-bootstrap
                  key: token
          resources:
            requests: { cpu: "500m", memory: "512Mi" }
            limits:   { cpu: "2",    memory: "2Gi"   }

For GPU pools, run a second Deployment with --label gpu=true and a nodeSelector pinning it to GPU nodes. The server routes the labelled workflows; Kubernetes routes the pods.

Shutting a worker down

Graceful drain is built in. The worker installs cooperative loop-level signal handlers for both SIGTERM and SIGINT (flux/worker.py); when either arrives it drains:

1. It stops accepting new work — no further claims.
2. Running executions continue, up to drain_timeout seconds (default 60; 0 = cancel immediately).
3. Executions still running at the deadline are cancelled.
4. Terminal checkpoints are flushed, observability is flushed, and the process exits cleanly.

A second SIGTERM aborts the drain and exits immediately. There is no CLI drain command and no pause verb — the signal is the interface, which means every ordinary systemctl stop, docker stop, and Kubernetes pod termination already drains correctly. The one thing you must do is give the supervisor more time than the drain: TimeoutStopSec (systemd), --stop-timeout (Docker), or terminationGracePeriodSeconds (Kubernetes) should exceed drain_timeout — budget drain_timeout + 30s. See Capacity and drain for the full mechanics.

If a worker is killed hard (SIGKILL, OOM, node loss) before it can drain, the dispatch is not lost — within heartbeat_timeout + eviction_grace_period (≤60 s with defaults), the server’s reaper notices the missing pong, evicts the worker, unclaims its in-flight executions, and re-dispatches them to a healthy worker, which replays the event log and resumes from the last checkpointed event. The cost is the eviction latency and one task body re-run for whatever was executing. Make tasks idempotent (see Idempotency) regardless.

What can go wrong

Most “worker won’t run” incidents fall into three buckets.

Missing bootstrap token

Symptom. The process exits at startup with RuntimeError: Worker bootstrap token is not configured. The worker never reaches the registration request.

Fix. Set FLUX_WORKERS__BOOTSTRAP_TOKEN in the environment (or [flux.workers] bootstrap_token in flux.toml). The value must match what the server is serving — fetch it with flux server bootstrap-token on the server host. A whitespace-only env var is treated as unset (see the normalisation in Worker.__init__), so a stray newline in a shell variable will produce the same error.

SSE drops repeatedly

Symptom. Logs show SSE connection closed, reconnecting... followed by Reconnecting in N.Ns... at increasing intervals. The worker re-registers, claims a few executions, then drops again.

Cause. Almost always an intermediary closing the long-lived connection: a load balancer with an idle timeout shorter than heartbeat_interval × N, a reverse proxy buffering responses, or a NAT box reaping the connection. The server pushes ping every 10 seconds and the worker POSTs pong in response, but if the proxy buffers the SSE stream, the worker never sees the ping and never sends the pong.

Fix. The proxy in front of the Flux server must (a) disable response buffering for the /workers/*/connect path and (b) set an idle timeout that is at least 5× heartbeat_interval — 60 seconds minimum, 300 seconds is comfortable. The nginx fragment in Running the server is the reference; if you are behind an AWS ALB, set the target group’s idle timeout to 300 seconds. Cloudflare’s free tier closes connections at 100 seconds and is not viable in front of Flux without an enterprise plan.

Eviction storms after a restart

Symptom. You restart a fleet of workers, and the server logs a burst of Worker X evicted (stale for >30s) followed by Worker X marked OFFLINE. Some executions get redispatched and run twice.

Cause. offline_ttl defaults to 7200 seconds — two hours. A worker that goes away and comes back inside that window reuses its registry row; a worker that comes back after a fresh registration under the same name without going through eviction first can trip races in the dispatch logic.

Fix. First check that the workers are actually being SIGTERMed, not SIGKILLed: a SIGTERM’d worker drains and disconnects cleanly, producing no eviction at all. Evictions on restart mean the supervisor’s stop timeout is shorter than drain_timeout (so it escalates to SIGKILL mid-drain) or something is killing the process outright. For rolling restarts, stagger them so no more than one worker is draining at a time. If your deploys consistently outpace the default two-hour offline_ttl, drop it to something like 600 seconds — there is no benefit to keeping a registry cache entry for a host that will not return.

Next

• Capacity and drain — slot sizing and the drain timeline in detail.
• Worker pools — labelling strategies, sizing, and the per-pool isolation patterns.
• Worker observability — the metrics and log lines that tell you whether a worker is healthy.
• How workers work — the mechanical walkthrough this page operationalises.
• Bootstrap tokens — rotation, distribution, and recovery.