Worker observability

This is the page you reach when an execution has been SCHEDULED for twenty minutes and no worker has claimed it, or when the same worker keeps dropping out of the pool. It covers the surfaces Flux exposes for worker state — REST endpoints, CLI shortcuts, logs, metrics — and the two triage workflows that account for almost every real incident.

The /workers REST surface

Two read endpoints (flux/api/worker_routes.py):

Both return a WorkerResponse:

{
  "name": "worker-a3f17b",
  "status": "online",
  "runtime": {"os_name": "Linux", "os_version": "6.6.87.2", "python_version": "3.14.0"},
  "resources": {"cpu_total": 16, "cpu_available": 14, "memory_total": 33554432000, "memory_available": 21474836480, "disk_total": 512000000000, "disk_free": 200000000000, "gpus": []},
  "packages": [{"name": "numpy", "version": "2.1.3"}],
  "labels": {"gpu": "true", "region": "eu-west-1"},
  "metrics": {"flux.running_executions": 3.0, "flux.loop_lag_p95_seconds": 0.011, "flux.cpu_percent": 41.2}
}

status is "online", "offline", or "unhealthy". Worker heartbeats persist (batched) to the workers.last_seen_at column, and GET /workers derives liveness from it: a worker is online while its last persisted heartbeat is within heartbeat_timeout + eviction_grace_period, matching the reaper’s staleness window. For workers attached to the replica you queried, the live connection state wins — a locally-disconnected worker reads "offline" immediately. Because the endpoint reads the table rather than per-replica memory, every replica returns the same fleet view — with one exception: "unhealthy".

"unhealthy" means the worker self-reported event-loop starvation on its heartbeat pong (see How workers work). It is still connected and finishing its running executions, but it declines new work and the server excludes it from dispatch until it reports healthy again. The unhealthy flag lives in per-replica memory, not the database — only the replica holding the worker’s SSE connection returns "unhealthy"; any other replica reports the same worker "online" from the persisted heartbeat. That’s a display quirk, not a dispatch hole: dispatch to a worker only ever flows through the replica holding its connection, which is the same replica receiving its pongs.

metrics is the latest snapshot the worker advertised on its heartbeat pong — a built-in flux.*-prefixed family (loop lag, running executions, free slots, CPU/memory, failure rates, and more) plus any user-configured provider keys. It exists to feed dynamic routing metric(...) selectors; it is persisted change-gated, so treat it as “as of the last change”, not a live gauge. The raw heartbeat timestamp and eviction reason are not in the public response — to diagnose flapping you read the server log (see Eviction loop below).

The other /workers/* routes — register, pong, connect, claim, checkpoint, release, progress, approvals, secrets/batch — are worker-only, authenticated with the worker’s session token, and not part of the observability surface.

CLI shortcuts

Two flux worker subcommands wrap the read endpoints (flux/cli.py, lines 779 and 825):

flux worker list                    # GET /workers, pretty-printed
flux worker list --format json      # raw JSON
flux worker show <name>             # GET /workers/{name}, JSON

flux worker list prints one line per worker:

Workers (3):
--------------------------------------------------
  worker-a3f17b                   Python 3.14.0  gpu=true, region=eu-west-1
  worker-b9c422                   Python 3.14.0  region=eu-west-1
  worker-d11e08                   Python 3.14.0  gpu=true, region=us-east-1

The list comes from the workers table, so it is complete and identical across server replicas. Workers that deregistered or died long ago still appear with status: "offline" — the table keeps their rows so a returning worker can re-register under the same name (offline_ttl only bounds the per-replica in-memory cache, which enriches the response with the full resource record).

Eviction states

A worker passes through four states. The numbers come from WorkersConfig in flux/config.py:

The state machine, with what /workers returns at each step:

1. Registered and connected. Worker holds an open SSE stream on GET /workers/{name}/connect and is replying to pings. GET /workers lists it with status: "online".
2. Stale. Last pong was more than heartbeat_timeout ago. The server logs Worker <name> missed heartbeat, marked STALE (grace period: 30s) and starts the grace clock. The worker still appears as "online" in the public response.
3. Evicted and offline. Stayed stale for the full grace period. The server logs Worker <name> evicted (stale for >30s), closes the SSE connection, and unclaims its executions. GET /workers now lists it with status: "offline".
4. Purged from the cache. Offline for longer than offline_ttl. The reaper drops it from the replica’s in-memory cache. It still appears in GET /workers (read from the database) with status: "offline", minus the cached resource details, and it can re-register under the same name.

Default end-to-end: a silent worker disappears from online after 60s (stale window plus grace) and from the replica cache after another two hours.

Two multi-replica additions to the state machine. Pongs persist to workers.last_seen_at, flushed as one batched UPDATE per reaper tick per replica — so if the replica a worker was attached to dies, any surviving replica’s reaper detects the globally-stale worker from the persisted timestamp and reclaims its executions. And eviction is fenced: checkpoints carry the claim generation, so a partitioned worker whose executions were reassigned gets HTTP 409 on its late checkpoints, logs stale-claim warnings, and aborts those local runs. Those warnings after a partition heals are the mechanism working, not an incident.

Orthogonal to all four states: a connected worker can also be "unhealthy" — self-reported event-loop starvation, advertised on the heartbeat pong. Unlike stale/evicted, the worker is answering pings; it is simply declining new work until its event loop recovers (three clean 1-second probes). The flag clears on recovery, and unconditionally on re-register or reconnect. Server log lines: Worker <name> reports unhealthy (event-loop starvation) and Worker <name> reports healthy again; resuming dispatch.

Triage: execution stuck in SCHEDULED

When an execution sits in SCHEDULED for more than a few seconds, the cause is almost always that no connected worker matches the execution’s resource requests or affinity labels — or that every matching worker is at capacity (max_concurrent_executions, default 16 slots; see Capacity and drain). Work through these in order:

1. Confirm what the execution wants:

   flux execution show <execution_id> --detailed

   Look at requests (CPU / memory / GPU) and any affinity labels — both come from @workflow.with_options(requests=..., affinity=...).

2. Confirm what workers are connected and what they offer:

   flux worker list --format json | jq '.[] | {name, status, labels, cpu: .resources.cpu_available}'

3. If no online worker has the required labels, start one:

   flux start worker my-gpu-worker --label gpu=true --label region=eu-west-1

4. If a worker matches but nothing is being claimed, check saturation first: flux_worker_executions_active pinned at the worker’s max_concurrent_executions means every slot is busy — the execution dispatches as soon as one frees. Otherwise dispatch is silently skipping the worker; set FLUX_LOG_LEVEL=DEBUG on the server and retry — the server logs the candidate-by-candidate dispatch decision, including which capability, runner, or resource check rejected the worker.

5. If the matching worker is "unhealthy", it is self-reporting event-loop starvation and declining new work by design. The execution dispatches elsewhere if another eligible worker exists; if this was the only eligible worker, the execution waits until the worker recovers (three clean lag probes) — check its logs for Event loop starved and see the health-transition triage under Metrics.

6. If the matching worker is "offline", it has been silent for at least 60s. Either it crashed or the network path is broken; see the next section.

Triage: worker keeps getting evicted

A worker that registers, runs briefly, then disappears or flips between online and offline is failing to keep its heartbeat alive. Three usual causes:

1. A load balancer or reverse proxy is closing idle connections. Workers hold GET /workers/{name}/connect open as an SSE stream that is idle between dispatches. Set the LB idle timeout to at least 5x heartbeat_interval — 50 seconds minimum, ideally 5 minutes. For nginx, see the proxy_read_timeout and proxy_buffering off fragment in Running the server.

2. The worker host is pausing longer than heartbeat_timeout. Laptops sleeping, containers being throttled, hypervisor live migration freezing the VM — any of these makes the worker miss its 10s pong window. Pin worker hosts to dedicated capacity, or raise heartbeat_timeout and eviction_grace_period in flux.toml.

3. A network partition. The worker log will show Connection lost (RemoteProtocolError: ...). Reconnecting in 1.5s... from the reconnect loop in flux/worker.py:168 wrapping aconnect_sse. The worker backs off up to reconnect_max_delay (default 60s) and retries forever; if the partition heals it re-registers cleanly.

To correlate worker-side reconnects with server-side evictions, grep the server log for marked STALE and evicted (stale for. Both are at WARNING.

Logs

Workers and the server use Python’s stdlib logging through flux.utils.get_logger. The default writes to stdout with a plain text formatter (flux/utils.py:213-224):

2026-05-14 09:21:03 - flux.worker - INFO - Worker starting up...

Two environment variables affect output:

Forward both worker and server stdout to your aggregator and filter on flux.worker versus flux.server. See Logs for routing patterns.

Metrics

When Flux is installed with the observability extra and FLUX_OBSERVABILITY__ENABLED=true, the server exposes a Prometheus /metrics endpoint. Worker-related metrics from flux/observability/metrics.py:

The two loop-lag instruments are recorded in the worker process, so they reach your backend through the worker’s own OTLP export ([flux.observability] on the worker host) — they do not appear on the server’s /metrics scrape. A flux_worker_health_transitions_total{state="unhealthy"} increment that is never followed by a recovered one within a minute or two is the triage signal: the worker’s event loop is persistently starved, usually by CPU-bound code on the inprocess runner or a throttled host. Correlate with the lag histogram’s p95, then either move the offending workflow to the subprocess runner or give the host dedicated capacity. Flapping between the two states means the load hovers at the threshold — spread work across more workers rather than raising loop_lag_threshold.

Don’t confuse these OTel instruments with the flux.*-prefixed scalars in the metrics field of GET /workers — the latter are worker-advertised routing inputs (see Dynamic routing), not Prometheus series. For dashboard suggestions see Metrics.

What can go wrong

• Worker registered but receives no work. The SSE stream silently dropped, or its labels match nothing you are dispatching. Compare flux worker show <name> against the workflow’s affinity and requests. If flux_worker_executions_active is flat at zero for one worker while others are claiming work, that worker’s session is wedged — restart it.

• Eviction loop. Same worker registers, gets marked STALE, gets evicted, re-registers seconds later. Almost always the LB idle timeout. Confirm with flux_worker_disconnections_total{reason="evicted"} climbing in lockstep with flux_worker_registrations_total for the same name.

• Phantom workers. Worker exited cleanly but flux worker list still shows it as offline. Expected — the list reads the workers table, and rows outlive the worker so it can re-register under the same name. Filter with ?status=online (or flux worker list --format json | jq on the status field) when you only care about live capacity.

• Stale-claim warnings in worker logs. Execution ... was reassigned (stale claim); aborting the local copy after a network partition or an eviction that raced a recovery. This is claim-generation fencing preventing a double-run — no action needed unless it happens without a corresponding partition or eviction.

• Unhealthy (event-loop lag); releasing execution ... for re-dispatch in worker logs. The worker was assigned work while self-reporting unhealthy (the assignment raced the pong) and handed it straight back — the execution re-dispatches to another worker immediately. Occasional occurrences around a health transition are normal; a steady stream means the worker keeps flipping unhealthy under load. See the health-transition triage under Metrics.

See also

• Running workers — bootstrapping, supervision, restart loops.
• Capacity and drain — slot limits and graceful shutdown.
• Worker pools — sizing, labels, pool topology.
• Logs — central log handling.
• Metrics — Prometheus surface and dashboards.
• How workers work — dispatch and heartbeat in depth.