Workers

Worker routes split cleanly into two groups: endpoints the worker process calls into the server, and read-only endpoints for operators.

Worker-side endpoints all require worker:*:* (built into the worker role) and additionally check that the bearer-token identity matches the path {name} via _verify_worker_identity. Calling them from a normal admin token will return 403.

Operator-facing reads

GET /workers

List known workers from the in-memory cache.

• Query: ?status=online|offline (optional). Without it, both are returned.
• Auth: any authenticated identity; no permission gate, since worker visibility is intentionally open.
• Response: array of WorkerResponse objects with name, status, runtime, resources, packages, labels.

GET /workers/{name}

Show one worker. Falls back to the database for workers that have disconnected but are still on record.

• Auth: any authenticated identity.
• 404: worker not known to either the cache or the DB.

Worker-side endpoints

These are called by flux start worker against the server. Operators rarely call them directly; they are documented here for completeness and for anyone building a custom worker.

POST /workers/register

One-shot registration. The worker presents a bootstrap token (not an API key) in Authorization: Bearer <bootstrap_token>. If the configured [flux.workers].bootstrap_token is unset, the server auto-generates one on first start; surface it with flux server bootstrap-token.

• Body (WorkerRegistration): name, runtime (OS, Python version), packages (installed dependency snapshot), resources (CPU, memory, disk, GPUs), labels (dict).
• Response: registration record plus a session_token — a freshly minted API key bound to the worker’s auto-provisioned service-account principal. The worker stores this and uses it on all subsequent calls.
• 403: invalid or missing bootstrap token.

GET /workers/{name}/connect

The dispatch SSE channel. Long-lived; the server pushes work to the worker through this stream.

• Auth: worker’s session API key; identity subject must equal {name}.
• Response: text/event-stream. Event types:
  • ping — keepalive every heartbeat_interval seconds.
  • execution_scheduled — data is JSON with workflow (source base64-encoded) and context; optionally an exec_token and a trace_context.
  • execution_resumed — same shape, sent when a paused execution is resumed.
  • execution_cancelled — data carries {"context": ...}; the worker should interrupt the named execution.
  • error — transport-level error.

The server tracks a per-worker connection generation; opening a new connect stream for the same name supersedes the old one.

POST /workers/{name}/claim/{execution_id}

Claim an execution the server just dispatched. Idempotent for re-claim attempts on the same worker; conflicts with claims from other workers return 409.

• Body: empty.
• Response: the claimed ExecutionContext summary.
• 404: execution not found. 409: execution in a state that doesn’t allow claim (already claimed elsewhere, finished, etc.).

POST /workers/{name}/checkpoint/{execution_id}

Persist a state-transition checkpoint. Called after every ExecutionEvent the worker emits while running the workflow.

• Body: full ExecutionContextDTO.
• Response: execution summary.
• Side effect: wakes any sync or stream caller waiting on the same execution_id.

POST /workers/{name}/progress/{execution_id}

Forward task-progress events (from ctx.progress(...)) to any SSE stream subscribed to the execution. Drops events silently if no stream is listening or the buffer is full.

• Body: list of progress events [{"task_id", "task_name", "value"}].
• Response: {"status": "ok"}.

POST /workers/{name}/pong

Heartbeat acknowledgement. Called by the worker in response to ping events on its connect stream.

• Body: empty.
• Response: {"status": "ok"}.

POST /workers/{name}/secrets/batch

Fetch secrets for an in-flight execution. The server enforces that the requested keys are declared in the workflow’s secret_requests metadata; undeclared keys cause a 403 listing the disallowed names.

• Body: {"execution_id": "<id>", "names": ["KEY_A", "KEY_B"]}.
• Response: {"KEY_A": "value", "KEY_B": "value"}.
• 403: worker does not own the execution, or requested keys are not declared on the workflow.
• 404: execution or workflow not found, or one of the requested secrets is missing from the store.

Worker eviction

The server reaper marks workers as stale when pongs stop arriving for longer than the configured grace period, then evicts them. Eviction triggers the connect-stream coroutine to yield and close the SSE response. The worker is expected to reconnect with exponential backoff.