Troubleshooting

Symptom-first. Find the error message or the behavior you’re seeing, get the root cause, apply the fix. Most entries map to a finding from Flux 0.56.0 operational work.

Server startup

RuntimeError: Worker bootstrap token is not configured

Symptom. flux start server raises on first worker registration, or flux start worker fails to register with a 500 from the server.

Root cause. Flux ships no default bootstrap token. The server raises when a worker tries to register against a server that has no token set.

Fix. Set the token before starting the server:

export FLUX_WORKERS__BOOTSTRAP_TOKEN="$(openssl rand -hex 32)"
flux start server

Or set it in flux.toml:

[flux.workers]
bootstrap_token = "..."

Auto-generated tokens are written to <home>/bootstrap-token on first start; print the value with flux server bootstrap-token. See Bootstrap token.

flux start server won’t bind to its port

Symptom. OSError: [Errno 98] Address already in use or similar.

Root cause. Another process is listening on the configured port (default 8000).

Fix. Either change the port via FLUX_SERVER_PORT=8001 flux start server (or server_port = 8001 in flux.toml), or stop the conflicting process — lsof -i :8000 will identify it.

RuntimeError: Refusing to start with incomplete security configuration

Symptom. The server exits at startup listing missing [flux.security] values.

Root cause. Auth is enabled but execution_token_secret and/or encryption.encryption_key are unset. The server fails fast by design — these used to surface as mid-traffic errors on the first worker-token mint or the first secret write.

Fix. Set both before starting:

export FLUX_SECURITY__EXECUTION_TOKEN_SECRET="$(openssl rand -hex 32)"
export FLUX_SECURITY__ENCRYPTION__ENCRYPTION_KEY="$(openssl rand -hex 32)"

Worker registration and dispatch

Worker stuck in SCHEDULED state forever

Symptom. An execution sits in SCHEDULED indefinitely; no worker picks it up.

Root cause. No registered worker matches the workflow’s requests (ResourceRequest) or affinity (label dict). The scheduler keeps the execution queued until a matching worker appears.

Fix. Check flux worker list and confirm at least one worker advertises the required CPU, memory, and labels. Start a worker with matching labels via flux start worker my-worker --label gpu=true. If you don’t need affinity, drop the affinity={...} from @workflow.with_options. See Affinity and resources.

Worker keeps getting evicted

Symptom. Workers register, claim a workflow, then disappear; the server logs worker evicted events.

Root cause. A load balancer in front of the server closes idle SSE connections before the worker’s heartbeat arrives. Default heartbeat interval is 10 seconds; default eviction is several missed heartbeats.

Fix. Set the LB idle timeout to at least 5× heartbeat_interval (50 seconds for the default). For AWS ALB, set the idle timeout to 60 seconds or higher; for nginx, set proxy_read_timeout 120s. If the server runs multiple replicas, also confirm the load balancer has connection affinity (source-IP or cookie) for the worker path — without stickiness a worker’s requests land on replicas that don’t hold its SSE stream. See Worker reverse-proxy notes and High availability.

429 Too Many Requests on POST /workers/register

Symptom. A burst of worker restarts produces 429 responses and workers that take minutes to come online.

Root cause. Worker registration is rate-limited per client IP — [flux.workers] register_rate_limit, default "30/minute" — to guard the shared bootstrap token against online brute force. A large fleet restarting behind one NAT, or a proxy that presents a single client IP, hits the limit.

Fix. Raise the limit (FLUX_WORKERS__REGISTER_RATE_LIMIT=300/minute; empty string disables), or configure the proxy to forward real client IPs so each worker counts separately.

flux workflow register succeeds but the workflow doesn’t run

Symptom. Registration returns 200, the workflow appears in flux workflow list, but nothing runs.

Root cause. Either (a) the workflow has no schedule attached and nothing called workflow.run(...), or (b) a schedule exists but no worker matches its resource/affinity requirements.

Fix. Check flux schedule list to confirm a schedule is attached and active. Run the workflow explicitly with flux workflow run <workflow_name> '<json-input>' to verify dispatch works at all. If a schedule is present but executions queue, see “Worker stuck in SCHEDULED state” above.

Scheduling

Schedule fires twice

Symptom. A cron schedule produces two executions per fire interval.

Root cause. Not server replica count — the scheduler dispatch cycle is a fleet-wide singleton per cycle via a PostgreSQL advisory lock, and run state is persisted, so N replicas fire each schedule once. The usual causes: a duplicate schedule row (auto-scheduling creates <workflow>_auto on registration alongside any manually created schedule for the same workflow — [flux.scheduling] auto_schedule_enabled = false disables the auto-creation), two separate Flux deployments sharing one database, or a pre-0.5x server still in the mix.

Fix. Run flux schedule list --all and delete the duplicate row, or upgrade/retire the outdated server process. See Schedule management.

Workflow execution

Workflow won’t replay deterministically

Symptom. Pause-then-resume produces different results than a single-run, or replays diverge on retry.

Root cause. Non-deterministic code in the workflow body — time.time(), random.random(), file reads, network calls. The body re-runs from the top on replay; events recorded during the first run are returned for await task(...) calls, but raw Python in the body executes fresh every time.

Fix. Move non-determinism into tasks. Replace time.time() with await now(), replace random.randint(...) with await randint(...), wrap any file or network read in a @task. See Determinism.

Authentication and identity

401 Unauthorized: OIDC token expired

Symptom. REST API calls that worked yesterday return 401 today.

Root cause. Either the token actually expired and needs refreshing, or the server’s clock has drifted from the IdP’s clock far enough that valid tokens look expired.

Fix. Run NTP on the server host. If clocks are correct and the token genuinely expired, refresh it client-side. As a last resort, raise [flux.security.auth.oidc] clock_skew (seconds) — but fix the clock first. See OIDC.

401 Unauthorized: Invalid audience

Symptom. OIDC tokens validate elsewhere but Flux rejects them.

Root cause. The aud claim in the token doesn’t match Flux’s configured audience.

Fix. Either fix the IdP to issue tokens with the audience Flux expects, or fix [flux.security.auth.oidc] audience to match the IdP’s claim. Token introspection (jwt.io or python-jose) reveals what the IdP is sending. See OIDC.

Secrets

Secret not found at runtime

Symptom. A task with secret_requests=["api_key"] raises at runtime that the secret doesn’t exist.

Root cause. The secret name in secret_requests doesn’t match a row in the secrets store.

Fix. Register the secret:

flux secrets set api_key "sk-..."
flux secrets list

The name in secret_requests must be an exact match. See Secrets.

Agents

Ollama response_format dropped when tools are present

Symptom. An Ollama-backed agent with both tools=[...] and response_format=Model returns free-form text on the final turn, and the worker log carries a warning that response_format was dropped.

Root cause. Ollama’s API does not accept response_format and tools together. Flux drops response_format and logs a warning so the behavior is visible rather than silent.

Fix. Either drop tools from the structured-output agent (run a separate agent for tool work, hand its result to a structured-output-only agent), or post-process the final message with a follow-up structured-output call. See Ollama provider notes.

Database

QueuePool limit of size 20 overflow 20 reached

Symptom. The server starts returning 500s under load; logs show SQLAlchemy queue-pool timeouts.

Root cause. The connection pool (default 20 base, 20 overflow per replica) is saturated. Either the database is slow, traffic exceeds the pool, or connections are being held open longer than expected.

Fix. Raise the pool sizes for production — these are top-level keys in flux.toml (env: FLUX_DATABASE_POOL_SIZE, etc.):

[flux]
database_pool_size = 40
database_max_overflow = 40
database_pool_timeout = 30
database_pool_recycle = 3600
database_executor_threads = 32   # keep at or below database_pool_size

Keep database_executor_threads at or below the pool size so DB threads never block waiting for a connection, and make sure PostgreSQL’s max_connections covers replicas × (pool_size + max_overflow) plus headroom. If raising the pool doesn’t help, check the database server for slow queries — long-running queries hold connections and starve the pool. Move to PostgreSQL if you’re still on SQLite for a busy server.

sqlite3.OperationalError: database is locked

Symptom. SQLite returns “database is locked” errors under concurrent load.

Root cause. SQLite serializes writes; under concurrent execution writes from multiple workers or the scheduler, contention manifests as lock errors.

Fix. Move to PostgreSQL — pip install flux-core[postgresql] and set FLUX_DATABASE_URL=postgresql://.... SQLite is fine for development and single-process deployments, not for a multi-worker production server. See Database backends.

Resumption and replay

Workflow resume produces different results than the original run

Symptom. A resumed execution returns a different value than a fresh execution with the same input would.

Root cause. Non-determinism in the workflow body (see “Workflow won’t replay deterministically” above), or the workflow’s source code changed between the original run and the resume — Flux versions workflows on registration, but if the in-memory catalog has a newer version, resume picks up the new source.

Fix. Move all non-determinism into tasks, and register workflows explicitly with flux workflow register rather than relying on inline auto-registration if you depend on resumability across deployments.

Pause never completes — execution stays in PAUSED forever

Symptom. await pause(name=...) runs, the execution transitions to PAUSED, and never resumes.

Root cause. Nothing is calling flux workflow resume <workflow_name> <execution_id> <input> (or the equivalent API). Pause requires an external trigger.

Fix. Resume via the CLI (flux workflow resume <workflow_name> <execution_id> <input>) or the REST API (POST /workflows/{namespace}/{name}/resume/{execution_id}/{mode}). pause is a builtin from flux.tasks — await pause(name=...) — and has no time-based auto-resume; an external caller must trigger the resume. See Pause and resume.

CLI

flux workflow list returns empty after registering a workflow

Symptom. flux workflow register my_workflow.py succeeds, but flux workflow list shows nothing.

Root cause. The CLI and the server are pointing at different databases. The CLI uses FLUX_DATABASE_URL (or the flux.toml value) directly; the server uses its own configured URL. If they diverge, the CLI sees a different catalog than the server.

Fix. Confirm both processes use the same FLUX_DATABASE_URL. For server-side queries, point the CLI at the server’s REST API via flux --server-url http://localhost:8000 workflow list to bypass direct DB access.

flux agent create fails with “encryption key not configured”

Symptom. flux agent create (or any command that writes encrypted data) raises about a missing encryption key.

Root cause. Flux ships no default encryption key. Secrets, configs, and agent definitions are encrypted at rest and require an explicit key.

Fix. Set the key before running the CLI:

export FLUX_SECURITY__ENCRYPTION__ENCRYPTION_KEY="$(openssl rand -hex 32)"

Or set [flux.security.encryption] encryption_key in flux.toml. The CLI and the server must use the same key. See Encryption.