System architecture

The four moving parts that make a Flux deployment — server, workers, storage, clients — and how a workflow run flows through them.

Is Flux a library or a system? Both, and the honest answer matters for how you reason about it.

The Python you write — @task, @workflow, ExecutionContext — is library code. You can pip install flux-core, decorate two async functions, and call workflow.run(...) in a script. That works, and it really is the same code that production runs.

But the durability guarantees described in Why durable execution — survive a crash, resume from where you left off, replay deterministically across worker hosts — only hold if the event log gets written to durable storage and a worker process is around to claim the next step. In production that means at least one server process, at least one worker process, and a database. Flux ships all three. They are part of the same package; you just run them.

This page is the orientation map. It names the four moving parts, shows what each one does, and traces a request through them. The follow-on pages — How workers work, Storage model, Workflows as services — go deeper on each.

The four moving parts

                                ┌────────────────────────────────┐
                                │           Clients              │
                                │                                │
                                │  Python SDK    CLI (flux …)    │
                                │  Browser UI    MCP clients     │
                                │  Agent UIs                     │
                                └───────────────┬────────────────┘
                                                │ HTTP / MCP
                                                ▼
                                ┌────────────────────────────────┐
                                │            Server              │
                                │   (FastAPI, 1..N replicas)     │
                                │                                │
                                │  • REST API (workflows,        │
                                │    executions, schedules,      │
                                │    workers, admin, services)   │
                                │  • Scheduler (in-process)      │
                                │  • Dispatcher (poll | event)   │
                                │  • MCP server (optional)       │
                                │  • Auth (OIDC + API keys)      │
                                └─────┬──────────────────┬───────┘
                                      │ writes           │ SSE dispatch
                                      ▼                  │ + REST checkpoints
                             ┌──────────────────┐        │
                             │     Storage      │        ▼
                             │                  │   ┌─────────────────┐
                             │  • Event log     │   │     Workers     │
                             │  • Catalog       │◄──┤                 │
                             │    (workflows)   │   │  • Claim work   │
                             │  • Workers       │   │  • Run @workflow│
                             │  • Schedules     │   │    + @task code │
                             │  • Secrets       │   │  • POST events  │
                             │  • Configs       │   │  • Heartbeat    │
                             │  • Agents        │   └─────────────────┘
                             │                  │
                             │  SQLite (dev)    │   ┌─────────────────┐
                             │  Postgres (prod) │   │ Output storage  │
                             │                  │   │ (artifacts)     │
                             │                  │   │                 │
                             │                  │   │ Local FS or S3  │
                             └──────────────────┘   └─────────────────┘

Four parts: clients, server, storage, workers. Clients always go through the server. Workers always go through the server for writes, and the server is the only thing that writes to durable storage. There is no message broker; workers receive dispatch over Server-Sent Events from the server itself. There is no separate scheduler daemon either: the scheduler runs inside the server process. And there is no leader-election component: when multiple server replicas share a PostgreSQL database, they coordinate through the database itself — advisory locks for the run-once pieces, SKIP LOCKED claims for dispatch, LISTEN/NOTIFY for cross-replica wakeups.

What the server is

The server is a FastAPI process. Its source lives in flux/server.py, which is a single large file (around 4,800 lines) full of route handlers. The FastAPI app gets constructed once in the start() method; the routes themselves are defined as nested functions decorated with @api.post(...) and @api.get(...). The runner is flux/servers/uvicorn_server.py, a thin wrapper that boots uvicorn with the right config.

You start it with:

flux start server

That single process exposes:

The scheduler is not a separate component. It is flux.schedule_manager.create_schedule_manager(...), started inside the server process. It polls the schedules table at scheduling.poll_interval (default 30 seconds), and when a schedule is due, it dispatches the same way a user-initiated run would. When several server replicas share one PostgreSQL database, each cycle is guarded by a session-scoped advisory lock (pg_try_advisory_lock): one replica dispatches, the others skip the cycle, and a dead holder’s lock auto-releases when its connection drops. Run state (next_run_at, last_run_at) is persisted per fire, so schedules neither re-fire on restart nor double-fire across replicas.

The dispatcher is the piece that hands claimed work to connected workers, and it has two modes ([flux.dispatch] mode). The default poll runs the legacy per-worker query loop; event runs one dispatcher task per replica that batch-claims work (up to batch_size, default 64) on wakeups — in-process signals plus PostgreSQL LISTEN/NOTIFY on the flux_work channel — with a safety-net tick every fallback_interval (default 15s) to cover missed notifications. Either way, claims use SELECT … FOR UPDATE SKIP LOCKED, so two replicas can never assign the same execution twice. Event mode is the scalable choice for large fleets on PostgreSQL; the trade-offs live on Dispatch modes.

The MCP server runs as a separate process (flux start mcp), proxying to the Flux HTTP server. It boots flux/mcp_server.py, which exposes registered workflows as MCP tools for AI clients (Claude Desktop, Cursor, etc.). Workflow-as-service support lives in flux/service_*.py modules; see Workflows as services for details.

Authentication is built in. Two providers ship out of the box: OIDC (flux/security/providers/oidc.py) and API keys (flux/security/providers/api_key.py). Most routes go through Depends(require_permission("workflow:{namespace}:{name}:run")) or a similar scope check, so the server enforces permissions at the route layer.

What a worker is

A worker is a long-running process that executes workflows. Source: flux/worker.py. You start it with:

flux start worker my-worker --server-url http://localhost:8000

A worker does five things in a loop:

  1. Register with the server using a bootstrap token. The server returns a session token; subsequent calls use that.
  2. Open an SSE connection to GET /workers/{name}/connect. The server pushes execution_scheduled, execution_resumed, execution_cancelled, and periodic ping events down that channel.
  3. Claim dispatched work via POST /workers/{name}/claim/{execution_id}. If two workers race for the same execution, the loser gets a 409 Conflict and drops the duplicate.
  4. Run the workflow. The server ships the workflow source down with the dispatch event, base64-encoded; the worker decodes it, exec-loads it into a synthetic module name (flux_workflow__<ns>__<name>__v<version>), and caches the compiled module for module_cache_ttl seconds (default 300). Then it invokes the workflow function with the ExecutionContext. Tasks inside the workflow do their work in the worker process.
  5. Checkpoint every event back via POST /workers/{name}/checkpoint/{execution_id}. The server is the one that writes events to durable storage; the worker never touches the database directly.

The worker also responds to ping events with POST /workers/{name}/pong (heartbeat — default every 10 seconds), handles reconnect with exponential backoff if the SSE stream drops, and emits a TASK_PROGRESS event via POST .../progress/{execution_id} when user code calls await ctx.progress(...).

Workers carry labels (--label gpu=true, --label region=us-east) that the dispatcher uses to match an execution to a worker. A workflow registered with @workflow.with_options(affinity={"gpu": "true"}) will only be dispatched to workers whose labels satisfy that match. The matching logic lives in flux/domain/resource_request.py::matches_labels. Workers also publish their resources (CPU, memory, disk, GPUs) at registration time, so resource requests via @workflow.with_options(requests=...) can be matched too — plus a capacity figure (max_concurrent_executions, default 16) that the server never assigns beyond. On SIGTERM a worker drains: it finishes running executions up to drain_timeout (default 60s) and flushes terminal checkpoints before exiting. See Worker capacity and drain.

A worker can also run inline, inside the same Python process as the client. workflow.run(...) called from a script auto-creates an in-process execution path, persists events to SQLite, and skips the server-and-worker hop. This is the path that examples in tests/examples/ use. Events are persisted to SQLite and can be resumed across process restarts via workflow.resume(execution_id). What’s lost on a process crash is the in-process state of any execution that was actively running — there’s no separate worker to pick it up.

See How workers work for the deeper picture: the claim queue, the module cache, the heartbeat lifecycle, eviction, and label-based dispatch.

Storage

Storage is three logical planes that share one physical database plus an optional artifact store.

The event log. The durable history of every execution. Every TASK_STARTED, TASK_COMPLETED, WORKFLOW_RESUMED, TASK_RETRY_FAILED, and so on lands as a row in the execution_events table (ExecutionEventModel in flux/models.py). This is the only thing that needs to survive a crash — replay reconstructs everything else from these events. Default backend is SQLite (sqlite:///.flux/flux.db per flux/config.py), which is single-node only: one server plus one colocated worker. For anything distributed, point Flux at PostgreSQL 14+ (psycopg v3 driver) via FLUX_DATABASE_URL=postgresql://...; the same schema works on both, and RepositoryFactory.create_repository() (in flux/models.py) dispatches on the URL scheme. The schema is Alembic-managed: opening the database migrates it to head automatically (guarded by a PostgreSQL advisory lock so concurrent replicas can’t race), and the flux db CLI (upgrade / current / history) exposes the same machinery for explicit control. See Upgrades and migrations.

The catalog. When you register a workflow, the server stores the source code (WorkflowModel) along with its version, namespace, resource requests, docstring, and — if the workflow’s input is a Pydantic BaseModel — the JSON Schema for the input type. The catalog also tracks workers (WorkerModel, with runtime info, resources, and labels), schedules (ScheduleModel), agents (AgentModel), API keys (APIKeyModel), roles, secrets (encrypted at rest), and configs. Everything except artifacts lives in the same database.

Output storage. Optional. When a task is decorated with @task.with_options(output_storage=LocalFileStorage()), the task’s return value is not embedded in the TASK_COMPLETED event — it is written out to a file (or S3 object) and the event carries an OutputStorageReference instead. The reference is an OutputStorageReference dataclass with storage_type, reference_id, and metadata fields. Two implementations ship: LocalFileStorage (writes to <home>/<local_storage_path>/<id>.<serializer>) and InlineOutputStorage (the default; embeds the value in the event itself). Source: flux/output_storage.py. S3 and other backends are user-implementable against the OutputStorage ABC.

Why split artifact storage out? Because a large task output (a generated PDF, an LLM transcript, a model artifact) would bloat the event log and slow every replay scan. Reading back a 50 KB event log to find one TASK_COMPLETED is fast; doing the same when each event carries a 5 MB blob is not.

See Storage model for the full schema, the event-table indexes, and the dispatch queries (ContextManager.next_execution, next_resume, next_cancellation) workers issue when claiming work.

Clients

The server is REST-shaped and the clients are whatever talks to it.

The Python SDK is the primary client. workflow.run(...) and client.run(...) are the two entry points. Under the hood they call the same POST /workflows/{ns}/{name}/run/{mode} route. The SDK is also how you define workflows in the first place — @task and @workflow are SDK constructs, not server APIs. So the SDK is both the authoring surface and the calling surface.

The CLI (flux/cli.py) is a thin wrapper over the same REST API. flux workflow run, flux execution show, flux schedule create — every command hits an HTTP endpoint. The CLI is what you use in CI and in shell scripts.

The browser UI for AI agents lives at flux/agents/web/index.html and flux/agents/ui/web.py; the terminal-mode agent UI is in flux/agents/ui/terminal.py and flux/agents/ui/textual_app.py. These are interfaces to the agent harness (flux/agents/) and the workflows it manages, not a general-purpose workflow UI.

The MCP server is for AI clients that speak MCP — Claude Desktop, Cursor, Claude Code. flux start mcp exposes the registered workflows as MCP tools so an AI client can list, describe, and invoke them. The MCP server is a separate process by default but draws from the same catalog as the REST API.

Topologies

Development. Run two processes: flux start server in one terminal, flux start worker in another. Storage is SQLite at .flux/flux.db. Output storage defaults to inline (no artifact files written). Total surface: one Python process for the server, one for the worker, one SQLite file. This is the topology the Quickstart sets up.

You can also run inline — one Python process, no separate server or worker, events persisted to SQLite directly. This is what tests/examples/ does. It exercises the full event/replay machinery, just without the cross-process dispatch.

Production. The server scales horizontally behind a load balancer: multiple FastAPI processes pointing at the same PostgreSQL database, coordinating through it — the scheduler and retention sweeps run as fleet-wide singletons per cycle via advisory locks, dispatch claims are SKIP LOCKED-safe, worker liveness is a persisted global view (workers.last_seen_at), and cross-replica wakeups travel over NOTIFY flux_work / flux_exec. The one routing requirement: a worker’s SSE stream lives on the replica it connected to, so the load balancer needs source-IP or cookie affinity for GET /workers/{name}/connect (round-robin is fine for everything else). Workers scale horizontally on their own; each connects to whichever server URL it is given and registers. Storage is Postgres (set FLUX_DATABASE_URL to a postgresql://... URL) and, for large task outputs, S3 (via a user-supplied OutputStorage implementation, or LocalFileStorage on a shared volume). See High availability for the deployment shape and Disaster recovery for the failure runbook.

Where each concept lives

A quick map of what is implemented where:

What to remember

Where this shows up