Storage model

A running Flux cluster has a lot of moving parts: a server, one or more workers, scheduled jobs, and the workflow code itself. State lives in only three places, and once you know which is which, “what survives a crash?” stops being a guessing game.

This page names those three planes, walks each one against the source, and then traces a single workflow run through all of them. It is the prerequisite for any conversation about backups, retention, or migrations.

Three storage planes

Flux persists state across three planes, and only three:

1. The event log — every state transition of every execution. This is the source of truth that powers replay and crash recovery.
2. Output storage — the actual return values of tasks and workflows, when those values are large enough to keep out of the event log.
3. The workflow catalog — the source code and metadata of every registered workflow.

The event log and the catalog live in the same relational database. Output storage is a separate plane with its own backend. Anything not in one of those three places — a running task’s local variables, a worker process’s memory, a half-formed HTTP request — is volatile by design.

Each plane has a different access pattern, a different growth curve, and a different backup story. Treating them as one undifferentiated “Flux database” is the easiest way to misjudge any of them.

Plane A: The event log

Every workflow execution is a sequence of ExecutionEvent rows. The schema is defined in flux/models.py as two related tables:

• executions (ExecutionContextModel) — one row per workflow execution, holding execution_id, workflow_id, workflow_namespace, workflow_name, state (an ExecutionState enum), the serialized input and output, the worker_name that owns the run, and a short-lived exec_token used for callbacks.
• execution_events (ExecutionEventModel) — one row per state transition, foreign-keyed back to executions.execution_id with ON DELETE CASCADE. Each event row carries an autoincrementing id, the source_id (the task or workflow id the event belongs to), an event_id, a type from the ExecutionEventType enum, a name, a time, and a value column where the task’s return value (or error payload) is stored as a PickleType blob serialized with dill.

The events are ordered by their integer id. Replay reads them in that order, so the event log records exactly what happened in this execution, in what order, and what each step produced.

By default, both tables live in SQLite. flux/config.py sets database_url = "sqlite:///.flux/flux.db" as the default — a file under the configured home directory (.flux by default). For production, swap that to a PostgreSQL URL via database_url in flux.toml or via the FLUX_DATABASE_URL environment variable; the database_type field auto-infers from the URL prefix. SQLite gets a few pragmas applied on connect (WAL journal mode, normal synchronous, an mmap region); PostgreSQL gets a pre-pinged connection pool whose size and recycle interval are also config-driven.

The same database also holds smaller satellite tables that travel with the event log: workers and its children (worker_runtimes, worker_resources, worker_packages), schedules, services, secrets (AES-GCM encrypted), and configs. None of these are conceptually part of the event log — they are operator-facing tables that happen to share the same SQLAlchemy Base.metadata.

There is no separate “checkpoint file.” The act of inserting a TASK_COMPLETED row into execution_events is the checkpoint. If the database has the row, the work is durable. If it doesn’t, the work is lost — there is nothing else to fall back on.

The event log grows without bound by default — every task of every execution is a persisted row. An optional retention job ([flux.retention] in configuration: enabled = false by default, retention_days = 30, sweep_interval = 3600, batch_size = 500) deletes terminal executions and their events once they age out. It is off by default so upgrades never silently remove history; production deployments should enable it. See Retention.

Plane B: Output storage

Inputs and outputs are normally stored inline on the executions row (the input and output columns, both PickleType). For small results — a status string, a count, a short list — inline storage is the right default. For large results, it wastes database space, slows queries, and inflates backups.

flux/output_storage.py introduces a side channel for those large results. Tasks and workflows can declare output_storage=... on their decorator, and when the returning value comes back, Flux calls storage.store(reference_id, value) and stores only the returned OutputStorageReference — storage_type, reference_id, and a metadata dict — back into the event log.

Two backends ship with Flux:

• InlineOutputStorage — the default, equivalent to no offload. The value goes into the reference’s metadata["value"].
• LocalFileStorage — writes the serialized value to disk and stores only the reference in the database. Files live at Path(settings.home) / settings.local_storage_path — .flux/.data by default. The serializer is settings.serializer, which is pkl (a dill pickle) out of the box; setting it to json writes .json files instead.

For a workflow’s own return value, the reference_id is set in flux/workflow.py as f"{ctx.workflow_name}_{ctx.execution_id}", so files end up named {workflow_name}_{execution_id}.pkl (or .json). Tasks construct their own reference ids the same way — the reference_id is the only handle the event log holds; the data lives entirely outside it.

OutputStorage is an abstract base class with three methods: store, retrieve, and delete. An S3-backed or GCS-backed implementation is a straightforward subclass. The reference’s metadata records the serializer that was used at write time, so retrieval works even if the cluster’s default serializer is later changed.

The build-side reference for the API is Workflow inputs and outputs.

Plane C: The workflow catalog

The catalog stores the source code of every workflow the server knows about. It lives in the workflows table (WorkflowModel), implemented by DatabaseWorkflowCatalog in flux/catalogs.py.

Each row holds:

• id, namespace, name, version — identity. (namespace, name, version) is unique; the catalog increments version on every fresh flux workflow register for the same (namespace, name) pair.
• source — the raw bytes of the workflow file, stored through a Base64Type column that dill-serializes and base64-encodes the bytes before they hit the database.
• imports — the module-level imports discovered by AST parsing the file.
• requests, affinity, wf_metadata — resource requests, label affinity, and the extracted metadata (workflow docstring, input schema if the input is a Pydantic BaseModel, list of task names called, nested workflows, secret requests).

Workers do not need a separate filesystem sync. When a worker claims an execution, the server sends it the catalog entry, the worker decodes the source, exec-loads it under a synthetic module name (flux_workflow__<ns>__<name>__v<version>), and caches the compiled module for workers.module_cache_ttl seconds (300 by default). Version bumps invalidate the cache because the synthetic module name includes the version.

Catalog rows are mutated only by flux workflow register and flux workflow delete. They are read on every execution dispatch. For the auto-registration path (workflow.run(...) from an inline script), the workflow registers itself on first call via _ensure_registered — same table, same code path.

Configuration recap

The defaults that govern the three planes all live in flux/config.py:

Override any of them via environment variable (FLUX_DATABASE_URL, FLUX_LOCAL_STORAGE_PATH, …), via flux.toml, or via [tool.flux] in pyproject.toml. Environment variables take precedence; defaults are applied last.

What is not durable

Anything not in the event log, the output storage, or the catalog is volatile. In particular:

• The Python state of a running task. Local variables, in-progress I/O, partially constructed objects — none of it is saved. A task that crashes mid-execution restarts from the last TASK_COMPLETED event the workflow recorded.
• Worker process memory. Cached compiled modules, in-flight HTTP requests, and the worker’s own claim queue all evaporate when the process dies. The worker’s identity and labels (workers table) survive; its working set does not.
• Queues. There is no broker. Workers pull from the server over an SSE stream, and the server makes dispatch decisions by querying the event log directly (ContextManager.next_execution). State that would live in a queue elsewhere is implicit in the state column of the executions table.

The corollary: every Flux durability guarantee can be expressed as a property of the event log. “Resumable on crash” means the event log has a TASK_COMPLETED for the last successful step. “Idempotent retry” means the event log already contains the events for steps that finished, so replay short-circuits them.

A workflow run, traced through storage

Putting the three planes together, here is what gets written and where during one execution:

1. Registration. flux workflow register hello.py parses the file, extracts the @workflow functions, and inserts (or upserts) rows into the workflows table. The source code is base64-encoded; the docstring and input schema are pulled out and stored alongside.
2. Run requested. flux workflow run hello (or workflow.run(...) inline) creates an executions row with state = CREATED, then appends a WORKFLOW_SCHEDULED event to execution_events.
3. Worker claims. A worker matching the resource and affinity constraints calls POST /workers/{name}/claim/{execution_id}. The server flips the executions.state to CLAIMED, sets worker_name, and appends WORKFLOW_CLAIMED.
4. Execution begins. The worker decodes the catalog source, loads the module, and runs the workflow function. WORKFLOW_STARTED is appended.
5. Tasks run. For each task, TASK_STARTED is appended when the task begins, TASK_COMPLETED (with the return value in the value column) when it finishes. Retries, fallbacks, and rollbacks each emit their own event types.
6. Large outputs land elsewhere. When a task or workflow declares output_storage=..., Flux serializes the value to plane B and records an OutputStorageReference (storage type plus reference id) on the event row instead of the raw value.
7. Workflow completes. WORKFLOW_COMPLETED is appended; the executions.state flips to COMPLETED; the workflow’s final output is stored on executions.output (inline) or as a reference (if output_storage was set on the workflow).

A resume reads the event log for the execution, fast-forwards through the events that already exist, and only re-enters the workflow at the first task without a TASK_COMPLETED.

Inspecting state

The CLI is the supported access path:

• flux execution list — list executions with their states.
• flux execution show <execution_id> --detailed — the full event log for one execution.
• flux workflow versions <namespace>/<name> — the catalog rows for one workflow.

Direct SQL access against the event-log database works (sqlite3 .flux/flux.db or psql against your Postgres) and is occasionally useful for debugging, but the schema is internal — names and types change between releases.

Backups and migrations

Backing up a Flux cluster is backing up its three planes:

• The database holds plane A (event log) and plane C (catalog). For SQLite, that is a single file under home; copy it with the server stopped, or use sqlite3 .flux/flux.db ".backup". For PostgreSQL, standard pg_dump works — Flux uses no Postgres extensions or unusual types beyond the pickled-blob columns.
• Output storage is backed up separately. For LocalFileStorage that means whatever directory home/local_storage_path resolves to. For a custom S3 backend it means bucket versioning, replication, or scheduled snapshots.
• The workflow source on disk is independent of all of the above — the catalog already holds the source the cluster is actually executing.

Schema migrations are Alembic-managed and ship inside the package (flux/migrations/). On first connect — server, worker, or inline run — Flux brings the database to the latest revision automatically: a fresh database gets the full migration chain, an Alembic-managed one gets any pending migrations, and a legacy pre-Alembic database (tables present but no alembic_version) is stamped at the baseline revision and upgraded in place with data preserved. On PostgreSQL the run is serialized across concurrent replicas by an advisory lock. The flux db upgrade / flux db current / flux db history commands give operators explicit control over the same machinery — see Upgrades and migrations and the flux db reference.

What to remember

• State lives in three places: the event log, output storage, and the workflow catalog. Everything else is volatile.
• The event log and the catalog share one database; output storage is a separate plane.
• The default database is SQLite at .flux/flux.db; the default output store is InlineOutputStorage (values live on the event row); LocalFileStorage writes to .flux/.data when opted in. Both are configured in flux/config.py.
• Backups are a per-plane concern. Schema migrations are Alembic-managed and run automatically on first connect.
• The event log grows without bound unless the [flux.retention] job is enabled.

Where this shows up

• System architecture — how the server, scheduler, and workers interact above these planes.
• The execution model — why the event log is enough to replay a workflow.
• Workflow inputs and outputs — the API for redirecting outputs to plane B.
• Operate: Storage — production storage backends and tuning.
• Operate: Backups — backup and disaster-recovery procedures.