The execution model

How Flux executes a workflow — events, checkpoints, replay, and the contract between workflow code and the engine.

A Flux workflow is a Python async function, but the runtime treats it as something more specific: a deterministic reduction over an append-only event log. The function is the recipe; the events are the ledger of what actually happened. Every guarantee Flux makes about resume, retry, and crash recovery comes from keeping those two things in sync.

This page walks through how that works. The previous page, Why durable execution, argued that durable replay is necessary for long-running work. This one explains how Flux implements it.

The event log is the source of truth

Each workflow execution has an ExecutionContext. The context carries the workflow’s input, its current state, and — most importantly — a list of ExecutionEvent objects. The list is append-only during a run. Every meaningful transition produces one event.

The full set of event types lives in flux/domain/events.py. The workflow-level events are:

The task-level events are:

A successful first run of a tiny workflow with two tasks looks like this:

time
 │  WORKFLOW_SCHEDULED         (worker picks up the execution)
 │  WORKFLOW_CLAIMED
 │  WORKFLOW_STARTED            input = {"order_id": "A-123"}
 │  TASK_STARTED                fetch_order            args={"order_id":"A-123"}
 │  TASK_COMPLETED              fetch_order            value={"total": 49.00, ...}
 │  TASK_STARTED                charge_card            args={"total": 49.00, ...}
 │  TASK_COMPLETED              charge_card            value={"receipt": "r_abc"}
 ▼  WORKFLOW_COMPLETED          value={"receipt": "r_abc"}

Every line in that diagram is one row that will be persisted before the workflow is considered done. That persistence is the entire mechanism — there is no separate “checkpoint file” or “state snapshot.” The act of recording a TASK_COMPLETED event to durable storage is the checkpoint.

The workflow function is a deterministic reduction

When a workflow is scheduled, Flux constructs an ExecutionContext and invokes the workflow function with it. The function calls one task after another. Each await task(...) does two things, in this order:

  1. Compute a stable task_id from the task name and its arguments.
  2. Scan ctx.events for any prior TASK_COMPLETED or TASK_FAILED event matching that task_id.

On a fresh run, the scan finds nothing. The task body runs, emits a TASK_STARTED event, runs to completion, and emits TASK_COMPLETED carrying the task’s output. The output is also returned to the workflow function, which then proceeds to the next await. Once the workflow function returns, the runtime appends WORKFLOW_COMPLETED.

On a resume, the scan finds something. The task body is not re-entered. Flux deserializes the stored output, returns it directly, and the workflow function moves on as if the task had just run. From the workflow’s perspective, there is no observable difference between “the task just ran” and “the task ran two days ago on a different worker.” That is the whole trick.

This is why we call the workflow a deterministic reduction over the event log. The events are the facts. The workflow function is the rule for combining those facts into the next step. Given the same event prefix and the same workflow code, the function will always reach the same next await — and that next await is what determines what gets recorded next.

First run vs. replay

The same code path handles both. On every await task(...), Flux looks for a matching completion event before running the task. The lookup happens in task.__call__:

events_for_this_task_id = [
    e for e in ctx.events
    if e.source_id == task_id
    and e.type in (TASK_COMPLETED, TASK_FAILED)
]
if events_for_this_task_id:
    return events_for_this_task_id[0].value   # replay
# otherwise, run the task body

The difference between “first run” and “replay” is just whether the lookup hits. There is no separate replay mode and no TASK_REPLAYED event — replay is silent at the task level. The only thing that distinguishes a resumed run in the log is the WORKFLOW_RESUMING and WORKFLOW_RESUMED events that bracket the resume itself.

Here is what a resume looks like for the same workflow above, where the worker crashed after fetch_order completed but before charge_card ran. Events from the first run are still in the log; resume appends the new ones:

time
 │  WORKFLOW_SCHEDULED              ← first run starts
 │  WORKFLOW_CLAIMED
 │  WORKFLOW_STARTED                input = {"order_id": "A-123"}
 │  TASK_STARTED   fetch_order
 │  TASK_COMPLETED fetch_order      value={"total": 49.00, ...}
 │  ...                             ← worker dies here, before charge_card runs
 │
 │  WORKFLOW_RESUMING               ← second worker picks up the execution
 │  WORKFLOW_RESUME_SCHEDULED
 │  WORKFLOW_RESUME_CLAIMED
 │  WORKFLOW_RESUMED
 │  TASK_STARTED   charge_card      ← fetch_order is NOT re-run; its TASK_COMPLETED
 │  TASK_COMPLETED charge_card        is already in the log and Flux returns it
 ▼  WORKFLOW_COMPLETED

The function runs from the top on resume. It hits await fetch_order(...), finds the existing TASK_COMPLETED, gets the recorded output back, and moves on. It hits await charge_card(...), finds nothing, and actually runs the task. The function does not skip forward. It re-executes everything between the awaits — the variable assignments, the conditionals, the loops, the helper calls — because that is what produces the next await correctly.

Determinism matters because of that re-execution between awaits.

Why determinism is load-bearing

Consider this workflow:

import random
from flux import ExecutionContext
from flux.task import task
from flux.workflow import workflow

@task
async def variant_a(): ...

@task
async def variant_b(): ...

@workflow
async def pick_a_path(ctx: ExecutionContext):
    if random.random() < 0.5:
        return await variant_a()
    else:
        return await variant_b()

On the first run, suppose the branch picks variant_a. A TASK_COMPLETED event is recorded with source_id derived from the name variant_a and its arguments.

On resume, the workflow function runs from the top again. It calls random.random() again, which returns a different number. This time the branch picks variant_b. Flux looks for a TASK_COMPLETED event matching variant_b’s task_id — and finds none. So it runs variant_b from scratch. The recorded TASK_COMPLETED for variant_a is now an orphan: an event in the log that no path through the code is asking about anymore.

The workflow’s behavior is now non-replayable. Worse, the user gets one outcome on the first run and a different outcome on resume.

The fix is not to outlaw randomness — Flux ships flux.tasks.choice, flux.tasks.randint, and flux.tasks.uuid4 for exactly this reason. The fix is to put the non-deterministic step inside a task, because task outputs are recorded:

from flux.tasks import choice
from flux import ExecutionContext
from flux.workflow import workflow

@workflow
async def pick_a_path(ctx: ExecutionContext):
    path = await choice(["a", "b"])
    if path == "a":
        return await variant_a()
    else:
        return await variant_b()

Now choice is a task. Its output is captured in a TASK_COMPLETED event on the first run. On resume, the cached output is returned — so the branch picks the same path it picked the first time, and variant_a (or variant_b) is found in the log exactly where it was recorded.

The same rule applies to datetime.now(), reading environment variables, hitting a network resource, or anything else whose value can shift between runs. The next page, Determinism, goes through the full constraint and the standard escape hatches.

What happens when a task fails

A task body can throw. The runtime catches the exception in task.__call__ and runs through three escape hatches in order, each of which produces its own events:

  1. Retry. If the task was decorated with retry_max_attempts > 0, Flux retries the body up to that many times with exponential backoff. Each attempt emits TASK_RETRY_STARTED and then either TASK_RETRY_COMPLETED (with the output) or TASK_RETRY_FAILED. If a retry succeeds, the surrounding lookup-and-return path still produces a final TASK_COMPLETED for the task.

  2. Fallback. If retries are exhausted (or there were none) and the task has a fallback= handler, Flux invokes the handler and emits TASK_FALLBACK_STARTED followed by either TASK_FALLBACK_COMPLETED or TASK_FALLBACK_FAILED. A successful fallback also produces a TASK_COMPLETED for the original task — the workflow function gets the fallback’s return value as if nothing had gone wrong.

  3. Rollback. If there is no fallback and the task has a rollback= handler, Flux invokes it and emits TASK_ROLLBACK_STARTED followed by either TASK_ROLLBACK_COMPLETED or TASK_ROLLBACK_FAILED. Rollback is meant for cleanup — it runs after the task has already failed terminally, and a TASK_FAILED event is emitted in addition.

The chain — retry, then fallback, then rollback — is recorded in the event log just like everything else. That means a resumed workflow can pick up mid-recovery: if the worker died after TASK_RETRY_COMPLETED but before WORKFLOW_COMPLETED, the resume sees the retry success and never re-enters the task. See Retry policy, Fallback, and Rollback and compensation for the API and the patterns.

The checkpoint is not a separate thing

It is tempting to imagine a “checkpoint” as a snapshot file written out alongside the events. There isn’t one. A checkpoint, in Flux, is just the moment a TASK_COMPLETED (or TASK_FAILED, or any other event) is durably persisted to the event store. The ctx.checkpoint() call that runs at the end of every task is exactly the I/O that flushes new events through the registered checkpoint callable — over HTTP to the server in distributed mode, or directly to the database when running inline.

That is the entire durability mechanism: append an event, fsync it. On the next start, the event is there. If the process crashed before the fsync, the event is not there and Flux will treat that task as not-yet-completed when the workflow resumes. This is why idempotency matters for any task that touches an external system: the narrow window between “task body returned” and “event is fsynced” is exactly where a duplicate side effect can hide.

Where the events live

In dev, the event store is SQLite at sqlite:///.flux/flux.db (the default database_url in flux/config.py). In production, you would point Flux at PostgreSQL — the same schema works under either backend, and flux/models.py defines the tables for both. The ExecutionEventModel row is what gets written when ctx.checkpoint() runs.

We will not labor over the storage layout here; the Storage model page covers the schema, the indexes, and how the worker queries the event store on claim.

What to remember

Where this shows up