Why durable execution

The argument for durable execution as a design pattern, and where Flux fits in it.

We built Flux because we kept writing the same broken thing.

A workflow that charges a customer, records the charge in the database, sends a receipt email, and updates an analytics counter. Four steps, three external services — the kind of thing every backend has a hundred of. The first version is a Celery task with a try/except. The second has an idempotency key. The third has a status column on a payments table — pending, charged, recorded, notified, complete — and a five-minute reconciliation cron that pushes stuck rows forward. By the fourth version you’re maintaining the same state machine in two places: once in the task code, once implicitly in whichever rows the cron decides to touch. When something breaks at 3am, the on-call engineer’s job is to read the logs, guess which step actually ran, and write a one-off script to push the row through.

If you’ve shipped a non-trivial backend, you already know this failure mode.

The shape of the problem

Walk that payment workflow through.

Step one charges the card via Stripe. It succeeds — Stripe takes the money, returns a charge ID. Step two inserts a row into payments recording the charge. The database connection times out. The transaction never commits. The exception propagates. The task gets re-queued. Celery retries it.

Now you have a problem. The card has been charged. The row has not been written. The retry will charge the card again, because nothing in your code knows the previous attempt got past step one. So you add an idempotency key, generate it from the order ID, and rely on Stripe to dedupe. Good — solved at one boundary.

But step three sends an email through a provider that has no idempotency keys. Step four hits an internal analytics service that does, but only if the worker sends the same key it sent last time, which means storing the key somewhere durable, which means another table. And step two needs a check that says “if a row already exists for this order, continue silently.”

Three patches in, the workflow has logic that exists for one reason: to make the code idempotent across the retry boundary the queue introduced. None of it does anything for the business. It exists to compensate for the queue having no idea what your code already did.

Now compound this. Long-running batches that die at step 17 of 200 and need to resume from step 18. Retry loops that turn into duplicate side effects when the retry hits a step the original run already completed. Cancellation requests that require finding the queue message, killing it, and writing a script to undo whatever already got written. State in three places — database, queue, worker memory — none of which can be trusted on its own.

What you’re already doing

Before they reach for a workflow engine, most teams handle this with a stack of well-understood techniques:

This works. For a long time, it works fine. We’re not telling you to throw it out — if your workload is small, your steps are short, and your operators are patient, you can ride this stack for years.

The cost shows up when one of those three things changes. The workload grows, the status table grows, the reconciliation job gets slower. The steps get longer — a six-hour batch and a worker restart in hour four means starting from scratch unless someone wrote checkpoint logic for that one job. Or the operators run out of patience, because reading logs and guessing which step ran is not a hobby anyone enjoys at 3am.

Where it breaks

The deeper problem is structural. With cron + queues + try/except, the state of an in-flight workflow lives in three places at once: the database row that says what status the job is in, the queue message that says what work is left, and the worker’s memory holding local variables and partial results.

Crashes happen at the boundaries between these. The row says processing but the queue message is already acked. The queue thinks the job is in flight but the worker process died. The worker has results in memory that never got written down. You spend time synchronizing the three views, and when they diverge, recovery means reading the database, reading the logs, and reasoning about what must have happened.

Observability has the same shape. You can’t ask “what state is this execution in?” and get a precise answer — you can only ask the database for the row’s status field and trust it got updated when it should have been. Anything not explicitly logged is gone.

Recovery is the worst part. When the system gets stuck, the fix is almost always to write a script — a one-off Python file that reads the row, decides what step needs to run next, calls the right function with the right arguments, and updates the status. The script is the missing logic that should have been in the workflow itself.

What durable execution is

Durable execution A design pattern where a workflow's progress is persisted as it runs, so it can survive process crashes and resume from where it stopped instead of restarting. Flux implements it by recording each task result to an event log and replaying the log on restart. Full definition → makes the whole class of problems go away by making one decision: the execution itself is what gets persisted.

Not the row. Not the queue message. The execution. Every step the workflow takes, every result it produces, every retry it attempts, every external call it makes — recorded to a durable event log as it happens. The workflow function is not the source of truth about what ran; the event log is.

That single decision changes everything downstream:

The pattern predates Flux by a long way — event sourcing applied to workflow execution. Temporal is the best-known implementation. AWS Step Functions is a managed take on it. The pattern is not ours; what’s ours is the surface we put on it.

What Flux is in this pattern

Flux is durable execution where the workflow is a Python function.

from flux import ExecutionContext, task, workflow

@task
async def charge_card(order_id: str) -> str:
    return await stripe.charge(order_id)

@task
async def record_payment(order_id: str, charge_id: str) -> None:
    await db.insert(order_id, charge_id)

@task
async def send_receipt(order_id: str) -> None:
    await email.send(order_id)

@workflow
async def process_order(ctx: ExecutionContext[str]):
    charge_id = await charge_card(ctx.input)
    await record_payment(ctx.input, charge_id)
    await send_receipt(ctx.input)

That’s the entire interface for the example we walked through at the top. No status table, no idempotency keys, no reconciliation cron. Each await on a @task produces an event in the durable log. If the worker dies between charge_card and record_payment, the next worker reads the log, sees charge_card completed with charge ID ch_xyz, and resumes by calling record_payment("ch_xyz") directly — charge_card is not re-run because the log says it already finished.

Under the hood: when the workflow function is replayed, each @task call checks the event log for a matching completed event before doing any work. If one is found, the recorded value is returned immediately; the function body never runs. The TASK_STARTED, TASK_COMPLETED, TASK_FAILED, TASK_RETRY_STARTED, and related events are defined in flux/domain/events.py and recorded by flux/task.py on every state transition. Workflows have their own lifecycle events — WORKFLOW_STARTED, WORKFLOW_COMPLETED, WORKFLOW_PAUSED, WORKFLOW_RESUMING — recorded by flux/workflow.py around the function body.

No DSL. No XML. No separate JSON definition file. The workflow is the function. The graph is whatever your code produces at runtime — branches, loops, fan-out, conditional steps — and the engine records what actually happened, not what the graph declared in advance.

The tradeoff, honestly

Durable execution is not free. Three costs are real.

Determinism. A workflow function will be replayed from the top during recovery, and on replay the recorded results stand in for the side effects that already happened. This means workflow bodies have to be deterministic with respect to those replays — you can’t read the system clock directly, generate random numbers inline, or call out to external services from the workflow body. You move those into tasks, which is where Flux records them. This is a learned discipline, not a heavy one, but it is real, and you will trip over it the first few times.

Infrastructure. Distributed Flux needs a server (FastAPI process holding the catalog and execution state) and at least one worker. SQLite is fine for development; production wants Postgres. This is more moving parts than a celery worker command. You also need to think about worker capacity, scheduling, and resource matching once you have a few different workflow types.

Learning curve. The mental model — tasks as recorded steps, workflows as orchestration, replay semantics, pause and resume — is unfamiliar if you’ve only done queue-based work. The first workflow takes longer to write than the first Celery task did. The second one does not.

And there are workloads where durable execution is the wrong choice:

The dividing line is whether the question “did this actually complete, and what state is it in right now?” is one you need to answer in production. If yes, you want durable execution. If you’re never going to ask that question, you don’t.

The choice is also not all-or-nothing within Flux. A workflow can opt out of task-level persistence with durability="transient" — it keeps the orchestration surface, dispatch, and terminal state, but skips every intermediate checkpoint and runs at-most-once. That is the right shape for high-frequency AI mesh hops and hot paths where replay is worthless and the event volume is the cost. See Durable vs transient workflows.

What to remember

If you’ve ever written a rerun script, you’re paying the cost of not having durable execution. The cost just shows up as on-call pages instead of a line item.

Where this shows up