Determinism

What makes a Flux workflow deterministic, what breaks it, and how to keep your code on the right side of replay.

A Flux workflow can be paused after twenty steps, persisted to disk, and resumed two days later on a different worker. The mechanism that makes this work — replaying past events from the log instead of re-executing past tasks — only holds together if the workflow body itself produces the same decisions on the second pass that it produced on the first. That property is Determinism The property that a workflow body produces the same calls in the same order every time it replays. Required for resume-after-pause to work correctly. Full definition → , and it is one of the most common sources of confusion for people new to durable execution.

This page defines what determinism means for a Flux workflow, lists the everyday Python idioms that quietly break it, and shows the pattern that fixes them.

What “deterministic” means here

Flux records every task call as an ExecutionEvent. When a workflow resumes, the runtime walks the workflow function again from the top. At each task call, it looks for a matching TASK_COMPLETED event in the log. If one exists, it returns the recorded output without entering the task body. If one does not, it executes the task for real and records a new event.

The match check ties a task call to a task_id derived from the task name and its arguments (see flux/task.py, around line 139, where the id is built as f"{full_name}_{abs(hash((full_name, make_hashable(task_args), make_hashable(args), make_hashable(kwargs))))}"). The lookup then scans events for source_id == task_id. The implication: if your workflow code computes a different argument on the second pass than it did on the first, the task_id will not match, no event will be found, and the task will execute again.

That is the operating constraint. Tasks are recorded; the workflow body is replayed. The workflow body must therefore reach the same task calls with the same arguments every time it runs against a given input and event log. Same inputs plus same event log must yield the same execution path.

What breaks determinism

The pattern is always the same: the workflow body reads something that varies from process to process, and that read flows into a branch decision or into an argument passed to a task. A few concrete forms:

Wall-clock reads in the workflow body.

import time
from datetime import datetime

@workflow
async def report(ctx: ExecutionContext):
    started = time.time()              # different on every replay
    if datetime.now().hour < 12:       # might be morning, might be evening
        return await morning_report()
    return await evening_report()

The first run calls morning_report; a resume at 3 p.m. calls evening_report. Both produce TASK_COMPLETED events, but for different tasks. The execution log no longer reflects a single coherent run.

Random numbers in the workflow body.

import random

@workflow
async def maybe_email(ctx: ExecutionContext):
    if random.random() < 0.1:          # different roll on every replay
        await send_email(ctx.input)

A 10% branch on the first pass becomes a 90% non-branch on a replay where the dice land differently. The send happens once or zero times — or twice — depending on how unlucky you are.

Direct external calls from the workflow body.

import httpx

@workflow
async def enrich(ctx: ExecutionContext):
    user = httpx.get(f"https://api.example.com/users/{ctx.input}").json()
    return await score_user(user)

This bypasses the event log entirely. Flux does not see the HTTP call, so the response is not recorded. On replay, the call runs again, returns possibly different data, and the workflow proceeds down a possibly different path. Worse: if the external service is down at replay time, the workflow fails for a reason that had nothing to do with the original failure.

UUIDs generated in the workflow body.

import uuid

@workflow
async def create_order(ctx: ExecutionContext):
    order_id = uuid.uuid4().hex       # new value on every replay
    return await provision_order(order_id)

order_id differs every pass. The task_id for provision_order therefore differs every pass. The recorded event from the first run is never matched, and provision_order runs again as if nothing happened — exactly the situation that durable execution exists to prevent.

Environment reads.

import os

@workflow
async def deploy(ctx: ExecutionContext):
    env = os.environ.get("DEPLOY_ENV", "staging")  # could change between runs
    return await ship(env)

The environment that the first worker saw is not the environment the second worker sees. Read-once-and-pass-down doesn’t help either if the read happens in workflow code: the resumed worker performs the read again, against its own environment.

Filesystem reads. Same shape: the file that was on disk during the original run may not be on disk on the worker that picks up the resume. If it is, its contents may have changed.

The fix: push it into a task

Every non-deterministic value should be produced by a task. Tasks have their outputs recorded; the workflow body sees only the recorded value on replay.

from flux.tasks import now, uuid4, randint
from flux.task import task

@task
async def get_env(name: str) -> str:
    import os
    return os.environ.get(name, "")

@task
async def fetch_user(user_id: str) -> dict:
    import httpx
    return httpx.get(f"https://api.example.com/users/{user_id}").json()

@workflow
async def enrich(ctx: ExecutionContext):
    started = await now()                  # datetime recorded once
    correlation_id = await uuid4()         # UUID recorded once
    env = await get_env("DEPLOY_ENV")      # env value recorded once
    user = await fetch_user(ctx.input)     # response recorded once
    return await score_user(user, correlation_id, started)

Every value the workflow body reads from the outside world is now a task call. On a replay the runtime walks the workflow, hits each await, finds a matching TASK_COMPLETED, and returns the recorded output. The branch decisions downstream stay identical because their inputs stay identical.

flux.tasks ships built-ins for the common cases: now, uuid4, randint, randrange, choice, and sleep. They are ordinary @task functions that wrap datetime.now(), uuid.uuid4(), random.randint, and so on. Use them in preference to rolling your own — they are tested, and the names make the intent obvious to a reader.

The examples/determinism.py walkthrough

The shipped example is short on purpose:

from flux import ExecutionContext
from flux.workflow import workflow
from flux.tasks import now, uuid4, randint, randrange

@workflow
async def determinism(ctx: ExecutionContext):
    start = await now()
    await uuid4()
    await randint(1, 5)
    await randrange(1, 10)
    end = await now()
    return end - start

Every line that touches a non-deterministic source is an await. The workflow body itself has no clock reads, no random calls, no uuid.uuid4(). Run it inline, then call workflow.resume(ctx.execution_id): the returned timedelta is identical to the first run. The start and end values were recorded as TASK_COMPLETED events; on replay they come back unchanged. Without the task wrappers — if start = datetime.now() sat in the workflow body — end - start would compute against a different start on every resume, and the return value would silently drift.

This is the pattern: find the non-determinism, wrap it in @task, and await it.

Subtle cases worth knowing about

Dict and set iteration order in older Python. Dict ordering is insertion-stable starting with CPython 3.7 (guaranteed by the language spec from 3.7+). If you target 3.7 or newer — and Flux requires Python 3.14 — iteration over a plain dict literal or comprehension is deterministic within a process. Sets are a different story: set iteration order depends on element hashes, which for strings are randomized per process. Iterating a set in the workflow body and feeding the items to tasks one by one is a determinism trap.

Hash randomization and PYTHONHASHSEED. Python hashes for strings, bytes, and a few other types are randomized at interpreter startup by default. The task_id Flux computes for a task call passes through hash(...) (see flux/task.py around line 139). Within a single process, that hash is stable; across processes, it is not, unless PYTHONHASHSEED is fixed to a specific value. In practice this means a workflow that is started on one worker and resumed on another can produce a different task_id for the same logical call, fail to match the recorded event, and re-execute the task. If you observe duplicate task execution on resume across workers, check whether PYTHONHASHSEED is set consistently in your deployment.

Sets passed as task arguments. A related trap: when a workflow builds a set and passes it as an argument, make_hashable converts it to a frozenset, which normalizes order for the hash input. So {"a", "b"} and {"b", "a"} produce the same task_id. But iterating that set in the workflow body before passing it down does not — the iteration order is still process-dependent, and any per-element task call inherits that order.

Where the determinism contract lives

Replay walks the workflow function and matches each await against the recorded event log by task_id. A mismatch is treated as “no recorded result for this call — run it for real” rather than “this is suspicious — fail loudly.” That keeps the model predictable: the workflow body is the contract, and the contract is don’t put non-determinism in it.

Two implications follow. First: a workflow that breaks the contract behaves like a normal bug — sometimes a duplicate side effect, sometimes a different return value, sometimes a crash if the wrong branch hits an unexpected state. Treat workflow drift the same way you’d treat any other correctness regression.

Second: keeping workflow bodies small is the best mitigation. The less code sits between the @workflow decorator and the first await, the less surface there is for a non-deterministic line to creep in. Put orchestration in the workflow. Put everything else in tasks.

Determinism vs idempotency

Determinism and idempotency are neighboring properties, often conflated. They are not the same.

Determinism is a property of the workflow body: given the same input and the same event log, the workflow function takes the same path. It is what makes replay possible.

Idempotency is a property of individual tasks: given the same call, the task produces the same observable effect on the outside world, no matter how many times it runs. It is what makes retries safe.

A workflow can be deterministic without its tasks being idempotent. A task can be idempotent without the workflow being deterministic. Both properties are needed, for different reasons: determinism so the workflow can replay; idempotency so a task that ran-but-did-not-checkpoint can run again without doubling its side effects.

The Idempotency page covers the task side of this in detail.

What to remember

Where this shows up