Execution runners

Every execution a worker claims runs through a runner — a pluggable strategy (Prefect-style) that decides where the workflow code actually executes. Flux ships three: subprocess (the default), inprocess, and docker. The runner changes isolation, latency, and crash behaviour; it does not change your workflow code, the event log, or the checkpoint protocol.

This page covers the developer-side surface: what each runner gives you, how to pin a workflow to one, and how dispatch enforces it. For fleet configuration (which runners a worker enables, container images, memory limits), see the worker-side keys in Server and worker settings.

The three runners

subprocess (default)

Each execution runs in its own credential-less child process (python -m flux.runners.child). The worker streams checkpoints, progress updates, secret and config requests, and approval-gate operations to and from the child over a stdio pipe, so the server-facing protocol is identical to in-process execution.

What the subprocess runner buys you:

• Fault isolation. A crash, OOM, or event-loop-blocking call in workflow code cannot take down the worker or its other running executions.
• A sanitized environment. The child never sees the worker’s credentials: the bootstrap token, every FLUX_SECURITY__* variable, and FLUX_DATABASE_URL are stripped before spawn. The only credential in the child is the short-lived, single-execution token used for call() hops.
• Enforceable cancellation. Cancellation and drain send SIGTERM, wait subprocess_term_grace seconds (default 10.0) for the child to finish its cancellation handling, then SIGKILL — which works even against code stuck in synchronous C calls.
• An optional memory ceiling. subprocess_memory_limit (Linux only, bytes, 0 = unlimited) bounds each child’s address space.

The cost is process spawn plus imports: roughly 0.55–0.7 s of overhead per execution, and each concurrent child is a full Python process (~50–100 MB baseline plus workflow memory). Concurrency amortizes the spawn cost; size worker capacity against memory accordingly (see Capacity and drain).

inprocess

The workflow runs as a task on the worker’s event loop. Lowest latency (~0.1 ms overhead), no isolation: a blocking call stalls the whole worker, and workflow code shares the worker’s process and environment. Reserve it for trusted, async-clean, latency-sensitive workflows — transient mesh hops especially, where pairing runner="inprocess" with durability="transient" is the lowest-overhead configuration for agent-to-agent calls.

docker (opt-in)

Each execution runs in its own container via docker run -i, speaking the same stdio child protocol — so containers hold no worker credentials either, and SIGTERM-based cancellation works unchanged. Use it for untrusted code, conflicting dependency sets, or filesystem isolation.

Workers must enable it explicitly and point docker_image at an image with flux-core installed at a worker-compatible version (pin the tag to the worker’s flux-core version — the child entrypoint and context wire format must match):

[flux.workers]
runners = ["inprocess", "subprocess", "docker"]
docker_image = "my-registry/flux-workflows:0.56.0"
# docker_network = ""       # "" = docker default network
# docker_memory = "512m"    # per-container memory limit
# docker_cpus = 1.0         # per-container CPU limit (0 = unlimited)
# docker_extra_args = []    # extra 'docker run' args: volumes, env, --user, --cap-drop, ...

A worker advertising docker must have a reachable Docker daemon — it fails at startup otherwise. Expect ~1.1–1.6 s per-execution overhead with a precompiled image; bake .pyc files into the image (RUN python -m compileall ...) or every execution pays flux’s import compilation again.

Pinning a workflow to a runner

By default a workflow runs under the worker’s default_runner (subprocess unless the worker changes it). To require a specific runner, declare it:

from flux import ExecutionContext
from flux.workflow import workflow

@workflow.with_options(runner="inprocess")
async def fast_hop(ctx: ExecutionContext[int]):
    ...

The declaration is dispatch-enforced. Workers advertise their enabled runners at registration ([flux.workers] runners, default ["inprocess", "subprocess"]), and a workflow that declares runner="docker" only dispatches to workers advertising docker. If no worker in the fleet advertises the requested runner, submission fails with RunnerNotAvailableError rather than queueing forever.

Valid values are "inprocess", "subprocess", and "docker". Running a workflow inline (workflow.run(...)) executes in the current process regardless of the runner option — runners only apply to executions dispatched to workers.

See the annotated runners example for all three declarations side by side, and the workflow SDK reference for the full with_options signature.

Crash semantics follow durability

If a runner child dies without reporting a result — segfault, OOM kill, os._exit — the worker surfaces WorkerProcessCrashed, and what happens next depends on the workflow’s durability:

• Durable (default): the claim is released back to the server (fenced by claim generation, pending checkpoints flushed first) and the execution is re-dispatched. Deterministic replay resumes from the last persisted task — completed tasks do not re-run.
• Transient: the execution fails terminally, honouring its at-most-once contract. The caller retries if it wants another attempt.

The module cache

Whatever the runner, workflow source arrives base64-encoded and is compiled into a module before execution. The compiled-module cache (flux/runners/loader.py) is TTL- and LRU-bounded and keyed by a hash of the source: module_cache_ttl (default 300 s) controls reuse, module_cache_max_size (default 64) bounds the entry count. Because the key includes the source hash, re-registering a workflow — even at the same version — recompiles immediately instead of serving stale source for the TTL.

Choosing a runner

What’s next

• Runners example — the three declarations in one runnable file.
• Durable vs transient workflows — how durability interacts with runner crashes and the mesh fast path.
• Capacity and drain — sizing max_concurrent_executions when every execution is a process.
• Server and worker settings — every runner-related config key.