vs. plain Python + queues

Most teams reading this page already have Postgres, already have Redis, and already have Celery (or RQ, or arq, or FastAPI’s BackgroundTasks) doing real work in production. The honest question isn’t “is Celery good?” — it shipped a decade of payment flows, video pipelines, and email queues. The question is: what does Flux give you that your existing queue doesn’t, and is it worth running another piece of infrastructure?

The short answer: Flux gives you durable execution as a property of the application — your code stops having to implement it. If you don’t need that property, plain queues are fine and cheaper to run.

What plain Python + queues does well

Celery is a small library. You pip install celery, decorate a function, point it at a broker, and ship. The same goes for RQ, arq, and the various BackgroundTasks integrations in FastAPI, Django Q, and friends. The barrier to entry is roughly zero, and the infrastructure under it — Redis as the broker, Postgres as the result backend — is what most teams already run.

The mental model fits in one sentence: a function gets enqueued, a worker picks it up, runs it, marks it done. Retries with exponential backoff are a decorator option. Scheduling is celery beat. Priority queues, rate limits, and routing have been settled features for years. Horizontal scale is the boring kind: more workers, more throughput, with the broker as the only contention point.

Failures are visible through channels you already watch — worker logs, queue depth, dead-letter queues — and the failure modes are ones an experienced backend engineer has debugged before. No new platform, no separate dashboard, no cluster to operate. For a meaningful fraction of background-work problems, this is the right answer.

What plain Python + queues doesn’t give you

The gap shows up when work has structure across more than one step, when individual jobs run long, or when “what state is this job actually in?” becomes a question someone asks at 3am.

Crash mid-job. A worker dies after step 3 of a 10-step job. With Celery, the whole job is re-queued and re-runs from step 1 — fine if every step is idempotent, though in practice many teams haven’t actually made them all idempotent. With Flux, the next worker reads the event log, sees steps 1–3 completed, and resumes at step 4. Completed steps don’t fire again because flux/task.py only runs a task body when no TASK_COMPLETED event exists for it.

Cross-step state. Step 3 produces a value step 7 needs. With Celery, you persist the intermediate result yourself — a row, a key, a blob — and write the logic that reads it back. With Flux, the result of an awaited @task call lives in its TASK_COMPLETED event (flux/domain/events.py, recorded by flux/task.py); step 7 reads it by await-ing the same task call, and on replay the recorded value is returned without re-running the step.

Mid-run inspection. “Show me what executed in this run, in what order, with what inputs and outputs.” With Celery, you read application logs and hope the operator who wrote them anticipated the question. With Flux, flux execution show <execution_id> --detailed reads the event log directly (flux/cli.py).

Deterministic replay. “Re-run this exact failing job with the same inputs against a new build of the code.” With Celery, you reconstruct the inputs from logs or database state and call the task by hand. With Flux, the inputs are in the event log; resume is built into the CLI (flux workflow resume <name> <execution_id> <input>).

Long-running jobs. A six-hour model training job. With Celery, worker timeouts terminate the task and retries restart from scratch unless you wrote checkpoint logic yourself. With Flux, every awaited task is a checkpoint by default — a worker restart in hour four resumes at the last completed task.

Multi-step compensation. Book a flight, book a hotel, book a car; if booking the car fails, unbook the hotel and the flight. With Celery, you write the rollback logic by hand: another task, another state machine, and a prayer that the unbooking task is itself idempotent. With Flux, every @task can declare a rollback callable (flux/task.py, exposed via task.with_options(rollback=...)); when a workflow fails, completed tasks run their rollbacks in reverse. The compensation is part of the task definition.

Human-in-the-loop pauses. A workflow that needs a manager’s approval at step 5. With Celery, you split the workflow in two, persist the in-between state somewhere, and schedule a resume task once approval arrives. With Flux, pause(name) from flux/tasks/pause.py raises a PauseRequested signal the engine catches, persists the workflow in PAUSED, and resumes through flux workflow resume with the approval payload as the resume input.

AI-agent semantics. Building an agent — LLM-driven tool dispatch, multi-step reasoning, memory across turns? Celery gives you a function-runner; you bring the rest. Flux ships agent(...) (flux/tasks/ai/agent.py) and a full agent harness (flux/agents/) that record every LLM call and tool dispatch as durable events on the same event log as the surrounding workflow.

None of these are impossible with Celery — they’re code you have to write, and code that, once written, is the thing on-call gets paged about. Flux is the framework you adopt when you stop wanting to maintain that code.

What Flux costs

Be clear-eyed about the price. You run a server process — a FastAPI app holding the workflow catalog and dispatch queries — alongside your application. You run at least one worker, separate from your application workers, claiming executions over an SSE stream. You give the event log a real database: Postgres in production, SQLite for development.

Your workflow code has to be deterministic. Time, randomness, and I/O move into tasks; the workflow function orchestrates and is replayable. Plan a day for an experienced Python dev to read the docs and write a real workflow; a second day to get comfortable with retries, rollbacks, and resume semantics.

The infrastructure is more parts than celery worker. It is many fewer parts than Temporal. If you’ve operated Postgres and a Python process, you’ve operated everything Flux needs.

When plain Python + queues is the right answer

• Single-step jobs. Send an email. Resize an image. Process a webhook. There is no multi-step graph to record.
• Fire-and-forget background work. The retry policy is “rerun the whole thing”; if that’s correct for your job, the durability question isn’t pulling its weight.
• Throughput-critical, latency-insensitive. A queue plus stateless workers is the most efficient way to absorb spikes. Flux doesn’t beat that pattern at its own game.
• Idempotent-by-design pipelines. If every step is trivially idempotent and short, the cost of writing a rerun script approaches zero.
• You don’t want another platform. Celery, Redis, and Postgres are already known operational quantities. Adding Flux is a real decision, not a free upgrade.

When Flux is the right answer

• Multi-step jobs where partial progress is expensive to throw away.
• Long-running jobs — anything past a few minutes where a restart from scratch hurts.
• Workflows that compose other workflows, or fan out and join.
• Anything with rollback or compensation logic across steps.
• Agents and LLM orchestration where every tool call should be recorded and resumable.
• Anywhere on your team a phrase like “I had to write a rerun script” appears more than once.

Closing

If you’ve ever written a rerun script, you’re already paying the cost of not having durable execution — it just shows up as on-call pages and one-off Python files instead of a line item. Whether Flux is the right way to stop paying that cost depends on the size of the cost. Small — short jobs, idempotent steps, patient operators — Celery is still the right answer. Large — long jobs, multi-step compensation, agents, mid-job restarts that hurt — Flux is the framework that makes the work go away rather than the one that makes it more manageable.

Where to read more

• Why durable execution — the underlying argument for the pattern.
• The execution model — events, checkpoints, replay in depth.
• Idempotency in Flux — what’s automatic and what you still have to think about.

Compared against Celery 5.x, RQ, and the queue-plus-Postgres pattern as commonly deployed, as of July 2026. Flux 0.56.0.