High availability

Running multiple Flux server replicas — the PostgreSQL coordination model, sticky routing for worker SSE, rolling upgrades, probe wiring, and connection-pool math.

Flux supports multiple server replicas against one PostgreSQL database. There is no leader-election component to deploy and no per-replica configuration to keep in sync: replicas coordinate entirely through PostgreSQL, and the pieces that must run once per fleet (the scheduler cycle, the retention sweep) elect themselves per cycle with advisory locks. This page covers when to run more than one replica, what the coordination actually relies on, and the two operational requirements that are easy to miss — sticky routing for worker connections and running the database migration before a rolling deploy.

When to go multi-replica

One server replica is still a fine deployment. It restarts fast, workers reconnect with backoff, and in-flight executions on workers keep running and buffer their checkpoints while the server is briefly gone. Add replicas when:

You need the API to survive a pod/node loss without a gap. A single replica means seconds-to-minutes of API unavailability per restart; schedules due during the gap fire once on recovery rather than on time.
You need zero-downtime rolling deploys. With one replica, every deploy is a blip. With two or more, the load balancer drains one replica while the others serve.
The HTTP path is saturated. Rare for the server (the bottleneck is usually the database or the worker pool), but sync/stream callers held open at scale are per-replica resources.

What replicas do not buy you: more scheduler throughput (the scheduler cycle is a fleet-wide singleton — replicas make it failover-safe, not faster) or protection from a database outage (see What still isn’t HA).

Prerequisites

PostgreSQL 14+, psycopg v3 driver (pip install 'flux-core[postgresql]'), shared by every replica via FLUX_DATABASE_URL. SQLite is single-node only — one server plus one colocated worker, or inline workflow.run(). SELECT … FOR UPDATE SKIP LOCKED and the advisory locks that make multi-replica safe are silently unenforced on SQLite, and the server logs a warning when a second worker registers against it.
Schema at head before rolling. The schema is Alembic-managed and auto-migrated when a server opens the database, guarded by a PostgreSQL advisory lock so concurrent replicas can’t race the migration. For rolling deploys, run flux db upgrade once (from a job or your deploy pipeline) before rolling the replicas, so old and new replicas never disagree about who migrates. See Upgrades and migrations and the flux db CLI reference.
A load balancer with connection affinity in front of the replicas (next section).

How coordination works

No replica is special. Each mechanism relies on a PostgreSQL primitive:

Dispatch is double-assign-safe. Every replica’s dispatcher claims work with SELECT … FOR UPDATE SKIP LOCKED — two replicas can never hand the same execution to two workers. New-work wakeups travel between replicas via NOTIFY flux_work; a missed notification is covered by the dispatcher’s fallback tick ([flux.dispatch] fallback_interval, default 15s). With PostgreSQL at scale, run [flux.dispatch] mode = "event" — see Dispatch modes.
The scheduler cycle is a fleet-wide singleton. Each cycle, one replica takes a session-scoped pg_try_advisory_lock; the others skip the cycle. Scheduler run state (next_run_at, last_run_at) is persisted per fire, so schedules neither double-fire across replicas nor re-fire on restart. If the lock holder dies mid-cycle, its connection drops and PostgreSQL releases the lock automatically — the next cycle elects a survivor.
The retention sweep is the same shape — one replica per sweep via advisory lock. See Retention.
Worker liveness is a persisted, global view. Heartbeats land in workers.last_seen_at (batched, one UPDATE per interval per replica), so any replica’s reaper can evict a dead worker and reclaim its executions — including workers that were attached to a replica that no longer exists.
Sync and stream callers wake across replicas. A caller held open on replica A is woken by a checkpoint landing on replica B via NOTIFY flux_exec, with a 30-second poll fallback as the safety net.
GET /workers is consistent everywhere — it reads the workers table, so every replica returns the same fleet view. One per-replica exception: the "unhealthy" status (a worker self-reporting event-loop starvation on its heartbeat pong) lives in in-memory state on the replica holding that worker’s SSE connection. Only that replica shows "unhealthy"; the others report the worker "online" from the persisted heartbeat. Dispatch is unaffected — work for a worker only ever flows through the replica holding its connection, which is also the replica receiving its pongs, so the unhealthy exclusion applies exactly where dispatch decisions for that worker are made.

One consistency caveat: auth resolution (token → identity → permissions) is cached per replica for [flux.security.auth] resolution_cache_ttl seconds (default 30). Revoking a credential takes effect immediately on the replica that processed the revocation and within the TTL on the others — keep it short.

Sticky routing for worker SSE

A worker’s SSE stream (GET /workers/{name}/connect), its dispatch queue, and its in-flight execution signals live on the replica it connected to. Configure the load balancer with connection affinity — source-IP or cookie stickiness — so a worker’s requests land on the replica holding its stream, and let workers reconnect through the same path. Plain HTTP round-robin is fine for every other route (CLI, SDK, REST callers).

If a replica dies, its workers’ streams drop; the workers reconnect through the load balancer and land on a surviving replica, and any executions orphaned by the dead replica are reclaimed by another replica’s reaper via the persisted heartbeats. Stickiness makes the steady state correct; the persisted liveness view makes the failure case correct.

Probes

Wire the two probes differently — that’s why there are two:

GET /health — liveness. Restart the process when this fails.
GET /ready — readiness. Performs a database round-trip and returns 503 when the DB is unreachable. Wire it to load-balancer membership, not restarts, so a database blip drains traffic from the replica instead of bouncing it.

Connection-pool math

Each replica holds its own pool. Per-replica defaults (top-level config keys / FLUX_-prefixed env vars):

[flux]
database_pool_size = 20          # base pool per replica
database_max_overflow = 20       # burst headroom per replica
database_executor_threads = 16   # thread pool for blocking DB calls; keep ≤ pool_size

Keep database_executor_threads at or below database_pool_size so DB threads never block waiting for a connection. Then size PostgreSQL:

max_connections ≥ replicas × (pool_size + max_overflow)
                + LISTEN connections (dispatch/exec notify, per replica)
                + advisory-lock sessions + migration + admin headroom

Three replicas at the defaults is 3 × 40 = 120 connections of pool ceiling alone — a stock max_connections = 100 PostgreSQL will fall over under burst. Either raise max_connections or front the pools with PgBouncer (session pooling; the advisory locks and LISTEN connections are session-scoped).

Also relevant behind a load balancer: POST /workers/register is rate-limited per client IP ([flux.workers] register_rate_limit, default "30/minute", slowapi syntax, "" disables). A large fleet restarting behind one NAT — or a proxy that hides real client IPs — trips it; raise the limit or make sure the proxy forwards real client IPs to uvicorn.

Rolling upgrades

flux db upgrade against the shared database (migrations are advisory-lock-guarded and idempotent).
Roll the server replicas with a RollingUpdate strategy — no Recreate needed anymore; two live replicas coordinate safely during the overlap.
Roll workers. Send SIGTERM and let each worker drain — it finishes running executions (up to drain_timeout, default 60s), flushes terminal checkpoints, then exits. Give the orchestrator a termination grace period of drain_timeout + 30s. See Worker capacity and drain.

What still isn’t HA

The database. Every coordination mechanism above lives in PostgreSQL; if it’s down, the fleet is down. /ready returns 503 on every replica and the load balancer drains all of them. Use managed multi-AZ PostgreSQL or your own failover story — see Disaster recovery.
A worker’s attachment to one replica. The SSE stream is a per-replica resource. Replica loss means every worker attached to it reconnects and re-registers — brief dispatch latency, not lost work.
TLS. Flux serves plain HTTP; TLS termination in front of every replica is your load balancer’s job, and it is a hard requirement — bearer tokens, worker session keys, and decrypted secrets travel over this channel.
Poll-mode dispatch at scale. The default [flux.dispatch] mode = "poll" degrades superlinearly with fleet size regardless of replica count. Multi-replica production on PostgreSQL should run mode = "event".

Worked example: three replicas on Kubernetes

The delta from the single-replica shape on Kubernetes: replicas: 3, a RollingUpdate strategy, split probes, and session affinity on the Service the workers connect through.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flux-server
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate: { maxUnavailable: 1, maxSurge: 1 }
  selector:
    matchLabels: { app: flux, role: server }
  template:
    metadata:
      labels: { app: flux, role: server }
    spec:
      initContainers:
        - name: db-upgrade
          image: your-registry/flux:0.56.0
          command: ["flux", "db", "upgrade"]
          env:
            - name: FLUX_DATABASE_URL
              valueFrom:
                secretKeyRef: { name: flux-server, key: database-url }
      containers:
        - name: server
          image: your-registry/flux:0.56.0
          args: ["flux", "start", "server", "--host", "0.0.0.0", "--port", "8000"]
          ports:
            - containerPort: 8000
              name: http
          env:
            - name: FLUX_DATABASE_URL
              valueFrom:
                secretKeyRef: { name: flux-server, key: database-url }
            - name: FLUX_WORKERS__BOOTSTRAP_TOKEN
              valueFrom:
                secretKeyRef: { name: flux-server, key: bootstrap-token }
            - name: FLUX_SECURITY__ENCRYPTION__ENCRYPTION_KEY
              valueFrom:
                secretKeyRef: { name: flux-server, key: encryption-key }
            - name: FLUX_DISPATCH__MODE
              value: "event"
          readinessProbe:
            httpGet: { path: /ready, port: http }   # DB round-trip; 503 drains the replica
            periodSeconds: 10
            timeoutSeconds: 3
          livenessProbe:
            httpGet: { path: /health, port: http }
            initialDelaySeconds: 30
            periodSeconds: 30
            timeoutSeconds: 3
---
apiVersion: v1
kind: Service
metadata:
  name: flux-server
spec:
  type: ClusterIP
  selector: { app: flux, role: server }
  sessionAffinity: ClientIP        # workers stick to the replica holding their SSE stream
  sessionAffinityConfig:
    clientIP: { timeoutSeconds: 10800 }
  ports:
    - name: http
      port: 8000
      targetPort: http

For traffic entering through an ingress instead of the ClusterIP Service, use your controller’s cookie affinity for the worker path — for ingress-nginx:

metadata:
  annotations:
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "flux-affinity"

The initContainers block runs flux db upgrade before each pod starts; because migrations are advisory-lock-guarded and idempotent, three pods racing the init container is safe — the first one migrates, the others observe head and continue.

What can go wrong

Workers churn between ONLINE and OFFLINE after adding replicas

Symptom. The fleet was stable with one replica; with three, flux worker list shows workers flapping and the server logs show repeated registrations.

Cause. No connection affinity — each worker reconnect lands on a random replica, and the replica that holds a worker’s dispatch queue isn’t the one receiving its traffic.

Fix. Add stickiness (Service sessionAffinity: ClientIP, or cookie affinity at the ingress) for the path workers use, then restart the workers once.

`429 Too Many Requests` on worker registration

Symptom. A fleet restart produces a burst of 429 responses from POST /workers/register.

Cause. All workers appear to the server as one client IP (NAT or a proxy that doesn’t forward client IPs), and the default register_rate_limit = "30/minute" per IP throttles the burst.

Fix. Raise it (FLUX_WORKERS__REGISTER_RATE_LIMIT=300/minute) or configure the proxy to forward real client IPs.

PostgreSQL runs out of connections after scaling up

Symptom. Replicas start failing /ready; PostgreSQL logs FATAL: sorry, too many clients already.

Cause. replicas × (pool_size + max_overflow) plus LISTEN connections exceeded max_connections.

Fix. Do the pool math above — lower the per-replica pool, raise max_connections, or add PgBouncer in session-pooling mode.

Kubernetes for the full manifest set the example above extends.
Production checklist for the remaining gate checks.
Dispatch modes for poll vs event dispatch in depth.
Disaster recovery for the cross-region story on top of in-region HA.