PostgreSQL

Run Flux against PostgreSQL — driver, DSN format, pool settings, managed options, and backups.

PostgreSQL is Flux’s recommended storage backend for any deployment that isn’t a single laptop. Flux uses Postgres for the workflow catalog, the execution event log, schedules, secrets, configs, agent definitions, and worker registration — everything that needs durability and cross-process visibility lives there.

This page is the operator’s recipe for plugging Flux into Postgres. For the full deployment story (high-availability, replication, performance tuning), see Operate → Server → Storage backends.

Install

Postgres support is an optional Poetry extra:

pip install "flux-core[postgresql]"

The extra pulls psycopg[binary,pool] ^3.2 — psycopg v3. Flux 0.56.0 uses one driver for both its synchronous SQLAlchemy engine and the async LISTEN/NOTIFY listener behind event-driven dispatch. Application tasks can use whichever driver they like; this only constrains Flux’s own engine.

DSN format

Set database_url in flux.toml or via the FLUX_DATABASE_URL env var:

[flux]
database_url = "postgresql://flux:${FLUX_DB_PASSWORD}@db.internal:5432/flux"

Two things to know:

The scheme must be postgresql://. The validator at flux/models.py::_validate_postgresql_url rejects postgres:// (the libpq short form). SQLAlchemy and most cloud-provider docs use postgresql:// already; some platforms (Heroku, older Render docs) emit postgres:// for compatibility — rewrite it before handing it to Flux.
${VAR} interpolation is supported. The field validator on database_url substitutes both ${VAR} and $VAR from the process environment. Use it for passwords so they don’t sit in plaintext in flux.toml.

The host segment must be non-empty — postgresql:///flux (no host, peer auth) is rejected. Use localhost explicitly when you mean it.

Keep the driverless postgresql:// form: Flux pins the psycopg (v3) dialect internally (flux/models.py::normalize_postgresql_url). Legacy postgresql+psycopg2:// URLs from pre-0.53 configs keep working — they are normalized to psycopg automatically.

On first connect Flux migrates the schema to the latest revision automatically (Alembic); to control the timing, run flux db upgrade yourself — see Upgrades and migrations.

Pool defaults

The shipping defaults in flux/config.py:

Setting	Default	Env var
`database_pool_size`	`20`	`FLUX_DATABASE_POOL_SIZE`
`database_max_overflow`	`20`	`FLUX_DATABASE_MAX_OVERFLOW`
`database_executor_threads`	`16`	`FLUX_DATABASE_EXECUTOR_THREADS`
`database_pool_timeout`	`30` (seconds)	`FLUX_DATABASE_POOL_TIMEOUT`
`database_pool_recycle`	`3600` (seconds)	`FLUX_DATABASE_POOL_RECYCLE`
`database_health_check_interval`	`300` (seconds)	`FLUX_DATABASE_HEALTH_CHECK_INTERVAL`

Effective max concurrency per process is pool_size + max_overflow = 40 connections by default. The API server, each worker process, and any external flux CLI invocation all open their own pool — size Postgres max_connections at replicas × (pool_size + max_overflow) plus the workers’ LISTEN connections plus headroom for ad-hoc admin sessions. Keep database_executor_threads at or below pool_size so the server’s DB threads never block waiting for a connection.

pool_recycle=3600 reopens connections every hour, which sidesteps the most common breakage when Postgres or a connection-pooler (PgBouncer, RDS Proxy) idle-times connections out.

Postgres version

PostgreSQL 14 or newer is expected. Multi-node Flux — a worker fleet, multiple server replicas — requires PostgreSQL; the dispatcher’s SKIP LOCKED claims, advisory-lock coordination, and LISTEN/NOTIFY signaling all assume PostgreSQL semantics, and the upstream Docker compose exercises postgres:16.

Managed options

Flux works with any standards-compliant Postgres, including:

AWS RDS for PostgreSQL and Aurora PostgreSQL. Pair with RDS Proxy if your worker fleet pushes connection counts beyond a few hundred.
Google Cloud SQL for PostgreSQL.
Azure Database for PostgreSQL (Flexible Server).
Neon. Serverless Postgres with branch-per-environment — fits Flux’s environment-per-server pattern well.
Supabase. Useful when you want pgvector and a REST/realtime layer alongside; just hand Flux the direct Postgres DSN, not the PostgREST endpoint.
Self-hosted. Any vanilla Postgres with the right max_connections works.

For any of these, the DSN goes into database_url. No code change.

Backups

Two patterns, neither Flux-specific:

Logical (pg_dump). Daily pg_dump --format=custom flux > flux-$(date -I).dump. Restore with pg_restore. Simple, portable across major versions, fine up to tens of GB.
Physical / WAL archiving (PITR). pg_basebackup plus continuous WAL archiving gives point-in-time recovery and faster restores at large sizes. Managed providers do this for you; self-hosted users want wal-g or pgbackrest.

The operational details (cron, encryption, off-site copies) belong in Operate → Maintenance → Backups and restore. The Flux-specific concern is just that the event log is the source of truth — a workflow that finished before the last backup is recoverable, anything claimed but not yet checkpointed after the backup is not.

What goes wrong

postgres:// URL rejected. Rewrite to postgresql://. Common when copy-pasting from Heroku, Render, or the libpq CLI.
Pool exhaustion. Symptom: QueuePool limit of size 20 overflow 20 reached, connection timed out. Causes: long-running tasks holding sessions, missing await session.close(), or simply too many workers for pool_size. Raise database_pool_size and the server-side max_connections together — bumping one without the other just moves the bottleneck.
DNS-mode quirks on AWS. RDS endpoints occasionally flap during failover; pool_recycle=3600 plus a small retry in your control plane absorbs this. The default is already tuned for it.
Encryption-at-rest is a separate concern. Flux encrypts secrets and configs inside the database via PyCryptodome; the database itself does not encrypt by default. Use Postgres-level (pgcrypto, TDE on managed services) or disk-level encryption if your compliance posture needs it.