Upgrades and migrations

Read this before your first upgrade. As of Flux 0.56.0, schema migrations ship in the box.

How migrations work

Flux manages its database schema with Alembic. Schema changes ship as migration scripts inside the package (flux/migrations/versions/), and the database is brought to the latest revision automatically — the first time a server, worker, or inline workflow.run() opens the database, Flux runs any pending migrations before serving traffic. There is no manual drop-and-recreate step, and Base.metadata.create_all is gone.

Three database states are handled automatically:

On PostgreSQL the migration run is guarded by a pg_advisory_lock, so multiple servers or workers starting simultaneously cannot run migrations at the same time — the first one to acquire the lock migrates, the rest wait and then proceed against the finished schema. SQLite is single-node, so the lock is a no-op there.

The flux db commands

For control over when migrations run, use the CLI instead of relying on startup:

flux db upgrade     # apply all pending migrations
flux db current     # show the database's current schema revision
flux db history     # list available revisions, newest first

flux db current and flux db history are read-only — they inspect the alembic_version table without triggering a migration, so they are safe to run against production at any time. The full command reference is at Reference → CLI → db.

Upgrade procedure

For a single-server deployment, an upgrade is:

1. Read the release notes for the target version. Note anything labelled breaking or schema.
2. Back up the database. Migrations modify schema in place; the backup is your rollback path. See Backups and restore.
3. Verify your encryption key is backed up. Execution input/output, event values, and schedule input are HMAC-signed with FLUX_SECURITY__ENCRYPTION__ENCRYPTION_KEY — restore the database without the key and that data is unreadable.
4. Deploy the new version and start it. Migrations run on first connection. Check the result with flux db current.
5. Validate with a known-good workflow before you reopen traffic.

Rehearse steps 2–5 in a non-prod environment first — a forty-second dry run is cheaper than a surprise at 3am.

Rolling upgrades and multiple replicas

The advisory lock makes concurrent startup safe: if you roll several server replicas at once against PostgreSQL, exactly one runs the migration and the others wait. But for a controlled multi-replica roll, migrate once up front:

flux db upgrade     # run once, from any host with the new Flux version and the prod DSN
# then roll the server replicas and workers

This decouples “schema is at head” from “replica N restarted”, so a slow rollout never leaves a replica blocked on the migration lock, and a failed migration surfaces in one controlled command instead of mid-deploy.

Version skew between servers and workers is a separate concern from schema:

• Same minor, different patch. Usually safe. Roll workers one at a time, then the servers.
• Workers ahead of servers, or servers ahead of workers. Risky across minor versions — the SSE event payloads and checkpoint shapes may not match. Keep the skew window short and read the release notes.
• Schema change involved. New-version processes migrate the schema on startup; old-version processes then run against a newer schema than they were built for. Additive migrations tolerate this; destructive ones do not. When the release notes flag a breaking schema change, stop everything, run flux db upgrade, then restart everything on the new version.

Recovery from a bad upgrade

If an upgrade goes wrong:

1. Stop the server and all workers. Do not let in-flight executions write more events.
2. Restore the database from the pre-upgrade backup. See Backups and restore.
3. Roll back the Flux package version (pip install flux-core==<old-version> or your image tag of choice). Do not start the new version against the restored database — it would immediately re-run the migration.
4. Restart the server and workers on the old version.
5. Reproduce the failure in non-prod before you retry. flux db upgrade against a copy of the production database is the fastest way to isolate a migration problem offline.

Downgrades are not automatic: Flux only migrates forward. The rollback path for schema is always the backup, which is why the backup in step 2 of the upgrade procedure is non-negotiable.

Upgrading a pre-Alembic database

If you are coming from a Flux version that predates the migration system (schema created by Base.metadata.create_all, no alembic_version table), nothing special is required. On first connect the new version detects the legacy schema, stamps it at the 0001_baseline revision, and upgrades in place — executions, event history, secrets, and catalog rows are preserved. Take a backup first anyway; this is still a schema change.

You can verify the state before and after:

flux db current     # "none (database is unmanaged or empty)" before; a revision id after

What can go wrong

• Startup hangs on “waiting for migration lock”. Another process is holding the PostgreSQL advisory lock — usually a replica mid-migration. If the holder died without releasing (its session survived), terminate that backend session in Postgres; the lock releases with the session.
• A migration fails partway. On PostgreSQL the migration run executes transactionally and rolls back on failure, so the database stays at its previous revision — fix the cause (usually permissions or disk) and re-run flux db upgrade. On SQLite, restore from backup if the file is left in a doubtful state.
• Restored database, unreadable payloads. Execution values are HMAC-signed with the encryption key. If the restored environment has a different FLUX_SECURITY__ENCRYPTION__ENCRYPTION_KEY, reads fail integrity checks. Restore the original key alongside the database — this is Plane C in Backups and restore.
• DB user lacks DDL permission. Migrations issue ALTER TABLE / CREATE TABLE; the connecting user needs owner-level rights on the schema. Either grant them, or run flux db upgrade once with a privileged DSN and run the servers with a lesser one.