Running the server

Starting, supervising, and gracefully shutting down a Flux server in production — and what to do when it won't start.

You have flux-core installed and a database reachable from that host. This page is the next step: starting the server, keeping it running, and recovering when it doesn’t.

The server is a FastAPI process (flux/server.py plus the route modules in flux/api/) wrapping uvicorn. The scheduler runs inside it. Multiple server replicas are supported on PostgreSQL — replicas coordinate through the database, and the scheduler runs as a fleet-wide singleton via a PostgreSQL advisory lock, so schedules never dispatch twice. There is no leader election to configure. See High availability for the multi-replica topology and Schedule management for schedule semantics.

The command

flux start server

That boots uvicorn, mounts the FastAPI app, starts the in-process scheduler, and begins the heartbeat reaper that evicts stale workers. Two optional flags:

flux start server --host 0.0.0.0 --port 8000

Defaults resolve from configuration (server_host = localhost, server_port = 8000). For anything beyond local dev, set --host so the process binds to a routable interface.

Everything else — database URL, auth, encryption keys — comes from configuration sources in this order: environment variables, then flux.toml, then [tool.flux] in pyproject.toml, then defaults.

Environment variables

The FLUX_ prefix and __ nested delimiter come from pydantic-settings (flux/config.py). The ones you will set most often:

VariableWhat it controls
FLUX_DATABASE_URLSQLAlchemy URL. Default sqlite:///.flux/flux.db. For production use postgresql://user:pass@host:5432/flux.
FLUX_HOMEFlux home directory (default .flux). The bootstrap-token file is persisted here.
FLUX_SECURITY__AUTH__OIDC__ENABLEDSet to true to require OIDC. See Authentication overview.
FLUX_SECURITY__AUTH__API_KEYS__ENABLEDSet to true to accept API-key auth.
FLUX_SECURITY__ENCRYPTION__ENCRYPTION_KEYRequired for the secrets store and payload signing. No default in 0.56.0; with auth enabled the server refuses to start without it.
FLUX_SECURITY__EXECUTION_TOKEN_SECRETSigns per-execution JWTs. Required when auth is enabled — the server refuses to start without it.

FLUX_DATABASE_URL supports ${VAR} interpolation, so postgresql://flux:${PG_PASSWORD}@db:5432/flux works.

If FLUX_WORKERS__BOOTSTRAP_TOKEN is unset the server generates one on first startup and writes it to <FLUX_HOME>/bootstrap-token (mode 0600). Retrieve it with flux server bootstrap-token.

TLS

flux start server does not expose --ssl-keyfile or --ssl-certfile. Terminate TLS at a reverse proxy and forward to Flux over plain HTTP on the loopback or a private network. A minimal nginx fragment:

server {
  listen 443 ssl;
  server_name flux.example.com;

  ssl_certificate     /etc/ssl/flux.crt;
  ssl_certificate_key /etc/ssl/flux.key;

  location / {
    proxy_pass         http://127.0.0.1:8000;
    proxy_http_version 1.1;
    proxy_set_header   Host              $host;
    proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto $scheme;

    # SSE dispatch and progress endpoints stream — disable buffering.
    proxy_buffering    off;
    proxy_read_timeout 1h;
  }
}

The proxy_buffering off and long read timeout matter. Workers hold GET /workers/{name}/connect open as an SSE stream; an aggressive proxy timeout severs it and forces reconnects.

Supervision

The process is foreground-by-default; wrap it in your init system.

systemd

# /etc/systemd/system/flux-server.service
[Unit]
Description=Flux server
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
Type=simple
User=flux
Group=flux
WorkingDirectory=/var/lib/flux
EnvironmentFile=/etc/flux/server.env
ExecStart=/usr/local/bin/flux start server --host 0.0.0.0 --port 8000
Restart=on-failure
RestartSec=5s
KillSignal=SIGTERM
TimeoutStopSec=30s

[Install]
WantedBy=multi-user.target

/etc/flux/server.env holds FLUX_DATABASE_URL, FLUX_SECURITY__ENCRYPTION__ENCRYPTION_KEY, and the auth toggles. chmod 600 it.

Docker

The image entrypoint runs flux start server. Mount FLUX_HOME as a volume so the persisted bootstrap-token survives restarts:

docker run -d --name flux-server \
  -p 8000:8000 \
  -e FLUX_DATABASE_URL="postgresql://flux:secret@db:5432/flux" \
  -e FLUX_SECURITY__AUTH__API_KEYS__ENABLED=true \
  -e FLUX_SECURITY__ENCRYPTION__ENCRYPTION_KEY="$(cat /etc/flux/encryption-key)" \
  -v flux-home:/root/.flux \
  ghcr.io/edurdias/flux:0.56.0

Kubernetes

Use a Deployment. Multiple replicas are supported when the database is PostgreSQL — schedule dispatch and retention sweeps are fleet-wide singletons coordinated through advisory locks, so a second replica does not double-fire schedules. Give worker SSE connections (/workers/{name}/connect) session affinity at the load balancer, and run flux db upgrade once before rolling replicas across a version boundary. The full multi-replica story is in High availability. On SQLite, keep replicas: 1 — SQLite is single-node only.

Either way, pin the bootstrap token explicitly (FLUX_WORKERS__BOOTSTRAP_TOKEN from a Secret) or mount a shared PersistentVolumeClaim at FLUX_HOME; a per-pod regenerated token will lock out every existing worker until they re-register. Workers run as their own Deployment and scale independently — see Running workers.

Graceful shutdown

Send SIGTERM. Uvicorn drains in-flight HTTP requests and closes the SSE worker connections; the scheduler and heartbeat reaper tasks stop in the FastAPI lifespan shutdown hook (flux/server.py::_create_api).

Workers do not lose state when the server goes away. Each worker reconnects with exponential backoff (capped at workers.reconnect_max_delay, default 60 seconds). In-flight executions on workers keep running during the outage — workers buffer checkpoint POSTs and retry them once the server is back.

The auth-disabled warning

On every startup with auth disabled, you will see:

CRITICAL - Authentication is DISABLED. All requests are treated as the ANONYMOUS
admin principal. This is not safe for production. Enable an auth provider via
[flux.security.auth.oidc] or [flux.security.auth.api_keys] before exposing this
server.

auth.enabled is a real, settable field (FLUX_SECURITY__AUTH__ENABLED). Enabling either oidc.enabled or api_keys.enabled also implies auth.enabled = true, and the server rejects auth.enabled = true with no provider configured at startup (flux/security/config.py). A production deployment must turn on at least one provider.

What can go wrong

Four things break first when the server won’t start.

Incomplete security configuration

Symptom. Startup raises RuntimeError: Refusing to start with incomplete security configuration.

Cause. With auth enabled, the server validates at startup that FLUX_SECURITY__EXECUTION_TOKEN_SECRET and FLUX_SECURITY__ENCRYPTION__ENCRYPTION_KEY are set — previously these surfaced only mid-traffic, when the first worker token was minted or the first secret stored. Debug mode is exempt.

Fix. Set both values (random 32+ bytes each) in the server environment. See Authentication overview.

Bind conflict

Symptom. Process exits with OSError: [Errno 98] Address already in use.

Fix. Find the owner with lsof -i :8000 or ss -ltnp 'sport = :8000'. Stop it, or pick a free port (flux start server --port 8001, or set FLUX_SERVER_PORT=8001).

Missing or unreachable database

Symptom. Startup fails before the Flux server started successfully log line. SQLAlchemy raises OperationalError (or psycopg.OperationalError). Schema migrations run on first connect (Alembic, flux/migrations/) — if the connection itself fails, the server exits.

Fix. Verify the URL from the server host: psql "$FLUX_DATABASE_URL" -c 'SELECT 1'. Common causes are a wrong password, a firewall rejecting the server’s IP, or the DB user lacking DDL permission on the target database (Flux applies migrations on first connect, so the user needs owner-level permissions — or run flux db upgrade once with a privileged DSN).

Bootstrap token state missing

Symptom. Workers fail registration with 401 Unauthorized on POST /workers/register after a server redeploy. The server logs nothing — it just doesn’t recognize the worker’s token.

Cause. The server resolves its bootstrap token from <FLUX_HOME>/bootstrap-token at first request (resolve_or_generate in flux/security/bootstrap_token.py). If FLUX_HOME is on ephemeral storage, the file is lost on redeploy, the server generates a new token, and the old workers’ tokens stop working.

Fix. Persist FLUX_HOME on durable storage (volume mount, PVC), or pin the token explicitly via FLUX_WORKERS__BOOTSTRAP_TOKEN. A configured value wins over the persisted file, so distribute the same value to every worker. With the override set, flux server bootstrap-token --rotate is a no-op until you remove it.

Next