Kubernetes

Kubernetes deployment shape for Flux — server Deployment (single or multi-replica), worker Deployments, liveness and readiness probes, sticky routing for worker SSE, and HPA scoped to workers only.

Three workload categories.

Secrets land in the platform’s preferred mechanism — Kubernetes Secret resources for simple cases, External Secrets Operator + cloud KMS for production.

Server Deployment

# server.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: flux-server
  labels: { app: flux, role: server }
spec:
  replicas: 1                # scale up freely on PostgreSQL — see /deployment/high-availability
  strategy:
    type: RollingUpdate       # replicas coordinate through PostgreSQL; overlap is safe
  selector:
    matchLabels: { app: flux, role: server }
  template:
    metadata:
      labels: { app: flux, role: server }
    spec:
      containers:
        - name: server
          image: your-registry/flux:0.56.0
          args: ["flux", "start", "server", "--host", "0.0.0.0", "--port", "8000"]
          ports:
            - containerPort: 8000
              name: http
          env:
            - name: FLUX_DATABASE_URL
              valueFrom:
                secretKeyRef: { name: flux-server, key: database-url }
            - name: FLUX_WORKERS__BOOTSTRAP_TOKEN
              valueFrom:
                secretKeyRef: { name: flux-server, key: bootstrap-token }
            - name: FLUX_SECURITY__ENCRYPTION__ENCRYPTION_KEY
              valueFrom:
                secretKeyRef: { name: flux-server, key: encryption-key }
            - name: FLUX_SECURITY__AUTH__API_KEYS__ENABLED
              value: "true"
          readinessProbe:
            # /ready performs a database round-trip and returns 503 when the
            # DB is unreachable — a not-ready replica is drained, not restarted.
            httpGet: { path: /ready, port: http }
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 3
          livenessProbe:
            httpGet: { path: /health, port: http }
            initialDelaySeconds: 30
            periodSeconds: 30
            timeoutSeconds: 3
            failureThreshold: 3
          volumeMounts:
            - name: flux-home
              mountPath: /var/lib/flux
      volumes:
        - name: flux-home
          emptyDir: {}        # see "Storage" below

The two probe endpoints exist so the two probes can mean different things: GET /health is liveness (restart the process when it fails), and GET /ready is readiness — it does a database round-trip and returns HTTP 503 when the DB is unreachable, so Kubernetes stops routing traffic to the replica until the database recovers instead of restarting the pod through a DB blip. The JSON bodies carry status and database fields for richer diagnostics, but probes can rely on the status code alone. See Health checks and readiness.

A Service to front it:

# server-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: flux-server
spec:
  type: ClusterIP
  selector: { app: flux, role: server }
  sessionAffinity: ClientIP    # required at replicas > 1: workers must stick to
  sessionAffinityConfig:       # the replica that holds their SSE stream
    clientIP: { timeoutSeconds: 10800 }
  ports:
    - name: http
      port: 8000
      targetPort: http

The sessionAffinity is a no-op at one replica and load-bearing above it: a worker’s SSE stream (GET /workers/{name}/connect), its dispatch queue, and its in-flight signals live on the replica it connected to, so worker traffic needs source-IP or cookie affinity while everything else round-robins fine. See High availability.

For external traffic, add an Ingress (or your platform’s equivalent — Gateway, ALB Controller, GCLB). Terminate TLS at the ingress; Flux’s flux start server has no --ssl-keyfile or --ssl-certfile flags, so it serves plain HTTP and relies on a reverse proxy for transport security.

Worker Deployment

# worker-default.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: flux-worker-default
  labels: { app: flux, role: worker, pool: default }
spec:
  replicas: 3
  selector:
    matchLabels: { app: flux, role: worker, pool: default }
  template:
    metadata:
      labels: { app: flux, role: worker, pool: default }
    spec:
      terminationGracePeriodSeconds: 90   # drain_timeout (60s) + 30s
      containers:
        - name: worker
          image: your-registry/flux:0.56.0
          args:
            - flux
            - start
            - worker
            - "$(POD_NAME)"
            - --label
            - pool=default
          env:
            - name: POD_NAME
              valueFrom: { fieldRef: { fieldPath: metadata.name } }
            - name: FLUX_WORKERS__SERVER_URL
              value: http://flux-server:8000
            - name: FLUX_WORKERS__BOOTSTRAP_TOKEN
              valueFrom:
                secretKeyRef: { name: flux-worker, key: bootstrap-token }

Workers don’t need probes. They open an SSE connection to the server (GET /workers/{name}/connect); the server-side heartbeat reaper evicts a worker whose stream has gone silent. Adding a Kubernetes probe on the worker side adds noise without signal. If you want a synthetic liveness signal, point a probe at a /metrics scrape if you’ve enabled Prometheus on the worker — but the eviction reaper is the source of truth.

Workers drain gracefully on SIGTERM: they stop accepting work, finish running executions up to FLUX_WORKERS__DRAIN_TIMEOUT (default 60s), flush terminal checkpoints, then exit; a second signal aborts the drain. Set terminationGracePeriodSeconds to drain_timeout + 30s as in the manifest above, and cap concurrency per worker with FLUX_WORKERS__MAX_CONCURRENT_EXECUTIONS (default 16, 0 = unlimited). See Worker capacity and drain.

The POD_NAME trick gives each worker a unique name across replicas. Worker names must be unique across registrations; reusing a name re-registers under the same identity, which is occasionally useful but usually not what you want.

For a GPU pool, copy the Deployment, change the labels (pool: gpu), pass --label pool=gpu in the container args, and add the appropriate nvidia.com/gpu resource request. Workflows then route via @workflow(workers="pool=gpu") (see Running workers).

Storage

The server writes to <FLUX_HOME>/bootstrap-token (the persisted bootstrap secret) and, if using SQLite, to <FLUX_HOME>/flux.db. Two options:

If you set FLUX_WORKERS__BOOTSTRAP_TOKEN via env (as in the manifest above), the server never reads the persisted file, so PVCs are not required for token survival.

Secrets

The Server and Worker manifests reference two Secret resources:

apiVersion: v1
kind: Secret
metadata: { name: flux-server }
type: Opaque
stringData:
  database-url: "postgresql://flux:..."
  bootstrap-token: "<64 hex chars>"
  encryption-key: "<64 hex chars, distinct value>"
---
apiVersion: v1
kind: Secret
metadata: { name: flux-worker }
type: Opaque
stringData:
  bootstrap-token: "<same value as server>"

In production, generate these via External Secrets Operator from your cloud KMS rather than checking values into manifests.

Bootstrap-token rotation requires a server restart. The server caches the resolved token in memory at startup; updating the Secret mid-run doesn’t take effect until the pod restarts. The flow:

  1. Rotate the value in the secret store.
  2. Update both flux-server and flux-worker Secrets.
  3. kubectl rollout restart deployment/flux-server — and the workers, in that order.

See Bootstrap tokens for the overlap-window pattern that avoids worker downtime during rotation.

Scaling

HPAs target worker Deployments only.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: flux-worker-default }
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: flux-worker-default
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 70 } }

The server scales horizontally too — replicas coordinate through PostgreSQL, so kubectl scale deployment/flux-server --replicas=3 is safe once sticky routing is in place — but keep the HPA off it anyway: replica count is a deliberate availability decision, not a load-response one, and each replica adds database_pool_size + database_max_overflow (40 by default) to your PostgreSQL connection budget. Before rolling multiple replicas for the first time, run flux db upgrade once against the shared database. The full multi-replica model — advisory-lock scheduler singleton, SKIP LOCKED dispatch, pool math — is on High availability.

What can go wrong

Server crash-loops

Symptom. kubectl get pods shows flux-server in CrashLoopBackOff. Logs show OperationalError from SQLAlchemy.

Cause. Database unreachable, wrong password, or the FLUX_DATABASE_URL value points at a Service that hasn’t been created. Alembic migrations run on first connect (advisory-lock-guarded); the server exits if the connection fails.

Fix. Run a debug pod with psql and try the URL by hand: kubectl run psql --rm -it --image postgres:16 -- psql "$FLUX_DATABASE_URL" -c 'SELECT 1'. Network policies often surface here.

Workers stay OFFLINE

Symptom. flux worker list returns rows in OFFLINE status, or the server logs 401 Unauthorized on registration.

Cause. Bootstrap-token mismatch. Common when the flux-server and flux-worker Secrets drift, or when one was updated and only one Deployment was rolled.

Fix. Confirm both Secrets carry the same value (kubectl get secret flux-server -o jsonpath='{.data.bootstrap-token}' | base64 -d), then kubectl rollout restart both Deployments.

HPA flaps

Symptom. Worker replicas oscillate; flux worker list cycles between ONLINE and OFFLINE.

Cause. Pods are being replaced faster than the server-side eviction reaper can clean up old worker registrations. Each new pod registers under a new name, the old name stays around as OFFLINE until the reaper sweeps, and the metric you’re scaling on becomes noisy.

Fix. Increase the HPA’s stabilization window (behavior.scaleDown.stabilizationWindowSeconds: 300). If you’re scaling on per-worker queue depth, switch to total queue depth so the metric is stable across rollouts.

Next