Health checks and readiness

Wiring up liveness and readiness probes for a Flux server, and what to alert on beyond probe failures.

An orchestrator needs two signals from the Flux server: process liveness, and traffic readiness. This page maps both onto what 0.56.0 exposes, plus the alerting and synthetic checks that catch what probes miss.

What the server exposes

Two endpoints, both defined in flux/api/system_routes.py, both unauthenticated so probes hit them without managing tokens:

/health response shape

HealthResponse is a three-field Pydantic model (flux/api/schemas.py):

{
  "status": "healthy",
  "database": true,
  "version": "0.56.0"
}

/ready response shape

{
  "status": "ready",
  "database": true
}

The DB round-trip runs off the event loop (asyncio.to_thread), so a slow database degrades the probe’s latency without wedging the server. On failure the body is {"status": "not-ready", "database": false}.

Status codes and latency

Both routes return 200 OK when the database check passes and 503 Service Unavailable when it fails — whether the check reports false or the handler catches an exception while running it. The status code carries the verdict, so a status-code-only probe is correct: a 200 means healthy, a 503 means pull the replica. The JSON bodies carry detail, but probes don’t have to parse them.

The DB round-trip is one SELECT 1: microseconds on SQLite, low-single-digit milliseconds on PostgreSQL. Budget a 2-3 second timeout to absorb cold pool acquisitions.

Liveness vs readiness

Wire each intent to its endpoint:

Wire /ready to load-balancer membership, not to restarts: restarting a healthy process because its database blipped just adds a cold start to the outage.

Kubernetes probe configuration

A working set — liveness on /health, readiness and startup on /ready:

livenessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 15
  periodSeconds: 30
  timeoutSeconds: 3
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /ready
    port: 8000
  initialDelaySeconds: 5
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 2

startupProbe:
  httpGet:
    path: /ready
    port: 8000
  failureThreshold: 30
  periodSeconds: 5

The startup probe gives slow first connects (PostgreSQL TLS handshake, the Alembic migration run on first connect after a deploy) up to 150 seconds before liveness takes over.

ALB and NLB health checks work the same way: point them at /ready and treat 200 as healthy, 503 as unhealthy — no body parsing required. Surfacing the JSON body in a Datadog or Prometheus check is still useful for richer alerting detail, but it isn’t needed for the probe to be correct.

What to alert on

Probes catch process-level failure. Two more signals catch the rest:

  1. /ready returns HTTP 503 for N consecutive minutes. A Datadog or Prometheus HTTP check on the status code. A blip is normal during failover; sustained means the DB is down.
  2. Online worker count drops below threshold. GET /workers?status=online (flux/api/worker_routes.py) reads the workers table, with liveness derived from persisted heartbeats (workers.last_seen_at) — so every server replica returns the same fleet view. If you expect five workers and see one, you have a worker outage even though the server is fine. This endpoint requires auth — provision a read-only API key for the alerting pipeline.

Synthetic workflow canary

The strongest signal is a workflow that actually ran. Register a tiny canary workflow that returns a small value, schedule it with cron("*/5 * * * *"), and alert when its most recent execution is FAILED or older than the schedule period. This proves the full pipeline: server accepts the schedule, scheduler dispatches, a worker claims it, storage persists the result. Use GET /schedules/{id}/history to surface the last N executions to the alerting system.

The canary catches three failure modes /ready misses: a wedged scheduler, workers online but unable to claim (bootstrap token mismatch after a redeploy), and storage that reads fine but blocks on writes.

What can go wrong

Probes pass but workflows hang

Symptom. /ready returns 200 and workers show as online, but workflows sit in SCHEDULED indefinitely.

Cause. /ready only checks DB connectivity. It does not verify the scheduler is dispatching or that workers are claiming work. A wedged scheduler is invisible to probes.

Fix. Add the synthetic-workflow canary above. Its execution state is the only signal that the full pipeline is moving.

Probes flap under load

Symptom. Pods restart during traffic spikes. Liveness failures appear in events but the server logs show no errors.

Cause. The probe’s SELECT 1 competes for a connection from the same pool as in-flight requests. If the pool is exhausted, the probe times out.

Fix. Raise the pool size (PostgreSQL: database_pool_size / database_max_overflow), or extend timeoutSeconds. Three seconds is the floor.

Probes fail immediately after deploy

Symptom. The pod never reaches Ready. Liveness failures start before the server has finished initializing.

Cause. The first connect after a deploy runs pending Alembic migrations before the server accepts traffic, which can be slow against a cold PostgreSQL — and on a multi-replica roll, replicas wait on the migration advisory lock while one of them migrates. The default initialDelaySeconds: 15 on livenessProbe is shorter than that initialization on some clusters.

Fix. Add a startupProbe (sample above). It runs before liveness/readiness with a generous failureThreshold. Once it passes, the regular probes take over. To take migrations out of the deploy path entirely, run flux db upgrade once before rolling replicas — see Upgrades and migrations.

Next