Grafana + Prometheus

Flux exposes Prometheus metrics at GET /metrics. The endpoint is gated by the admin:metrics:read permission, so a scraper needs an API key from a principal that holds either the built-in admin role or a custom role with that permission.

Enable observability on the Flux server

The observability extra brings in OpenTelemetry SDK plus the Prometheus exporter:

pip install 'flux-core[observability]'

Turn it on in flux.toml:

[flux.observability]
enabled = true
# prometheus_enabled = true   # default; the master flag above is what controls it
service_name = "flux-prod"

enabled = true is the master switch. With it on and prometheus_enabled left at its default (true), the server registers a /metrics route that returns Prometheus text format. Restart the server to pick up the config:

flux start server

Verify the endpoint is wired up:

curl -i http://localhost:8000/metrics
# Without auth, expect:
# HTTP/1.1 401 Unauthorized

A 401 is the right answer — the endpoint exists but the request lacks credentials.

Provision a scraper principal

The built-in roles in Flux 0.56.0 are admin, operator, viewer, and worker. Only admin has admin:metrics:read. Giving a scraper the admin role would be overkill — it would also be able to manage roles, create principals, and shut down the server. Create a custom role first:

flux roles create metrics-scraper \
  --permissions admin:metrics:read

Create a service-account principal and grant it the role:

flux principals create metrics-scraper \
  --type service_account \
  --display-name "Prometheus scraper" \
  --role metrics-scraper

Mint an API key for the principal. The key is shown once — store it before moving on:

flux principals create-key metrics-scraper \
  --key-name prometheus-1 \
  --expires 365d
# API key created: <copy-this-once>

Test the key against the metrics endpoint:

curl -s -H "Authorization: Bearer <api-key>" http://localhost:8000/metrics | head -20

You should see Prometheus metric lines starting with flux_workflow_executions_total, flux_task_executions_total, and so on.

Prometheus scrape config

Store the API key in a file Prometheus can read (mode 0600), then reference it from prometheus.yml:

# prometheus.yml
scrape_configs:
  - job_name: flux
    metrics_path: /metrics
    scrape_interval: 15s
    static_configs:
      - targets: ['flux-server:8000']
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/flux-api-key

For multiple Flux server instances behind a load balancer, list them all under targets: — Prometheus scrapes each independently. The flux_workers_active gauge is per-server (each server tracks its own connected workers), so you will see per-instance counts.

Reload Prometheus (SIGHUP or curl -X POST localhost:9090/-/reload) and check the targets page: the flux job should be UP.

The metric catalog

The full catalog is 25 metrics across six concerns. The names below are exactly what Prometheus will scrape — Flux’s OTel meter exports the metric names as-is. (Not in this catalog: the dot-named flux.* scalars in GET /workers — those are worker-advertised routing inputs for dynamic routing, not Prometheus series.)

Workflow lifecycle:

• flux_workflow_executions_total — counter, labelled by workflow_namespace, workflow_name, status (started, completed, failed, cancelled, paused).
• flux_workflow_execution_duration_seconds — histogram, labelled by namespace and name.

Task lifecycle:

• flux_task_executions_total — counter, labelled by namespace, workflow, task_name, status.
• flux_task_execution_duration_seconds — histogram, same labels.
• flux_task_retries_total — counter, labelled by namespace, workflow, task.

Dispatch:

• flux_execution_queue_depth — up/down counter; executions waiting for a worker claim.
• flux_execution_schedule_to_start_seconds — histogram; time from queued to claim.
• flux_checkpoints_total — counter, labelled by namespace and workflow.
• flux_checkpoint_duration_seconds — histogram, same labels.
• flux_transient_hops_total — counter, labelled by workflow_namespace, workflow_name, outcome; in-process transient call() hops (the mesh fast path bypasses the server, so this is its aggregate observability).
• flux_transient_hop_duration_seconds — histogram, labelled by namespace and workflow.

Workers:

• flux_workers_active — up/down counter.
• flux_worker_registrations_total — counter, labelled by worker_name.
• flux_worker_disconnections_total — counter, labelled by worker_name, reason.
• flux_worker_executions_active — up/down counter per worker.
• flux_worker_auth_events_total — counter, labelled by worker_name, event.
• flux_worker_loop_lag_seconds — histogram, no labels; the worker’s event-loop scheduling lag, one sample per self-health probe (recorded in the worker process).
• flux_worker_health_transitions_total — counter, labelled by state (unhealthy, recovered); each flip of the worker’s self-health state (recorded in the worker process).

Scheduling:

• flux_schedule_triggers_total — counter, labelled by schedule_name, outcome.

Resume queue:

• flux_resume_queue_depth — up/down counter.
• flux_resume_schedule_to_start_seconds — histogram.
• flux_resume_claim_duration_seconds — histogram.

HTTP + module cache:

• flux_http_requests_total — counter, labelled by method, endpoint, status_code. Endpoints with execution IDs and worker names are normalized (e.g. /executions/{execution_id}) so cardinality stays bounded.
• flux_http_request_duration_seconds — histogram, labelled by method and endpoint.
• flux_module_cache_total — counter, labelled by result (hit, miss, eviction).

Grafana dashboard panels

Connect Grafana to the Prometheus data source and start with these seven panels. Each query assumes the flux job from the scrape config above.

1. Workers active (stat panel)

sum(flux_workers_active)

Set thresholds: green when >=1, red at 0.

2. Execution queue depth (time series)

sum(flux_execution_queue_depth)

A growing queue means workers cannot keep up with dispatch. Alert on > 50 sustained for five minutes.

3. Schedule-to-start p95 latency (time series)

histogram_quantile(0.95,
  sum(rate(flux_execution_schedule_to_start_seconds_bucket[5m])) by (le)
)

How long executions sit in the queue before a worker claims them. Healthy values are under one second on an idle worker pool.

4. Workflow success rate (time series)

sum(rate(flux_workflow_executions_total{status="completed"}[5m]))
  /
sum(rate(flux_workflow_executions_total{status=~"completed|failed|cancelled"}[5m]))

A drop here is the first signal of a regression. Alert on < 0.95 sustained.

5. Task retry rate (time series, per task)

sum by (workflow_name, task_name) (rate(flux_task_retries_total[5m]))

Spot tasks that are retrying constantly — often a sign of an external dependency in trouble.

6. Worker disconnections (time series)

sum by (reason) (rate(flux_worker_disconnections_total[5m]))

Group by reason to distinguish clean shutdowns from network drops.

7. HTTP error rate (time series)

sum by (status_code) (rate(flux_http_requests_total{status_code=~"4..|5.."}[5m]))

Catches misconfigured clients (lots of 401/403) and server errors (5xx spikes).

What goes wrong

Scrape returns 403. The principal exists and the key is valid, but the role does not include admin:metrics:read. Check with flux roles show metrics-scraper — the permissions list should contain that exact string. Built-in roles other than admin do not have it.

Scrape returns 401. The Authorization header is missing or malformed. Confirm the credentials file on the Prometheus host has no trailing newline (hexdump -C /etc/prometheus/flux-api-key | tail).

Metrics appear but values are flat at zero. Observability is enabled in config but the server has not been restarted since the change. flux.observability config is read at startup only.

Cardinality explosion. Flux normalizes path segments in flux_http_requests_total (execution IDs, worker names, schedule IDs all become placeholders) so the built-in HTTP labels stay bounded. Per-workflow labels (workflow_namespace, workflow_name) are bounded by the catalog size. Trouble starts if your workflows emit custom Prometheus metrics with high-cardinality labels (per-user, per-request) — that is on you, not on Flux’s instrumentation.

Metrics endpoint missing from a worker. The /metrics endpoint lives on the Flux server, not on workers. Workers do not run an HTTP server, so instruments recorded in the worker process (flux_worker_executions_active, flux_worker_loop_lag_seconds, flux_worker_health_transitions_total, the transient-hop pair) cannot be scraped from the worker directly — configure [flux.observability] with an otlp_endpoint on the worker hosts so those series reach your backend over OTLP push. The server’s own scrape covers everything the server records (queue depth, registrations, disconnections, HTTP, scheduling).

Production checklist

• enabled = true in flux.toml on every server.
• A dedicated metrics-scraper principal with a custom role; no shared admin keys for scrapers.
• API key expiry set (recommend rotating annually).
• Prometheus scrape interval at 15s or 30s — Flux’s metric export interval is 60s by default, so faster scraping does not help.
• Grafana alerting on flux_workers_active == 0 and flux_execution_queue_depth > 50.

Derived against Flux 0.56.0 with Prometheus 2.x and Grafana 11, 2026-07.