Metrics

Flux ships a Prometheus /metrics endpoint backed by an OpenTelemetry MeterProvider. The endpoint exposes counters, gauges, and histograms covering workflow execution, task lifecycle, worker fleet health, scheduling, HTTP traffic, and resume queue depth. It is mounted on the same FastAPI app as the REST surface (flux/server.py).

Enabling

You need the observability extra installed and two settings on:

pip install 'flux-core[observability]'

# flux.toml
[flux.observability]
enabled = true
prometheus_enabled = true   # default

Or via environment:

FLUX_OBSERVABILITY__ENABLED=true
FLUX_OBSERVABILITY__PROMETHEUS_ENABLED=true   # default

enabled is the master switch — when it is false, no providers are set up and /metrics is not registered (flux/server.py:1072). prometheus_enabled defaults to true in ObservabilityConfig (flux/observability/config.py), so the only required variable is FLUX_OBSERVABILITY__ENABLED=true.

The endpoint is GET /metrics and is protected by require_permission("admin:metrics:read"). Grant your Prometheus scraper an API key whose role includes that permission (the built-in admin role’s * wildcard works).

metric_export_interval (default 60 seconds) only governs OTLP push exports — Prometheus uses a pull-based PrometheusMetricReader, so the scrape interval is controlled by Prometheus itself.

Metrics catalog

All 25 instruments are defined in flux/observability/metrics.py::FluxMetrics.

One disambiguation up front: these OTel instruments are not the flux.*-prefixed worker metrics (flux.loop_lag_p95_seconds, flux.slots_free, flux.failure_rate, …) that appear in the metrics field of GET /workers. Those are scalar snapshots each worker advertises on its heartbeat pong, persisted to the workers table to feed routing-policy metric(...) selectors — they never appear on /metrics and are not time series. See Dynamic routing for that family. The underscore-named flux_* instruments below are the Prometheus/OTLP surface.

Workflow lifecycle

Task lifecycle

Dispatch queue

The transient-hop pair is recorded worker-side: in-process transient call() hops bypass the server entirely, so this counter is the mesh fast path’s only aggregate observability.

Worker fleet

The last two are recorded in the worker process by the self-health probe (see How workers work): the loop-lag histogram gets one sample per probe (default every second; zero-lag samples are skipped), and the transition counter increments each time a worker flips unhealthy or recovers. Because workers don’t serve /metrics, these reach your backend through the worker’s OTLP export, not the server scrape.

Scheduling

Resume queue

HTTP surface

MetricsMiddleware normalizes paths before recording so per-execution and per-worker IDs collapse into placeholders like /executions/{execution_id} and /workers/{worker}/.... The /metrics path itself is excluded from HTTP recording.

Module cache

Alerts

Recommended starting points — tune thresholds to your fleet size and SLOs.

- alert: FluxWorkersDown
  expr: flux_workers_active < 1
  for: 2m
  annotations:
    summary: "No Flux workers connected"

- alert: FluxQueueBacklog
  expr: flux_execution_queue_depth > 50
  for: 5m
  annotations:
    summary: "Execution queue depth above threshold — scale workers"

- alert: FluxScheduleToStartSlow
  expr: histogram_quantile(0.95, rate(flux_execution_schedule_to_start_seconds_bucket[5m])) > 30
  for: 10m
  annotations:
    summary: "p95 queue wait above 30s"

- alert: FluxWorkerChurn
  expr: rate(flux_worker_disconnections_total[5m]) > 0.1
  for: 5m
  annotations:
    summary: "Worker disconnect rate elevated"

- alert: FluxWorkflowFailureRate
  expr: |
    sum(rate(flux_workflow_executions_total{status="failed"}[5m]))
      / sum(rate(flux_workflow_executions_total{status=~"completed|failed|cancelled"}[5m])) > 0.05
  for: 10m
  annotations:
    summary: "Workflow failure rate above 5%"

Scrape config

scrape_configs:
  - job_name: flux
    metrics_path: /metrics
    scrape_interval: 30s
    static_configs:
      - targets: ['flux-server.internal:8000']
    authorization:
      type: Bearer
      credentials: "<api-key-with-admin:metrics:read>"

A 30-second scrape interval matches the histogram bucket granularity without inflating cardinality on multi-worker fleets.

What can go wrong

• /metrics returns 404 or an empty body. The endpoint is only mounted when FLUX_OBSERVABILITY__ENABLED=true. prometheus_enabled defaults to true but does nothing if enabled is off. Confirm with flux config get observability.enabled and check the server startup log for the Observability initialized (prometheus=...) line emitted by flux/observability/provider.py.
• Cardinality from worker labels. Labels like worker_name, task_name, and workflow_name multiply when you run many short-lived workers or generate task names dynamically. Stable identifiers (long-lived worker names from flux start worker my-worker, hardcoded task names) keep the time-series count bounded. Avoid building task or workflow names from execution IDs.
• Metrics reset on restart. Counters and gauges live in process memory; no state is persisted. A server restart zeros flux_workers_active, flux_execution_queue_depth, and every _total counter. Use Prometheus rate() and increase() for counter math — both already tolerate resets — and rely on the workers’ next connect to repopulate the active-worker gauge.