OpenTelemetry

Flux’s OpenTelemetry integration emits traces, metrics, and logs through the standard OTel SDK. Configuration is one TOML block.

Enable OTel

Install the observability extra:

pip install 'flux-core[observability]'

The extra pulls the OTel SDK, the OTLP exporters, and the Prometheus exporter. Configure in flux.toml:

[flux.observability]
enabled = true
service_name = "flux-prod"
otlp_endpoint = "http://otel-collector:4317"
otlp_protocol = "grpc"
trace_sample_rate = 0.1
metric_export_interval = 60

The settings:

• enabled — master switch. Without it, nothing else in this section runs.
• service_name — sets service.name on every OTel resource. Required for most backends to group spans.
• otlp_endpoint — OTLP collector endpoint. Traces, metrics, and logs all go here when set.
• otlp_protocol — "grpc" or "http" (HTTP/protobuf). Defaults to "grpc".
• trace_sample_rate — 0.0 to 1.0. The provider uses TraceIdRatioBased sampling, so the rate is applied to the trace ID hash. Defaults to 1.0 (sample everything).
• metric_export_interval — seconds between OTLP metric pushes. Defaults to 60.
• resource_attributes — optional dict of extra resource attributes (environment, region, etc.).

Restart the server and worker after changes — observability config is read once at startup.

OTLP transport protocol

The OTLP exporter protocol is configurable through otlp_protocol. It applies uniformly to the trace, metric, and log exporters:

• otlp_protocol = "grpc" (the default) uses the gRPC OTLP exporters. For an OpenTelemetry Collector, the gRPC receiver listens on port 4317.
• otlp_protocol = "http" uses the OTLP HTTP/protobuf exporters. The Collector’s HTTP receiver listens on port 4318.

Set otlp_endpoint to a host/port that matches the chosen protocol. If your trace backend only accepts one of the two, pick the matching otlp_protocol, or run a Collector in the middle to convert. Honeycomb and Datadog both support OTLP/gRPC and OTLP/HTTP.

A typical Collector config:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 10s
    send_batch_size: 1000
  resource:
    attributes:
      - key: environment
        value: production
        action: upsert

exporters:
  otlphttp/backend:
    endpoint: https://your-backend.example.com/otlp
    headers:
      api-key: ${env:BACKEND_API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [otlphttp/backend]
    metrics:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [otlphttp/backend]

Run the Collector alongside Flux and point otlp_endpoint at it.

The span shape

Flux instruments exactly three span names. All emitted by the worker process:

The workflow span is the parent of every task span in the same execution. A typical trace has one workflow span at the root with N task spans as children.

What is not instrumented as a span:

• HTTP requests (server-side route handlers). HTTP duration is captured as a metric, not a span.
• Worker registration, claim, or checkpoint calls.
• Agent loop iterations or individual tool calls (these live inside flux.task.execute).
• LLM provider calls.
• Schedule firing.

If you need spans for any of those, instrument them yourself with flux.observability.tracing.traced:

from flux.observability.tracing import traced
from flux import task


@task
@traced("my.task.preprocess", attributes={"step": "preprocess"})
async def preprocess(data: dict) -> dict:
    # ... your code
    return data

The decorator wraps the function in an OTel span; nested under whatever span is current when the task runs.

Trace context propagation

Flux propagates trace context server → worker on every dispatch event. The server injects the W3C traceparent headers into the event payload, and the worker extracts them before starting its span. This applies to all three dispatch paths:

• execution_scheduled — the worker uses the inherited context as the parent of flux.workflow.execute.
• execution_resumed — the worker uses the inherited context as the parent of flux.workflow.resume.
• execution_cancelled — the worker uses the inherited context when handling the cancellation.

A trace started by an HTTP request on the server flows into the worker’s workflow span, and a workflow that pauses and resumes stitches together as a single connected trace. Spans across the initial execute and the resume share the same root trace, so they line up in the trace UI without manual correlation by execution ID.

Span status on failure

Failed spans set the OTel ERROR status, so trace UIs that color spans by status render failures correctly:

• flux.task.execute — when the task body raises, the span calls set_status(Status(StatusCode.ERROR, ...)) and records the exception with record_exception. The exception type and message are attached as span events.
• flux.workflow.execute and flux.workflow.resume — when the workflow fails, the span sets StatusCode.ERROR with the failure output.

A failed task shows up red in the trace UI, and the exception detail is available on the span without cross-referencing the worker logs.

Sampling for production

trace_sample_rate = 1.0 is fine for development. In production, dropping to 0.1 or 0.01 keeps backend costs and ingestion volume down without losing the ability to find issues:

[flux.observability]
trace_sample_rate = 0.1  # 10% of traces

The sampler is TraceIdRatioBased, which means the decision is made at trace creation time based on the trace ID. Every span in a sampled trace is exported; no span in an unsampled trace is. This is consistent with how every OTel-aware system samples — once a trace is in or out, it stays that way across processes.

Metrics are not sampled. The full 21-metric catalog (see the Grafana + Prometheus page) exports every 60 seconds regardless of trace_sample_rate.

Logs over OTLP

When otlp_endpoint is set and observability is enabled, Flux installs an OTLP log exporter on the root flux logger. Log records get trace/span IDs attached (via OTelTraceLogFilter) when a span is active, so you can pivot from a span to its logs in backends that support it.

Two things to know:

• Flux logs go to stdout, not stderr. Container log collectors that read only stderr miss everything.
• There is no JSON formatter in 0.56.0. FLUX_LOG_FORMAT accepts a stdlib logging.Formatter template string. If your log pipeline needs JSON, transform downstream.

Configuration reference

Full set of options under [flux.observability]:

Common problems

No spans appear. Check enabled = true and that otlp_endpoint is reachable from the server/worker. Flux logs Observability initialized (prometheus=..., otlp=enabled, sample_rate=...) at startup when setup succeeds.

Spans appear from the worker but not from the server. That is expected. The server emits HTTP metrics but does not instrument route handlers as spans. Workflow spans live on the worker.

Endpoint refused / wrong port. The port must match otlp_protocol: gRPC receivers listen on 4317, HTTP receivers on 4318. A protocol/port mismatch surfaces as connection or export errors.

Resume spans start a new trace. Should not happen — trace context propagates on execution_resumed. Confirm both server and worker have observability enabled and were restarted after config changes.

Derived against Flux 0.56.0 with OpenTelemetry SDK 1.x, 2026-07.