Tracing

Wiring up OpenTelemetry traces for Flux — what's traced, OTLP exporter setup, sampling, and the span context limits between server and workers.

Flux ships an OpenTelemetry tracer that wraps every workflow execution, every workflow resume, and every task call. Spans propagate from the server’s SSE dispatch into the worker, so a single trace stitches the scheduling decision and the worker-side execution together. This page covers how to turn tracing on, what you get, and where the instrumentation stops.

Enabling the tracer

Tracing rides on the same [flux.observability] block as metrics. Install the extra and flip the flag:

poetry install --extras observability

# flux.toml
[flux.observability]
enabled = true
service_name = "flux"
otlp_endpoint = "http://localhost:4317"
trace_sample_rate = 1.0
metric_export_interval = 60

[flux.observability.resource_attributes]
deployment.environment = "prod"

The same fields are reachable via env vars (standard FLUX_ prefix, __ as nested delimiter): FLUX_OBSERVABILITY__ENABLED, FLUX_OBSERVABILITY__OTLP_ENDPOINT, FLUX_OBSERVABILITY__TRACE_SAMPLE_RATE, and so on. resource_attributes is a dict and is awkward to set from the environment; prefer flux.toml for it.

Set the same values on every server and worker process — the tracer is initialized per-process at startup.

What gets a span

Three span names exist in the codebase today:

Span	Where it starts	Key attributes
`flux.workflow.execute`	Worker, on `execution_scheduled` SSE event	`flux.workflow.name`, `flux.execution.id`, `flux.worker.name`
`flux.workflow.resume`	Worker, on `execution_resumed` SSE event	same as above
`flux.task.execute`	Inside `task.__call__`, parented to the active workflow span	`flux.task.name`, `flux.workflow.name`

Workflow spans set the span status to ERROR with ctx.output as the message when the execution finishes in a failed state. Task spans do the same: a task that raises sets its flux.task.execute span to ERROR with the exception text (flux/task.py:322), so a failed task is visible directly on its own span without walking up to the parent workflow.

There are no automatic HTTP-request spans. The MetricsMiddleware in flux/observability/middleware.py records HTTP count and duration as metrics, but does not start a span. There are no spans on agent(...) calls, on tool invocations, or on MCP-client requests either — wrap those yourself with the @traced decorator from flux.observability.tracing if you need them.

Exporting to a collector

The exporter is OTLP, and the wire protocol is configurable. otlp_protocol accepts grpc (the default) or http (HTTP/protobuf):

[flux.observability]
otlp_endpoint = "http://localhost:4318"
otlp_protocol = "http"

Flux selects the matching OpenTelemetry exporter at startup based on this setting (flux/observability/provider.py). Use grpc against port 4317 and http against port 4318 — match the protocol to the endpoint your collector exposes.

There is no built-in Jaeger thrift exporter or console exporter. To land traces in Jaeger, Honeycomb, Datadog APM, Grafana Tempo, Lightstep, or AWS X-Ray, point otlp_endpoint at an OpenTelemetry Collector. Spans go through a BatchSpanProcessor, so expect a few seconds of latency before they show up.

Sampling

trace_sample_rate is a head-based ratio passed into OTel’s TraceIdRatioBased sampler. 1.0 traces everything, 0.0 traces nothing, 0.1 keeps roughly one in ten root traces. The decision is made at the root and propagates: once a workflow span is sampled in, every task span underneath it is too. The default of 1.0 is fine for development; drop it under production load.

Context propagation across processes

The server injects W3C trace context (traceparent, tracestate) as a trace_context dict into every workflow-dispatch SSE event — execution_scheduled, execution_resumed, and execution_cancelled alike (flux/server.py:_inject_trace_context). The worker reads it back, calls extract_trace_context, and uses it as the parent when starting flux.workflow.execute / flux.workflow.resume. Result: when a server-side request triggers, resumes, or cancels a workflow, the worker-side spans hang off the original trace, not a fresh one — a single trace spans the whole pause-and-resume lifecycle.

Within a single workflow run, every flux.task.execute span is a child of the surrounding workflow span by virtue of running inside the same start_as_current_span block.

Common failures

No spans in your backend. Check that otlp_endpoint is reachable from the Flux host and that otlp_protocol matches what the collector speaks — grpc on 4317, http on 4318. A protocol/port mismatch produces silent export failures.
High overhead at full sampling. Drop trace_sample_rate to 0.1 or lower.