Honeycomb

Ship Flux traces to Honeycomb directly over OTLP — covers service-map shape and high-cardinality workflow attributes.

Honeycomb is OTLP-native, so Flux ships traces directly to api.honeycomb.io with no Collector hop required. The integration is a few lines of flux.toml plus a Honeycomb API key.

Setup

Install the observability extra:

pip install 'flux-core[observability]'

Honeycomb accepts OTLP over gRPC at api.honeycomb.io:443 with the API key passed as a header. Flux’s exporter wiring does not expose a headers parameter directly, so the standard approach is to set the OTEL_EXPORTER_OTLP_HEADERS environment variable, which the OTel exporter reads:

export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=<your-api-key>"
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="x-honeycomb-team=<your-api-key>"

Then in flux.toml:

[flux.observability]
enabled = true
service_name = "flux-prod"
otlp_endpoint = "https://api.honeycomb.io:443"
trace_sample_rate = 0.1

Notes on the endpoint:

The default otlp_protocol is grpc. The OTLP gRPC exporter accepts https:// URLs to enable TLS; without the scheme, the default is plaintext gRPC.
To send over OTLP/HTTP instead, set otlp_protocol = "http" and point otlp_endpoint at Honeycomb’s HTTP endpoint (https://api.honeycomb.io).
For Honeycomb EU instances, use https://api.eu1.honeycomb.io:443.

For production, store the API key as a Flux secret and inject it through a worker startup hook rather than baking it into the environment.

Service map

Honeycomb auto-generates a service map from spans. Flux’s instrumentation produces a flat shape with at most three nodes per execution:

The service that originates the workflow trace (often the server, when an HTTP request triggers a workflow).
flux-prod (whatever you set service_name to) — emits flux.workflow.execute and flux.workflow.resume.
flux-prod again as the parent of flux.task.execute spans.

The map is intentionally shallow because Flux instruments only three span names. HTTP request handlers, agent loops, individual tool calls, and LLM provider calls do not show up as spans. The OpenTelemetry page covers this in detail; the short version is that everything between workflow start and workflow end is one parent span with task children.

To get a richer map, instrument your own code with the traced helper:

from flux.observability.tracing import traced
from flux import task


@task
@traced("integration.salesforce.query", attributes={"system": "salesforce"})
async def query_salesforce(account_id: str) -> dict:
    # ... real call
    return {"account_id": account_id}

The decorator wraps the function in an OTel span that nests under the surrounding flux.task.execute. Honeycomb sees it as a child; the service map shows a salesforce edge.

High-cardinality attributes (and why they’re fine)

Honeycomb’s positioning is “high cardinality is free.” Two attribute conventions in Flux play to that:

flux.workflow.namespace and flux.workflow.name are on every workflow span and every task span. Faceting by workflow name is the first BubbleUp query you will want.
flux.task.name is on every task span. Faceting by task name spots slow tasks across all workflows.
flux.execution.id is on every workflow span. This is the one tag that gives you per-execution drilldown — useful for jumping from “this execution failed” to “show me everything that happened in this trace.”

In a traditional metrics system, flux.execution.id would be a cardinality bomb. In Honeycomb, it is the most useful attribute on the trace.

BubbleUp on task duration

A typical investigation: “task duration spiked at 14:00, what changed?” In Honeycomb:

Filter name = flux.task.execute.
Heatmap duration_ms.
BubbleUp the spike.

The likely outcomes:

BubbleUp highlights flux.workflow.name = report_export → a specific workflow regressed.
BubbleUp highlights flux.task.name = upload_to_s3 → a specific task across workflows got slower (likely an upstream).
BubbleUp shows nothing distinctive → something cross-cutting changed (worker host, network, downstream API).

The third case is where you should reach for metrics. The 25 Flux Prometheus metrics — covered in detail on the Grafana + Prometheus page — answer “is the queue backed up?”, “are workers disconnecting?”, “is a worker’s event loop starving?” (flux_worker_loop_lag_seconds, flux_worker_health_transitions_total), “is the module cache hit rate dropping?” Spans answer “what is one execution doing?”; metrics answer “what is the fleet doing in aggregate?”

What Flux does not span (and what to add)

Spans that are missing from Flux 0.56.0’s instrumentation and worth adding yourself:

Missing	How to add
HTTP request handling on the server	Honeycomb has no FastAPI auto-instrumentation hook in Flux’s setup. Install `opentelemetry-instrumentation-fastapi` and call its `instrument_app(app)` inside a custom startup hook.
Agent loop iterations	Use `@traced("agent.iteration", attributes=...)` inside custom tools, or wrap your `agent()` call site with `traced`.
LLM provider calls	Wrap calls inside `@task` bodies with `@traced("llm.openai.chat", attributes={"model": "gpt-4o"})`.
Tool calls	Each tool is a `@task`, so it already gets `flux.task.execute`. Add a richer span name with `@traced` if you want to distinguish a tool from a regular task.

Pause/resume traces

As covered on the OpenTelemetry page, trace context propagates across pause and resume, so a workflow that pauses and resumes appears as one connected trace in Honeycomb — the flux.workflow.resume span shares the root trace with the original flux.workflow.execute span. To pull every span for one execution regardless of pause/resume, query by flux.execution.id:

flux.execution.id = <id>

Sort by start time; you get the execute span first and any resume spans after.

Derived against Honeycomb (US instance) and Flux 0.56.0, 2026-07.