Datadog

Datadog accepts OTLP traces and metrics directly, so Flux’s OpenTelemetry output lands in Datadog APM and Datadog Metrics without a translation layer. The path goes through the Datadog Agent.

Architecture

Two parallel pipelines:

1. Traces: Flux server + workers → OTLP/gRPC → Datadog Agent’s OTLP receiver → Datadog APM.
2. Metrics: Datadog Agent’s openmetrics check scrapes Flux’s /metrics endpoint → Datadog Metrics.

The Datadog Agent runs as a sidecar (in Kubernetes) or as a host-level daemon, and Flux talks to it locally. The Agent then handles the network egress to Datadog’s intake. You can also point Flux directly at Datadog’s OTLP intake — but going through the Agent gives you tag enrichment (kube_namespace, kube_deployment, etc.) for free.

Configure the Datadog Agent

In the Agent’s datadog.yaml, enable the OTLP receiver:

otlp_config:
  receiver:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
  traces:
    enabled: true
  metrics:
    enabled: true
  logs:
    enabled: true

This opens port 4317 on the Agent for OTLP/gRPC, which matches Flux’s default otlp_protocol = "grpc". To use OTLP/HTTP instead, enable the Agent’s http protocol (port 4318) and set otlp_protocol = "http" in flux.toml.

For the Prometheus scrape, add the openmetrics integration:

# /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml
init_config:

instances:
  - openmetrics_endpoint: http://flux-server:8000/metrics
    namespace: flux
    metrics:
      - flux_workflow_executions_total
      - flux_workflow_execution_duration_seconds
      - flux_task_executions_total
      - flux_task_execution_duration_seconds
      - flux_task_retries_total
      - flux_execution_queue_depth
      - flux_execution_schedule_to_start_seconds
      - flux_workers_active
      - flux_worker_disconnections_total
      - flux_worker_executions_active
      - flux_schedule_triggers_total
      - flux_resume_queue_depth
      - flux_http_requests_total
      - flux_http_request_duration_seconds
      - flux_module_cache_total
    headers:
      Authorization: Bearer <api-key>

The API key needs admin:metrics:read. See the Grafana + Prometheus page for how to provision a custom role and a scraper principal — the steps are the same.

Restart the Agent after editing either file.

Configure Flux

Point otlp_endpoint at the Datadog Agent’s OTLP port:

[flux.observability]
enabled = true
service_name = "flux-prod"
otlp_endpoint = "http://datadog-agent:4317"
trace_sample_rate = 0.1

If the Agent runs as a sidecar in the same pod, http://localhost:4317 works. If the Agent runs on a separate node, use the node’s address.

The full catalog of metrics ends up in Datadog as flux.flux_workflow_executions_total, flux.flux_task_execution_duration_seconds, and so on (the namespace: flux prefix in the openmetrics config prepends flux. to each metric name).

APM and trace correlation

Datadog APM expects each span to belong to a service. Flux’s service_name config (which sets the OTel service.name resource attribute) becomes the Datadog service name. Set it to something distinguishable per environment:

service_name = "flux-prod"   # production
service_name = "flux-staging"  # staging

Spans appear in APM under that service. The three span names — flux.workflow.execute, flux.workflow.resume, flux.task.execute — show up as resources. Workflow attributes (flux.workflow.namespace, flux.workflow.name) become span tags that you can group by in APM views.

Two things to know up front, both covered in detail on the OpenTelemetry page:

• HTTP request handling is not instrumented as spans, so Datadog APM does not show the server’s REST API as endpoints. HTTP metrics are still in Datadog Metrics through the openmetrics scrape.
• Trace context propagates across pause and resume, so a paused-and-resumed workflow appears as one connected trace. Failed task spans set the OTel ERROR status and record the exception, so they surface as errors in APM.

Log correlation

If your Flux processes log to stdout and the Datadog Agent collects logs (logs_enabled: true in datadog.yaml), trace IDs end up in log records via the OTel log filter Flux installs. Datadog correlates logs to traces by the trace_id field.

Two adjustments to make this work cleanly:

• Flux logs to stdout in a stdlib Formatter template. To get JSON-formatted logs with trace_id as a top-level field, set FLUX_LOG_FORMAT to include %(otelTraceID)s and post-process upstream, or pre-parse in a log pipeline.
• The Datadog log pipeline for Flux should remap otelTraceID to trace_id and otelSpanID to span_id.

Dashboard suggestions

Build a Datadog dashboard with these widgets. Each query assumes the openmetrics namespace prefix from the config above.

• Workers active (query value): avg:flux.flux_workers_active{*}.
• Execution queue depth (timeseries): avg:flux.flux_execution_queue_depth{*}.
• p95 schedule-to-start latency (timeseries): p95:flux.flux_execution_schedule_to_start_seconds{*}.
• Workflow failure rate (timeseries): sum:flux.flux_workflow_executions_total{status:failed}.as_rate().
• Task retries by workflow (top list): sum:flux.flux_task_retries_total{*} by {workflow_name}.as_rate().
• HTTP 5xx rate (timeseries): sum:flux.flux_http_requests_total{status_code:5*}.as_rate().
• APM service map for flux-prod: shows workflow → task call structure.

Monitors

Two starting alerts:

• Workers down: alert when avg:flux.flux_workers_active{*} <= 0 for 2 minutes.
• Queue backed up: alert when avg:flux.flux_execution_queue_depth{*} > 50 for 5 minutes.

Common problems

Spans appear but service map is empty. Datadog needs a service.name resource attribute. Confirm service_name is set in flux.toml and the Agent is not overriding it via otlp_config.metrics.resource_attributes.

Metrics show up but with no tags. Datadog Agent v7.x adds container/pod tags automatically when running in Kubernetes. If you are on a VM, set tags: in the openmetrics check config.

Trace and metrics quotas spike. Lower trace_sample_rate to 0.05 or 0.01. Metrics are not sampled — Datadog ingestion charges by metric × tag combination, so high-cardinality custom tags in your workflows can drive costs up faster than trace volume.

Derived against Datadog Agent 7.x and Flux 0.56.0, 2026-07.