Datadog exporter (via OTLP)
Configure the Datadog exporter for tracing
Enable and configure the Datadog exporter for tracing in the Apollo Router.
For general tracing configuration, refer to Router Tracing Configuration.
OTLP configuration
To export traces to Datadog via OTLP, you must do the following:
- Configure the Datadog agent to accept OTLP traces.
- Configure the router to send traces to the Datadog agent.
To configure the Datadog agent, add OTLP configuration to your datadog.yaml
. For example:
otlp_config:receiver:protocols:grpc:endpoint: <dd-agent-ip>:4317
To configure the router, enable the OTLP exporter and set endpoint: <datadog-agent-endpoint>
. For example:
telemetry:exporters:tracing:otlp:enabled: true# Optional endpoint, either 'default' or a URL (Defaults to http://127.0.0.1:4317)endpoint: "${env.DATADOG_AGENT_HOST}:4317"
For more details about Datadog configuration, see Datadog Agent configuration.
Enabling log correlation
To enable Datadog log correlation, you must configure dd.trace_id
to appear on the router
span:
telemetry:instrumentation:spans:mode: spec_compliantrouter:attributes:dd.trace_id: true
Your JSON formatted log messages will automatically output dd.trace_id
on each log message if dd.trace_id
was detected on the router
span.
Datadog native configuration
⚠️ CAUTION
Native Datadog tracing is not part of the OpenTelemetry spec, and given that Datadog supports OTLP we will be deprecating native Datadog tracing in the future. Use OTLP configuration instead.
The Apollo Router can be configured to connect to either the native, default Datadog agent address or a URL:
telemetry:exporters:tracing:datadog:enabled: true# Optional endpoint, either 'default' or a URL (Defaults to http://127.0.0.1:8126)endpoint: "http://${env.DATADOG_AGENT_HOST}:8126"
enabled
Set to true to enable the Datadog exporter. Defaults to false.
enable_span_mapping
There are some incompatibilities between Datadog and OpenTelemetry, the Datadog exporter might not provide meaningful contextual information in the exported spans. To fix this, you can configure the Apollo Router to perform a mapping for the span name and the span resource name.
telemetry:exporters:tracing:datadog:enabled: trueenable_span_mapping: true
When enable_span_mapping: true
, the Apollo Router will perform the following mapping:
- Use the OpenTelemetry span name to set the Datadog span operation name.
- Use the OpenTelemetry span attributes to set the Datadog span resource name.
Example trace
Let's say we send a query MyQuery
to the Apollo Router, then the Router using the operation's query plan will send a query to my-subgraph-name
and the following trace will be created:
| apollo_router request || apollo_router router || apollo_router supergraph || apollo_router query_planning | apollo_router execution || apollo_router fetch || apollo_router subgraph || apollo_router subgraph_request |
As you can see, there is no clear information about the name of the query, the name of the subgraph, and the name of the query sent to the subgraph.
Instead, when enable_span_mapping
is set to true
the following trace will be created:
| request /graphql || router || supergraph MyQuery || query_planning MyQuery | execution || fetch fetch || subgraph my-subgraph-name || subgraph_request MyQuery__my-subgraph-name__0 |
batch_processor
All exporters support configuration of a batch span processor with batch_processor
.
You must tune your batch_processor
configuration if you see any of the following messages in your logs:
OpenTelemetry trace error occurred: cannot send span to the batch span processor because the channel is full
OpenTelemetry metrics error occurred: cannot send span to the batch span processor because the channel is full
The exact settings depend on the bandwidth available for you to send data to your application peformance monitor (APM) and the bandwidth configuration of your APM. Expect to tune these settings over time as your application changes.
telemetry:exporters:tracing:datadog:batch_processor:max_export_batch_size: 512max_concurrent_exports: 1max_export_timeout: 30smax_queue_size: 2048scheduled_delay: 5s
batch_processor
configuration reference
Attribute | Default | Description |
---|---|---|
scheduled_delay | 5s | The delay in seconds from receiving the first span to sending the batch. |
max_concurrent_exports | 1 | The maximum number of overlapping export requests. |
max_export_batch_size | 512 | The number of spans to include in a batch. May be limited by maximum message size limits. |
max_export_timeout | 30s | The timeout in seconds for sending spans before dropping the data. |
max_queue_size | 5 | The maximum number of spans to be buffered before dropping span data. |
Datadog native configuration reference
Attribute | Default | Description |
---|---|---|
enabled | false | Enable the OTLP exporter. |
enable_span_mapping | false | If span mapping should be used. |
endpoint | http://localhost:8126/v0.4/traces | The endpoint to send spans to. |
batch_processor | The batch processor settings. |