Datadog exporter (via OTLP)

Configure the Datadog exporter for tracing


Enable and configure the Datadog exporter for tracing in the GraphOS Router or Apollo Router Core.

For general tracing configuration, refer to Router Tracing Configuration .

OTLP configuration

To export traces to Datadog via OTLP, you must do the following:

  • Configure the Datadog agent to accept OTLP traces.

  • Configure the router to send traces to the Datadog agent.

To configure the Datadog agent, add OTLP configuration to your datadog.yaml. For example:

YAML
datadog.yaml
1otlp_config:
2  receiver:
3    protocols:
4      grpc:
5        endpoint: <dd-agent-ip>:4317

To configure the router, enable the OTLP exporter and set endpoint: <datadog-agent-endpoint>. For example:

YAML
router.yaml
1telemetry:
2  exporters:
3    tracing:
4      otlp:
5        enabled: true
6   
7        # Optional endpoint, either 'default' or a URL (Defaults to http://127.0.0.1:4317)
8        endpoint: "${env.DATADOG_AGENT_HOST}:4317"
9

For more details about Datadog configuration, see Datadog Agent configuration .

Enabling log correlation

To enable Datadog log correlation, you must configure dd.trace_id to appear on the router span:

YAML
router.yaml
1telemetry:
2  instrumentation:
3    spans:
4      mode: spec_compliant
5      router:
6        attributes:
7          dd.trace_id: true #highlight-line

Your JSON formatted log messages will automatically output dd.trace_id on each log message if dd.trace_id was detected on the router span.

Datadog native configuration

⚠️ caution
Native Datadog tracing is not part of the OpenTelemetry spec, and given that Datadog supports OTLP we will be deprecating native Datadog tracing in the future. Use OTLP configuration instead.

The router can be configured to connect to either the native, default Datadog agent address or a URL:

YAML
router.yaml
1telemetry:
2  exporters:
3    tracing:
4     datadog:
5       enabled: true
6       # Optional endpoint, either 'default' or a URL (Defaults to http://127.0.0.1:8126)
7       endpoint: "http://${env.DATADOG_AGENT_HOST}:8126"
8
9  # Enable graphql.operation.name attribute on supergraph spans.
10  instrumentation:
11    spans:
12      mode: spec_compliant
13      supergraph:
14        attributes:
15          graphql.operation.name: true

enabled

Set to true to enable the Datadog exporter. Defaults to false.

enable_span_mapping (default: true)

There are some incompatibilities between Datadog and OpenTelemetry, the Datadog exporter might not provide meaningful contextual information in the exported spans. To fix this, you can configure the router to perform a mapping for the span name and the span resource name.

YAML
router.yaml
1telemetry:
2  exporters:
3     tracing:
4       datadog:
5         enabled: true
6         enable_span_mapping: true

With enable_span_mapping: true, the router performs the following mapping:

  1. Use the OpenTelemetry span name to set the Datadog span operation name.

  2. Use the OpenTelemetry span attributes to set the Datadog span resource name.

Example trace

For example, assume a client sends a query MyQuery to the router. The router's query planner sends a subgraph query to my-subgraph-name and creates the following trace:

Text
1    | apollo_router request                                                                 |
2        | apollo_router router                                                              |
3            | apollo_router supergraph                                                      |
4            | apollo_router query_planning  | apollo_router execution                       |
5                                                | apollo_router fetch                       |
6                                                    | apollo_router subgraph                |
7                                                        | apollo_router subgraph_request    |

As you can see, there is no clear information about the name of the query, the name of the subgraph, and the name of the query sent to the subgraph.

Instead, when enable_span_mapping is set to true the following trace will be created:

Text
1    | request /graphql                                                                                   |
2        | router /graphql                                                                                         |
3            | supergraph MyQuery                                                                         |
4                | query_planning MyQuery  | execution                                                    |
5                                              | fetch fetch                                              |
6                                                  | subgraph my-subgraph-name                            |
7                                                      | subgraph_request MyQuery__my-subgraph-name__0    |

fixed_span_names (default: true)

When fixed_span_names: true, the apollo router to use the original span names instead of the dynamic ones as described by OTel semantic conventions.

YAML
router.yaml
1telemetry:
2  exporters:
3     tracing:
4       datadog:
5         enabled: true
6         fixed_span_names: true

This will allow you to have a finite list of operation names in Datadog on the APM view.

resource_mapping

When set, resource_mapping allows you to specify which attribute to use in the Datadog APM and Trace view. The default resource mappings are:

OpenTelemetry Span NameDatadog Span Operation Name
requesthttp.route
routerhttp.route
supergraphgraphql.operation.name
query_planninggraphql.operation.name
subgraphsubgraph.name
subgraph_requestgraphql.operation.name
http_requesthttp.route

You may override these mappings by specifying the resource_mapping configuration:

YAML
router.yaml
1telemetry:
2  exporters:
3     tracing:
4       datadog:
5         enabled: true
6         resource_mapping:
7           # Use `my.span.attribute` as the resource name for the `router` span
8           router: "my.span.attribute"
9  instrumentation:
10    spans:
11      router:
12        attributes:
13          # Add a custom attribute to the `router` span
14          my.span.attribute:
15            request_header: x-custom-header

If you have introduced a new span in a custom build of the Router you can enable resource mapping for it by adding it to the resource_mapping configuration.

span_metrics

When set, span_metrics allows you to specify which spans will show span metrics in the Datadog APM and Trace view. By default, span metrics are enabled for:

  • request

  • router

  • supergraph

  • subgraph

  • subgraph_request

  • http_request

  • query_planning

  • execution

  • query_parsing

You may override these defaults by specifying span_metrics configuration:

The following will disable span metrics for the supergraph span.

YAML
router.yaml
1telemetry:
2  exporters:
3    tracing:
4      datadog:
5        enabled: true
6        span_metrics:
7          # Disable span metrics for supergraph
8          supergraph: false
9          # Enable span metrics for my_custom_span
10          my_custom_span: true

If you have introduced a new span in a custom build of the Router you can enable span metrics for it by adding it to the span_metrics configuration.

batch_processor

All exporters support configuration of a batch span processor with batch_processor.

You must tune your batch_processor configuration if you see any of the following messages in your logs:

  • OpenTelemetry trace error occurred: cannot send span to the batch span processor because the channel is full

  • OpenTelemetry metrics error occurred: cannot send span to the batch span processor because the channel is full

The exact settings depend on the bandwidth available for you to send data to your application peformance monitor (APM) and the bandwidth configuration of your APM. Expect to tune these settings over time as your application changes.

YAML
1telemetry:
2  exporters:
3    tracing:
4      datadog: 
5        batch_processor: 
6          max_export_batch_size: 512
7          max_concurrent_exports: 1
8          max_export_timeout: 30s 
9          max_queue_size: 2048
10          scheduled_delay: 5s

batch_processor configuration reference

AttributeDefaultDescription
scheduled_delay5sThe delay in seconds from receiving the first span to sending the batch.
max_concurrent_exports1The maximum number of overlapping export requests.
max_export_batch_size512The number of spans to include in a batch. May be limited by maximum message size limits.
max_export_timeout30sThe timeout in seconds for sending spans before dropping the data.
max_queue_size2048The maximum number of spans to be buffered before dropping span data.

Datadog native configuration reference

AttributeDefaultDescription
enabledfalseEnable the OTLP exporter.
enable_span_mappingfalseIf span mapping should be used.
endpointhttp://localhost:8126/v0.4/tracesThe endpoint to send spans to.
batch_processorThe batch processor settings.
resource_mappingSee config A map of span names to attribute names.
span_metricsSee config A map of span names to boolean.