Tracing in the Apollo Router

Collect tracing information

The Apollo Router supports OpenTelemetry, with exporters for:

Jaeger
Zipkin
Datadog
OpenTelemetry Protocol (OTLP) over HTTP or gRPC

The Apollo Router generates spans that include the various phases of serving a request and associated dependencies. This is useful for showing how response time is affected by:

Sub-request response times
Query shape (sub-request dependencies)
Apollo Router post-processing

Span data is sent to a collector such as Jaeger, which can assemble spans into a gantt chart for analysis.

💡 TIP

To get the most out of distributed tracing, all components in your system should be instrumented.

Common configuration

Service name

router.yaml

1
telemetry:
2
  tracing:
3
    common:
4
      # (Optional) Set the service name to easily find traces related to the apollo-router in your metrics dashboards
5
      service_name: "router"

Service name discovery is handled in the following order:

OTEL_SERVICE_NAME env
OTEL_RESOURCE_ATTRIBUTES env
router.yaml service_name
router.yaml resource

If none of the above are found then the service name will be set to unknown_service:router or unknown_service if the executable name cannot be determined.

Resource

A Resource is a set of key-value pairs that provide additional global information exported with your traces.

router.yaml

1
telemetry:
2
  tracing:
3
    common:
4
      resource:
5
        "environment.name": "production"
6
        "environment.namespace": "{env.MY_K8_NAMESPACE_ENV_VARIABLE}"

See OpenTelemetry conventions for resources.

Sampling

To prevent sending to many traces to your APM you may want to enable sampling.

router.yaml

1
telemetry:
2
  tracing:
3
    common:
4
      sampler: always_on # (default) all requests are sampled (always_on|always_off|<0.0-1.0>)
5
      parent_based_sampler: true # (default) If an incoming span has OpenTelemetry headers then the request will always be sampled.

Setting sampler to 0.1 will result in only 10% of your requests being sampled.

parent_based_sampler enables clients to make the sampling decision. This guarantees that a trace that starts at a client will also have spans at the router. You may wish to turn this off if your router is exposed directly to the internet.

Propagation

The propagation section allows you to configure which propagators are active in addition to those automatically activated by using an exporter.

router.yaml

1
telemetry:
2
  tracing:
3
    propagation:
4
      # https://www.w3.org/TR/baggage/
5
      baggage: false
6

7
      # https://www.datadoghq.com/
8
      datadog: false
9

10
      # https://www.jaegertracing.io/ (compliant with opentracing)
11
      jaeger: false
12

13
      # https://www.w3.org/TR/trace-context/
14
      trace_context: false
15

16
      # https://zipkin.io/ (compliant with opentracing)
17
      zipkin: false
18

19
      # https://aws.amazon.com/xray/ (compliant with opentracing)
20
      aws_xray: false
21

22
      # If you have your own way to generate a trace id and you want to pass it via a custom request header
23
      request:
24
        header_name: my-trace-id

Specifying explicit propagation is generally only required if you're using an exporter that supports multiple trace ID formats (e.g., OpenTelemetry Collector, Jaeger, or OpenTracing compatible exporters).

Span limits

You may set limits on spans to prevent sending too much data to your APM.

router.yaml

1
telemetry:
2
  tracing:
3
    common:
4

5
      # Optional limits
6
      max_attributes_per_event: 10
7
      max_attributes_per_link: 10
8
      max_attributes_per_span: 10
9
      max_events_per_span: 10
10
      max_links_per_span: 10

Trace ID

This feature is currently experimental. Your questions and feedback are highly valued—don't hesitate to get in touch with your Apollo contact or on the official Apollo GraphQL Discord. You can also give feedback in the discussion on GitHub.

If you want to expose in response headers the generated trace ID or the one you provided using propagation headers you can use this configuration:

router.yaml

1
telemetry:
2
  tracing:
3
    experimental_response_trace_id:
4
      enabled: true # default: false
5
      header_name: "my-trace-id" # default: "apollo-trace-id"

Using this configuration you will have a response header called my-trace-id containing the trace ID. It could help you to debug a specific query if you want to grep your log with this trace id to have more context.

Batch Processor

All trace exporters (apollo|datadog|zipkin|jaeger|otlp) have batch span processor configuration, it will be necessary to tune this if you see the following in your logs:

OpenTelemetry trace error occurred: cannot send span to the batch span processor because the channel is full

scheduled_delay The delay from receiving the first span to the batch being sent.
max_concurrent_exports The maximum number of overlapping export requests. For instance if ingest is taking a long time to respond there may be several overlapping export requests.
max_export_batch_size The number of spans to include in a batch. Your ingest may have max message size limits.
max_export_timeout The timeout for sending spans before dropping the data.
max_queue_size The maximum number of spans to be buffered before dropping span data.

router.yaml

1
telemetry:
2
  # Apollo tracing
3
  apollo:
4
    batch_processor:
5
      scheduled_delay: 100ms
6
      max_concurrent_exports: 1000
7
      max_export_batch_size: 10000
8
      max_export_timeout: 100s
9
      max_queue_size: 10000
10

11
  tracing:
12
    # Datadog
13
    datadog:
14
      enabled: true
15
      batch_processor:
16
        scheduled_delay: 100ms
17
        max_concurrent_exports: 1000
18
        max_export_batch_size: 10000
19
        max_export_timeout: 100s
20
        max_queue_size: 10000
21
      endpoint: default
22
    # Jaeger
23
    # Otlp
24
    # Zipkin

You will need to experiment to find the setting that are appropriate for your use case.

Using Datadog (native)

The Apollo Router can be configured to connect to either the default agent address or a URL.

router.yaml

1
telemetry:
2
  tracing:
3
    datadog:
4
      enabled: true
5
      # Optional endpoint, either 'default' or a URL (Defaults to http://localhost:8126/v0.4/traces)
6
      endpoint: "http://${env.DATADOG_AGENT_HOST}:8126/v0.4/traces"

Given that there are some incompatibilities between Datadog and OpenTelemetry, the DataDog exporter might not provide meaningful contextual information in the exported spans. To fix this, you can configure the Apollo Router to perform a mapping for the span name and the span resource name.

router.yaml

1
telemetry:
2
  tracing:
3
    datadog:
4
      enabled: true
5
      enable_span_mapping: true

when enable_span_mapping is set to true, the Apollo Router will perform the following mapping:

Use the Open Telemetry span name to set the Data Dog span operation name.
Use the Open Telemetry span attributes to set the DataDog span resource name.

Example:

Lets say we send a query MyQuery to the Apollo Router, then the Router using the operation's query plan will send a query to my-subgraph-name and the following trace will be created:

1
| apollo_router request                                                                 |
2
        | apollo_router router                                                              |
3
            | apollo_router supergraph                                                      |
4
            | apollo_router query_planning  | apollo_router execution                       |
5
                                                | apollo_router fetch                       |
6
                                                    | apollo_router subgraph                |
7
                                                        | apollo_router subgraph_request    |

As you can see, there is no clear information about the name of the query, the name of the subgraph, and the name of the query sent to the subgraph.

Instead when enable_span_mapping is set to true the following trace will be created:

1
| request /graphql                                                                                   |
2
        | router                                                                                         |
3
            | supergraph MyQuery                                                                         |
4
                | query_planning MyQuery  | execution                                                    |
5
                                              | fetch fetch                                              |
6
                                                  | subgraph my-subgraph-name                            |
7
                                                      | subgraph_request MyQuery__my-subgraph-name__0    |

Using Jaeger (native)

:warning: Jaeger native is deprecated and will be removed in a future release of the Apollo Router. Instead, OTLP should be used.

The Apollo Router can be configured to export tracing data to Jaeger either via an agent or http collector.

Unless explicitly configured to use a collector, Jaeger will use an agent by default.

Agent config

router.yaml

1
telemetry:
2
  tracing:
3
    jaeger:
4
      enabled: true
5
      # Optional agent configuration,
6
      agent:
7
        # Optional endpoint, either 'default' or a socket address (Defaults to 127.0.0.1:6832)
8
        endpoint: "${env.JAEGER_HOST}:6832"

Collector config

router.yaml

1
telemetry:
2
  tracing:
3
    jaeger:
4
      enabled: true
5
      # Optional collector configuration,
6
      collector:
7
        # Optional endpoint, either 'default' or a URL (Defaults to http://127.0.0.1:14268/api/traces)
8
        endpoint: "http://${env.JAEGER_HOST}:14268/api/traces"
9
        username: "${env.JAEGER_USERNAME}"
10
        password: "${env.JAEGER_PASSWORD}"

OTLP

OTLP (OpenTelemetry protocol) is the native protocol for open telemetry. It can be used to export traces to a variety of backends including:

OpenTelemetry Collector
Datadog
Honeycomb
Lightstep

router.yaml

1
telemetry:
2
  tracing:
3
    otlp:
4
      enabled: true
5
      
6
      # Optional endpoint, either 'default' or a URL (Defaults to http://127.0.0.1:4317 for gRPC and http://127.0.0.1:4318 for HTTP)
7
      endpoint: default
8

9
      # Optional protocol (Defaults to grpc)
10
      protocol: grpc
11

12
      # Optional Grpc configuration
13
      grpc:
14
        domain_name: "my.domain"
15
        key: ""
16
        ca: ""
17
        cert: ""
18
        metadata:
19
          foo: bar
20

21
      # Optional Http configuration
22
      http:
23
        headers:
24
          foo: bar

Remember that file. and env. prefixes can be used for expansion in config yaml. e.g. ${file.ca.txt}.

Datadog

To use Datadog, you must configure the Datadog agent to accept OTLP traces. This can be done by adding the following to your datadog.yaml:

datadog.yaml

1
otlp_config:
2
  receiver:
3
    protocols:
4
      grpc:
5
        endpoint: <dd-agent-ip>:4317

The router must also be configured to send traces to the Datadog agent:

router.yaml

1
telemetry:
2
  tracing:
3
    otlp:
4
      enabled: true
5

6
      # Optional endpoint, either 'default' or a URL (Defaults to http://127.0.0.1:4317)
7
      endpoint: "${env.DATADOG_AGENT_HOST}:4317"

See Datadog Agent configuration for more details

Jaeger

Since 1.35.0, Jaeger supports native OTLP ingestion.

Ensure that when running Jaeger in docker port 4317 is exposed and that COLLECTOR_OTLP_ENABLED is set to true:

1
docker run --name jaeger \
2
  -e COLLECTOR_OTLP_ENABLED=true \
3
  -p 16686:16686 \
4
  -p 4317:4317 \
5
  -p 4318:4318 \
6
  jaegertracing/all-in-one:1.35

Then configure the router to send traces via OTLP:

router.yaml

1
telemetry:
2
  tracing:
3
    otlp:
4
      enabled: true
5

6
      # Optional endpoint, either 'default' or a URL (Defaults to http://127.0.0.1:4317)
7
      endpoint: "http://${env.JAEGER_HOST}:4317"

Using Zipkin

The Apollo Router can be configured to export tracing data to either the default collector address or a URL:

router.yaml

1
telemetry:
2
  tracing:
3
    zipkin:
4
      enabled: true
5

6
      # Optional endpoint, either 'default' or a URL (Defaults to http://127.0.0.1:9411/api/v2/spans)
7
      endpoint: "http://${env.ZIPKIN_HOST}:9411/api/v2/spans}"

GraphOS reporting

Configuring metrics collectors