Collecting metrics in the Apollo Router

The Apollo Router provides built-in support for metrics collection via Prometheus and OpenTelemetry Collector.

Using Prometheus

You can use Prometheus and Grafana to collect metrics and visualize the router metrics.

router.yaml

1
telemetry:
2
  metrics:
3
    common:
4
      # (Optional, default to "apollo-router") Set the service name to easily find metrics related to the apollo-router in your metrics dashboards
5
      service_name: "apollo-router"
6
      # (Optional)
7
      service_namespace: "apollo"
8
    prometheus:
9
      # By setting this endpoint you enable the prometheus exporter
10
      # All our endpoints exposed by plugins are namespaced by the name of the plugin
11
      enabled: true
12
      listen: 127.0.0.1:9090
13
      path: /metrics

Using in a containers environment

The prometheus endpoint listens to 127.0.0.1 by default, which won't allow connections issued from a network. While this is a safe default, other containers won't be able to access the prometheus endpoint, which will disable metric scraping.

You can change this by setting:

router.yaml

1
telemetry:
2
  metrics:
3
    prometheus:
4
      # By setting this endpoint you enable other containers and pods to access the prometheus endpoint
5
      enabled: true
6
      listen: 0.0.0.0:9090
7
      path: /metrics

Assuming you're running locally:

Run a query against the router.
Navigate to http://localhost:9090/metrics to see something like:

1
# HELP apollo_router_http_request_duration_seconds Total number of HTTP requests made.
2
# TYPE apollo_router_http_request_duration_seconds histogram
3
apollo_router_http_request_duration_seconds_bucket{le="0.5"} 1
4
apollo_router_http_request_duration_seconds_bucket{le="0.9"} 1
5
---SNIP---

Note that if you haven't run a query against the router yet, you'll see a blank page because no metrics have been generated!

Currently available metrics

The following metrics are available using Prometheus:

HTTP router request duration: apollo_router_http_request_duration_seconds_bucket
HTTP request duration by subgraph: apollo_router_http_request_duration_seconds_bucket with attribute subgraph
Total number of HTTP requests by HTTP Status: apollo_router_http_requests_total
Number of currently connected clients: apollo_router_session_count_total
Number of in flight GraphQL requests: apollo_router_session_count_active
Number of cache hits for different kind of cache (apq, query planner, introspection) and for different storage (memory, redis): apollo_router_cache_hit_count
Number of cache misses for different kind of cache (apq, query planner, introspection) and for different storage (memory, redis): apollo_router_cache_miss_count
Time to hit the cache for different kind of cache (apq, query planner, introspection) and for different storage (memory, redis), in seconds: apollo_router_cache_hit_time
Time to miss the cache for different kind of cache (apq, query planner, introspection) and for different storage (memory, redis), in seconds: apollo_router_cache_miss_time
Time spent processing a request, outside of waiting for external or subgraph requests, in seconds (apollo_router_processing_time)
Number of triggered timeouts: apollo_router_timeout

Using OpenTelemetry Collector

You can send metrics to OpenTelemetry Collector for processing and reporting metrics.

router.yaml

1
telemetry:
2
  metrics:
3
    otlp:
4
      # Either 'default' or a URL
5
      endpoint: default
6

7
      # Optional protocol. Only grpc is supported currently.
8
      # Setting to http will result in configuration failure.
9
      protocol: grpc
10

11
      # Optional Grpc configuration
12
      grpc:
13
        domain_name: "my.domain"
14
        key: ""
15
        ca: ""
16
        cert: ""
17
        metadata:
18
          foo: bar
19

20
      # Optional batch_processor configuration
21
      batch_processor:
22
        scheduled_delay: 100ms
23
        max_concurrent_exports: 1000
24
        max_export_batch_size: 10000
25
        max_export_timeout: 100s
26
        max_queue_size: 10000

Remember that file. and env. prefixes can be used for expansion in config yaml. e.g. ${file.ca.txt}.

Adding custom attributes/labels

You can add custom attributes (OpenTelemetry) and labels (Prometheus) to your generated metrics. You can apply these across all requests, or you can selectively apply them based on the details of a particular request. These details include:

The presence of a particular HTTP header
The value at a particular JSON path within a request or response body (either from a subgraph or from the router itself)
- See examples of querying a JSON path.
A custom value provided via the router plugin context

Examples of all of these are shown in the file below:

router.yaml

1
telemetry:
2
  metrics:
3
    common:
4
      attributes:
5
        supergraph: # Attribute configuration for requests to/responses from the router
6
          static:
7
            - name: "version"
8
              value: "v1.0.0"
9
          request:
10
            header:
11
              - named: "content-type"
12
                rename: "payload_type"
13
                default: "application/json"
14
              - named: "x-custom-header-to-add"
15
          response:
16
            body:
17
              # Apply the value of the provided path of the router's response body as an attribute
18
              - path: .errors[0].extensions.status
19
                name: error_from_body
20
          context:
21
            # Apply the indicated element from the plugin chain's context as an attribute
22
            - named: my_key
23
        subgraph: # Attribute configuration for requests to/responses from subgraphs
24
          all:
25
            static:
26
              # Always apply this attribute to all metrics for all subgraphs
27
              - name: kind
28
                value: subgraph_request
29
            errors: # Only work if it's a valid GraphQL error (for example if the subgraph returns an http error or if the router can't reach the subgraph)
30
              include_messages: true # Will include the error message in a message attribute
31
              extensions: # Include extensions data
32
                - name: subgraph_error_extended_type # Name of the attribute
33
                  path: .type # JSON query path to fetch data from extensions
34
                - name: message
35
                  path: .reason
36
            # Will create this kind of metric for example apollo_router_http_requests_error_total{message="cannot contact the subgraph",service_name="apollo-router",subgraph="my_subgraph_name",subgraph_error_extended_type="SubrequestHttpError"}
37
          subgraphs:
38
            my_subgraph_name: # Apply these rules only for the subgraph named `my_subgraph_name`
39
              request:
40
                header:
41
                  - named: "x-custom-header"
42
                body:
43
                  # Apply the value of the provided path of the router's request body as an attribute (here it's the query)
44
                  - path: .query
45
                    name: query
46
                    default: UNKNOWN

Example JSON path queries

Let's say you have a JSON request body with the following structure:

1
{
2
  "items": [
3
    {
4
      "unwanted": 7,
5
      "wanted": { "x": 3, "y": 7 },
6
      "array": [3, 2, 1]
7
    },
8
    {
9
      "isImportant": true
10
    }
11
  ]
12
}

To fetch the value of the field isImportant, the corresponding path is .items[1].isImportant.

To fetch the value of the field x, the corresponding path is .items[0].wanted.x.

JSON path queries always begin with a period .

Adding custom resources

Resources are similar to attributes, but there are more globals. They're configured directly on the metrics exporter, which means they're always present on each of your metrics.

As an example, it can be useful to set a environment_name resource to help you identify metrics related to a particular environment:

router.yaml

1
telemetry:
2
  metrics:
3
    common:
4
      resources:
5
        environment_name: "production"

See OpenTelemetry conventions for resources.

For example, if you want to use a Datadog agent and specify a service name, you should set the service.name resource as shown above and described in the conventions document.

Edit on GitHub

Apollo Studio reporting

Tracing