Collecting metrics in the Apollo Router

The Apollo Router provides built-in support for metrics collection via Prometheus and OpenTelemetry Collector.

Using Prometheus

You can use Prometheus and Grafana to collect metrics and visualize the router metrics.

router.yaml

1
telemetry:
2
  metrics:
3
    common:
4
      # (Optional, default to "apollo-router") Set the service name to easily find metrics related to the apollo-router in your metrics dashboards
5
      service_name: "apollo-router"
6
      # (Optional)
7
      service_namespace: "apollo"
8
    prometheus:
9
      # By setting this endpoint you enable the prometheus exporter
10
      # All our endpoints exposed by plugins are namespaced by the name of the plugin
11
      enabled: true
12
      listen: 127.0.0.1:9090
13
      path: /metrics

Using in a containers environment

The prometheus endpoint listens to 127.0.0.1 by default, which won't allow connections issued from a network. While this is a safe default, other containers won't be able to access the prometheus endpoint, which will disable metric scraping.

You can change this by setting:

router.yaml

1
telemetry:
2
  metrics:
3
    prometheus:
4
      # By setting this endpoint you enable other containers and pods to access the prometheus endpoint
5
      enabled: true
6
      listen: 0.0.0.0:9090
7
      path: /metrics

Assuming you're running locally:

Run a query against the router.
Navigate to http://localhost:9090/metrics to see something like:

1
# HELP apollo_router_http_request_duration_seconds Total number of HTTP requests made.
2
# TYPE apollo_router_http_request_duration_seconds histogram
3
apollo_router_http_request_duration_seconds_bucket{le="0.5"} 1
4
apollo_router_http_request_duration_seconds_bucket{le="0.9"} 1
5
---SNIP---

Note that if you haven't run a query against the router yet, you'll see a blank page because no metrics have been generated!

Available metrics

The following metrics are available when using Prometheus. Attributes are listed where applicable.

HTTP

apollo_router_http_request_duration_seconds_bucket - HTTP router request duration
apollo_router_http_request_duration_seconds_bucket - HTTP subgraph request duration, attributes:
- subgraph: (Optional) The subgraph being queried
apollo_router_http_requests_total - Total number of HTTP requests by HTTP status
apollo_router_timeout - Number of triggered timeouts
apollo_router_http_request_retry_total - Number of subgraph requests retried, attributes:
- subgraph: The subgraph being queried
- status : If the retry was aborted (aborted)

Session

apollo_router_session_count_total - Number of currently connected clients
apollo_router_session_count_active - Number of in-flight GraphQL requests

Cache

apollo_router_cache_size — Number of entries in the cache
apollo_router_cache_hit_count - Number of cache hits
apollo_router_cache_miss_count - Number of cache misses
apollo_router_cache_hit_time - Time to hit the cache in seconds
apollo_router_cache_miss_time - Time to miss the cache in seconds

All cache metrics listed above have the following attributes:

kind: the cache being queried (apq, query planner, introspection)
storage: The backend storage of the cache (memory, redis)

Performance

apollo_router_processing_time - Time spent processing a request (outside of waiting for external or subgraph requests) in seconds.
apollo_router_query_planning_time - Time spent planning queries in seconds.

Uplink

apollo_router_uplink_fetch_duration_seconds_bucket - Uplink request duration, attributes:
- url: The Uplink URL that was polled
- query: The query that the router sent to Uplink (SupergraphSdl or Entitlement)
- kind: (new, unchanged, http_error, uplink_error)
- code: The error code depending on type (if an error occurred)
- error: The error message (if an error occurred)
apollo_router_uplink_fetch_count_total
- status: (success, failure)
- query: The query that the router sent to Uplink (SupergraphSdl or Entitlement)

Note that the initial call to uplink during router startup will not be reflected in metrics.

Using OpenTelemetry Collector

You can send metrics to OpenTelemetry Collector for processing and reporting metrics.

router.yaml

1
telemetry:
2
  metrics:
3
    otlp:
4
      # Either 'default' or a URL
5
      endpoint: default
6

7
      # Optional protocol. Only grpc is supported currently.
8
      # Setting to http will result in configuration failure.
9
      protocol: grpc
10

11
      # Optional Grpc configuration
12
      grpc:
13
        domain_name: "my.domain"
14
        key: ""
15
        ca: ""
16
        cert: ""
17
        metadata:
18
          foo: bar
19

20
      # Optional batch_processor configuration
21
      batch_processor:
22
        scheduled_delay: 100ms
23
        max_concurrent_exports: 1000
24
        max_export_batch_size: 10000
25
        max_export_timeout: 100s
26
        max_queue_size: 10000

Remember that file. and env. prefixes can be used for expansion in config yaml. e.g. ${file.ca.txt}.

Adding custom attributes/labels

You can add custom attributes (OpenTelemetry) and labels (Prometheus) to your generated metrics. You can apply these across all requests, or you can selectively apply them based on the details of a particular request. These details include:

The presence of a particular HTTP header
The value at a particular JSON path within a request or response body (either from a subgraph or from the router itself)
- See examples of querying a JSON path.
A custom value provided via the router plugin context

Examples of all of these are shown in the file below:

router.yaml

1
telemetry:
2
  metrics:
3
    common:
4
      attributes:
5
        supergraph: # Attribute configuration for requests to/responses from the router
6
          static:
7
            - name: "version"
8
              value: "v1.0.0"
9
          request:
10
            header:
11
              - named: "content-type"
12
                rename: "payload_type"
13
                default: "application/json"
14
              - named: "x-custom-header-to-add"
15
          response:
16
            body:
17
              # Apply the value of the provided path of the router's response body as an attribute
18
              - path: .errors[0].extensions.http.status
19
                name: error_from_body
20
              # Use the unique extension code to identify the kind of error
21
              - path: .errors[0].extensions.code
22
                name: error_code
23
          context:
24
            # Apply the indicated element from the plugin chain's context as an attribute
25
            - named: my_key
26
        subgraph: # Attribute configuration for requests to/responses from subgraphs
27
          all:
28
            static:
29
              # Always apply this attribute to all metrics for all subgraphs
30
              - name: kind
31
                value: subgraph_request
32
            errors: # Only work if it's a valid GraphQL error (for example if the subgraph returns an http error or if the router can't reach the subgraph)
33
              include_messages: true # Will include the error message in a message attribute
34
              extensions: # Include extensions data
35
                - name: subgraph_error_extended_type # Name of the attribute
36
                  path: .type # JSON query path to fetch data from extensions
37
                - name: message
38
                  path: .reason
39
            # Will create this kind of metric for example apollo_router_http_requests_error_total{message="cannot contact the subgraph",service_name="apollo-router",subgraph="my_subgraph_name",subgraph_error_extended_type="SubrequestHttpError"}
40
          subgraphs:
41
            my_subgraph_name: # Apply these rules only for the subgraph named `my_subgraph_name`
42
              request:
43
                header:
44
                  - named: "x-custom-header"
45
                body:
46
                  # Apply the value of the provided path of the router's request body as an attribute (here it's the query)
47
                  - path: .query
48
                    name: query
49
                    default: UNKNOWN

Example JSON path queries

Let's say you have a JSON request body with the following structure:

1
{
2
  "items": [
3
    {
4
      "unwanted": 7,
5
      "wanted": { "x": 3, "y": 7 },
6
      "array": [3, 2, 1]
7
    },
8
    {
9
      "isImportant": true
10
    }
11
  ]
12
}

To fetch the value of the field isImportant, the corresponding path is .items[1].isImportant.

To fetch the value of the field x, the corresponding path is .items[0].wanted.x.

JSON path queries always begin with a period .

Changing default buckets for histograms

You can customize the buckets for all generated histograms:

router.yaml

1
telemetry:
2
  metrics:
3
    common:
4
      buckets:
5
        - 0.05
6
        - 0.10
7
        - 0.25
8
        - 0.50
9
        - 1.00
10
        - 2.50
11
        - 5.00
12
        - 10.00
13
        - 20.00

Adding custom resources

Resources are similar to attributes, but there are more globals. They're configured directly on the metrics exporter, which means they're always present on each of your metrics.

As an example, it can be useful to set a environment_name resource to help you identify metrics related to a particular environment:

router.yaml

1
telemetry:
2
  metrics:
3
    common:
4
      resources:
5
        environment_name: "production"

See OpenTelemetry conventions for resources.

For example, if you want to use a Datadog agent and specify a service name, you should set the service.name resource as shown above and described in the conventions document.

OpenTelemetry tracing

Client awareness