Instruments
Collect measurements with standard and custom instruments
An instrument in the router collects data and reports measurements to a metric backend. Supported instruments include standard instruments from OpenTelemetry, standard instruments for the router's request lifecycle, and custom instruments. Supported instrument kinds are counters and histograms.
You can configure instruments in router.yaml
with telemetry.instrumentation.instruments
.
OpenTelemetry standard instruments
OpenTelemetry specifies multiple standard metric instruments that are available in the router:
In the router service:
http.server.active_requests
- The number of active requests in flight.http.server.request.body.size
- A histogram of request body sizes for requests handled by the router.http.server.request.duration
- A histogram of request durations for requests handled by the router.
In the subgraph service:
http.client.request.body.size
- A histogram of request body sizes for requests handled by subgraphs.http.client.request.duration
- A histogram of request durations for requests handled by subgraphs.http.client.response.body.size
- A histogram of response body sizes for requests handled by subgraphs.
default_requirement_level
setting configures whether or not these instruments are enabled by default. Out of the box, its default value of required
enables them. You must explicitly configure an instrument for different behavior.These instruments are configurable in router.yaml
:
1telemetry:
2 instrumentation:
3 instruments:
4 router:
5 http.server.active_requests: true # (default false)
6 http.server.request.body.size: true # (default false)
7 http.server.request.duration: true # (default false)
8 subgraph:
9 http.client.request.body.size: true # (default false)
10 http.client.request.duration: true # (default false)
11 http.client.response.body.size: true # (default false)
They can be customized by attaching or removing attributes. See attributes to learn more about configuring attributes.
1telemetry:
2 instrumentation:
3 instruments:
4 default_requirement_level: required
5 router:
6 http.server.active_requests:
7 attributes:
8 http.request.method: true
9 subgraph:
10 http.client.request.duration:
11 attributes:
12 subgraph.name: true
Apollo standard instruments
To learn about Apollo-provided standard metric instruments for the router's request lifecycle, see router instruments.
Custom instruments
You can define custom instruments on the router, supergraph, and subgraph services in the router pipeline. You can also define custom instruments for each JSON element in the response data the router returns to clients.
The example configuration below defines four custom instruments:
acme.request.duration
on therouter
serviceacme.graphql.requests
on thesupergraph
serviceacme.graphql.subgraph.errors
on thesubgraph
serviceacme.graphql.list.lengths
on each JSON element returned to the client (defined ongraphql
)
1telemetry:
2 instrumentation:
3 instruments:
4 router:
5 http.server.active_requests: true
6 acme.request.duration:
7 value: duration
8 type: counter
9 unit: kb
10 description: "my description"
11 condition:
12 eq:
13 - 200
14 - response_status: code
15 attributes:
16 http.response.status_code: true
17 "my_attribute":
18 response_header: "x-my-header"
19
20 supergraph:
21 acme.graphql.requests:
22 value: unit
23 type: counter
24 unit: count
25 description: "supergraph requests"
26
27 subgraph:
28 acme.graphql.subgraph.errors:
29 value: unit
30 type: counter
31 unit: count
32 description: "my description"
33
34 graphql:
35 acme.graphql.list.lengths:
36 value:
37 list_length: value
38 type: histogram
39 unit: count
40 description: "my description"
- Custom metrics, events, and attributes consume more processing resources than standard metrics. Adding too many (standard or custom) can slow your router down.
- Configurations such as
events.*.request|error|response
that produce output for all router lifecycle services should only be used for development or debugging, not for production.
info
. Set the values of RUST_LOG
or APOLLO_ROUTER_LOG
environment variables and the --log
CLI option to info
. Using less verbose logging, such as error
, can cause some attributes to be dropped.Instrument naming conventions
When defining a custom instrument, make sure to reference OpenTelemetry (OTel) semantic conventions. The OTel semantic conventions help guide you to:
Choose a good name for your instrument.
See which standard attributes can be attached to your instrument.
Some particular guidelines to note:
Don't include the unit name in the metric name. For example,
size_kb
should besize
and the unit should bekb
.Don't include
_total
as a suffix. For example, usehttp.server.active_requests
, nothttp.server.active_requests_total
.Use dot notation to separate namespaces in the metric name. For example, use
http.server.active_requests
, nothttp_server_active_requests
.
Instrument configuration
default_requirement_level
The default_requirement_level
option sets the default attributes to attach to default standard instruments, as defined by OpenTelemetry semantic conventions.
Valid values:
required
(default) - required attributes will be attached to standard instruments by default.recommended
- recommended attributes will be attached to standard instruments by default.
1telemetry:
2 instrumentation:
3 instruments:
4 # Set the default requirement level
5 default_requirement_level: required
Attributes can be configured individually, so that required
attributes can be overridden or disabled. For example, http.response.status_code
is set individually to override the standard value:
1telemetry:
2 instrumentation:
3 instruments:
4 # Set the default requirement level
5 default_requirement_level: required
6 router:
7 # Standard metrics
8 http.server.request.body.size:
9 attributes:
10 # Standard attributes
11 http.response.status_code: false
12 # Custom attribute
13 "acme.my_attribute":
14 response_header: "x-my-header"
15 # Standard metrics
16 http.server.active_requests:
17 attributes:
18 # Standard attributes, different than other ones provides in standard metrics, custom attributes are not available on this standard metric
19 http.request.method: false
20 server.address: true
21 server.port: true
22 url.scheme: true
opt-in
must be configured individually.Router request lifecycle services
A router's request lifecycle has three major services:
Router service - Handles an incoming request before it is parsed. Works within a context of opaque bytes.
Supergraph service - Handles a request after it has been parsed and before it is sent to the subgraph. Works within a GraphQL context.
Subgraph service - Handles a request after it has been sent to the subgraph. Works within a GraphQL context.
Additionally, you can define instruments on graphql
for each JSON element returned to the client.
To define a custom instrument, add a new key to router.yaml
as telemetry.instruments.<service>.<custom-instrument>
. For example, add a custom instrument acme.request.duration
:
1telemetry:
2 instrumentation:
3 instruments:
4 router:
5 acme.request.duration: # The name of your custom instrument/metric
6 value: duration
7 type: counter
8 unit: s
9 description: "my description"
value
The service you define an instrument on determines its possible values.
Value | Definition | Available services |
---|---|---|
| The duration of the pipeline service. | router , supergraph , subgraph |
| The number of times the pipeline service has been executed. | router , supergraph , subgraph , graphql |
| A custom value extracted from the pipeline service. See selectors for more information. | router , supergraph , subgraph , graphql |
| The duration of an event in the pipeline service. | supergraph |
| The number of times an event in the pipeline service has been executed. | supergraph |
| A custom value extracted from the event in the pipeline service. See selectors for more information. | supergraph |
event_*
are mandantory when you want to use a selector on the supergraph response body (response_data
and response_errors
).Values of custom metrics can be extracted from the pipeline using custom attributes. For example, to sum the contents of a request header, create a counter with value set as the request header:
1telemetry:
2 instrumentation:
3 instruments:
4 router:
5 acme.metric:
6 # ...
7 type: counter
8 value:
9 request_header: "x-my-header"
type
Instruments come in two different types:
counter
- A monotonic counter. For example, requests served, tasks completed, or errors occurred.histogram
- A histogram of values. For example, request durations or response body sizes.
1telemetry:
2 instrumentation:
3 instruments:
4 router:
5 acme.metric:
6 # ...
7 type: counter # counter, histogram
unit
A free format unit that is displayed in your APM.
A unit
is recommended to use SI units and definitions from The Unified Code for Units of Measure.
1telemetry:
2 instrumentation:
3 instruments:
4 router:
5 acme.metric:
6 # ...
7 unit: s # seconds
description
A free format description of the instrument that will be displayed in your APM.
1telemetry:
2 instrumentation:
3 instruments:
4 router:
5 acme.metric:
6 # ...
7 description: "my description"
condition
You may only want to mutate an instrument under certain conditions. For example, you may only want to increment a counter if the response status code is 200.
To do this use a condition:
1telemetry:
2 instrumentation:
3 instruments:
4 router:
5 acme.metric:
6 # ...
7 condition:
8 eq:
9 - 200
10 - response_status: code
attributes
Instruments may have attributes attached to them from the router pipeline. These attributes are used to filter and group metrics in your APM.
Attributes may be drawn from standard attributes or selectors except for the standard metric http.server.active_requests
.
The attributes available depend on the service of the pipeline.
1telemetry:
2 instrumentation:
3 instruments:
4 router:
5 # Standard metrics
6 http.server.request.body.size:
7 attributes:
8 # Standard attributes
9 http.response.status_code: false
10 # Custom attribute
11 "acme.my_attribute":
12 response_header: "x-my-header"
13 # Standard metrics
14 http.server.active_requests:
15 attributes:
16 # Standard attributes, different than other ones provides in standard metrics, custom attributes are not available on this standard metric
17 http.request.method: false
18 server.address: true
19 server.port: true
20 url.scheme: true
21 # Custom metric
22 acme.metric:
23 value: duration
24 type: counter
25 unit: s
26 description: "my description"
27 attributes:
28 http.response.status_code: true
29 "my_attribute":
30 # ...
31 response_header: "x-my-header"
32 subgraph:
33 requests.timeout:
34 value: unit
35 type: counter
36 unit: request
37 description: "subgraph requests containing subgraph timeout"
38 attributes:
39 subgraph.name: true
40 condition:
41 eq:
42 - "request timed out"
43 - error: reason
44
45 graphql:
46 acme.graphql.list.lengths:
47 value:
48 list_length: value
49 type: histogram
50 unit: count
51 description: "my description"
52 attributes:
53 graphql.type.name: true
Instrument configuration reference
Option | Values | Default | Description |
---|---|---|---|
<attribute-name> | The name of the custom attribute. | ||
<instrument-name> | The name of the custom instrument. | ||
attributes | standard attributes or selectors | The attributes of the custom instrument. | |
condition | conditions | The condition for mutating the instrument. | |
default_requirement_level | required |recommended | required | The default attribute requirement level. |
type | counter |histogram | The name of the custom instrument. | |
unit | A unit name, for example By or {request} . | ||
description | The description of the custom instrument. | ||
value | unit |duration |<custom> |event_unit |event_duration |event_custom | The value of the instrument. |