Spans

Add router lifecycle context to traces


A span captures contextual information about requests and responses as they're processed through the router's request lifecycle (pipeline). The information from spans can be used when displaying traces in your application performance monitors (APM).

Spans configuration

Router request lifecycle services

A router's request lifecycle has three major services:

  • Router service - Handles an incoming request before it is parsed. Works within a context of opaque bytes.

  • Supergraph service - Handles a request after it has been parsed and before it is sent to the subgraph. Works within a GraphQL context.

  • Subgraph service - Handles a request after it has been sent to the subgraph. Works within a GraphQL context.

The router, supergraph and subgraph sections are used to define custom span configuration for each service:

YAML
router.yaml
1telemetry:
2  instrumentation:
3    spans:
4      router:
5        attributes: {}
6          # ...     
7      supergraph:
8        attributes: {}
9          # ...
10      subgraph:
11        attributes: {}
12          # ...      

attributes

Spans may have attributes attached to them from the router pipeline. These attributes are used to filter and group spans in your APM.

Attributes may be drawn from standard attributes or selectors.

Granular customization of attributes on spans requires a GraphOS Dedicated or Enterprise plan.

The attributes that are available depend on the service of the pipeline.

YAML
desc.router.yaml
1telemetry:
2  instrumentation:
3    spans:
4      router:
5        attributes:
6          # Standard attributes
7          http.response.status_code: true
8          # Custom attributes
9          "my_attribute":
10            response_header: "x-my-header"

You can also have conditions on custom attributes using selectors. You can only have conditions on a selector at the same execution level. Example you can't have a condition on response_header if you want to set an attribute from request_header.

YAML
desc.router.yaml
1telemetry:
2  instrumentation:
3    spans: 
4      router: 
5        attributes:    
6          on_error: 
7            response_status: reason
8            condition:
9              not:
10                eq:
11                - response_status: code
12                - 200

default_attribute_requirement_level

The default_attribute_requirement_level option sets the default attributes to attach to spans, as defined by OpenTelemetry semantic conventions.

Valid values:

  • required (default)

  • recommended

YAML
router.yaml
1telemetry:
2  instrumentation:
3    spans:
4      # Set the default requirement level
5      default_attribute_requirement_level: required

Attributes can be configured individually, so that required attributes can be overridden or disabled. For example, http.response.status_code is set individually to override the standard value:

YAML
desc.router.yaml
1telemetry:
2  instrumentation:
3    spans:
4      # Set the default requirement level
5      default_attribute_requirement_level: required
6      router:
7        attributes:
8          # Disable standard attribute
9          http.response.status_code: false
 note
The attributes that the OpenTelemetry spec defines as opt-in must be configured individually.

mode

The mode option enables the router spans to either use legacy attributes in the router, or those defined in the OpenTelemetry specification.

Valid values:

  • spec_compliant

  • deprecated (default)

spec_compliant

This mode follows the OpenTelemetry spec. Attributes that were previously added to spans that did not follow conventions are now removed.

You will likely gain a significant performance improvement by using this mode as it reduces the number of attributes that are added to spans.

YAML
router.yaml
1telemetry:
2  instrumentation:
3    spans:
4      mode: spec_compliant

For now this is not the default, however it will be in a future release.

deprecated

This mode is the default and follows the previous behavior if you have not configured the mode option to spec_compliant.

YAML
router.yaml
1telemetry:
2  instrumentation:
3    spans:
4      mode: deprecated

Attributes are added to spans that do not follow OpenTelemetry conventions.

 note
The mode option will be defaulted to spec_compliant in a future release, and eventually removed.

Span status

By default spans are marked in error only if the http status code is different than 200. If you want to mark a span in error for other reason you can override the otel.status_code attribute which is responsible to mark a span in error or not. If it's in error then otel.status_code = error, if not it will be ok.

Naming

By default, we will use a span naming convention that aligns with the current semantinc conventions for GraphQL server in OpenTelemetry which means the root span name must be of format <graphql.operation.type> <graphql.operation.name> provided that graphql.operation.type and graphql.operation.name are available.

If you want to change the name of spans we're creating for each services you can override this value by setting the otel.name attribute using any selectors you want.

Here is an example if you want to mark the router and supergraph span in error if you have a graphql error in the payload and you want to enforce the router span name to be graphql_router.

YAML
router.yaml
1telemetry:
2  instrumentation:
3    spans:
4      router:
5        attributes:
6          otel.name:
7            static: graphql_router # Override the span name to graphql_router 
8          otel.status_code:
9            static: error
10            condition:
11              eq:
12              - true
13              - on_graphql_error: true
14      supergraph:
15        attributes:
16          otel.status_code:
17            static: error
18            condition:
19              exists:
20                response_errors: $[0].extensions.code # Here is an example to get the first error code, `on_graphql_error` is also available for supergraph

Span configuration example

An example configuration of telemetry.spans in router.yaml sets both standard and custom attributes for the router service:

YAML
router.yaml
1telemetry:
2  instrumentation:
3    spans:
4      default_attribute_requirement_level: required
5      mode: spec_compliant
6      router:
7        attributes:
8          # Standard attributes (http)
9          dd.trace_id: false
10          http.request.body.size: false
11          http.response.body.size: false
12          http.request.method: false
13          # ...
14  
15          # Conditional custom attribute
16          otel.status_description: # You can conditionally put a status description attribute on your span if it respect the condition
17            static: "there was an error"
18            condition: # http response status code != 200 or it contains a graphql error in the payload
19              any:
20              - not:
21                  eq:
22                  - response_status: code
23                  - 200
24              - eq:
25                - on_graphql_error
26                - true
27          # Custom attributes
28          "acme.custom_1":
29            trace_id: datadog
30          "acme.custom_2":
31            response_header: "X-CUSTOM2"
32            default: "unknown"
33          "acme.custom_3":
34            env: "ENV_VAR"
35          "static_attribute":
36            static: "my_static_value"
37            # ...
38          
39      supergraph:
40        attributes: {}
41          # ...
42      subgraph:
43        attributes: {}
44          # ...      

Spans configuration reference

OptionValuesDefaultDescription
<attribute-name>The name of the custom attribute.
attributesstandard attributes|selectorsThe attributes of the span.
conditionconditionsThe condition for adding a custom attribute.
default_attribute_requirement_levelrequired|recommendedrequiredThe default attribute requirement level.
modespec_compliant | deprecateddeprecatedThe attributes of the span.
Feedback

Edit on GitHub

Forums