OpenTelemetry Integration

AI agents create unpredictable usage patterns and complex request flows that are hard to monitor with traditional methods. The Apollo MCP Server's OpenTelemetry integration provides the visibility you need to run a reliable service for AI agents.

What you can monitor

• Agent behavior: Which tools and operations are used most frequently

• Performance: Response times and bottlenecks across tool executions and GraphQL operations

• Reliability: Error rates, failed operations, and request success patterns

• Distributed request flows: Complete traces from agent request through your Apollo Router and subgraphs, with automatic trace context propagation

How it works

Apollo MCP Server exports metrics and traces using the OpenTelemetry Protocol (OTLP). Application logs use the configured logging output instead of an OTLP logs exporter.

MCP requests can carry W3C traceparent, tracestate, and baggage strings in params._meta over stdio or Streamable HTTP. A valid _meta.traceparent becomes the direct parent of the MCP handler span, and the handler span links to its HTTP request span, so you can still find the transport a request arrived on. Otherwise, the handler uses its HTTP request span as parent, or starts a new trace over stdio. The HTTP request span continues to use HTTP tracing headers independently. Malformed tracing metadata does not fail the MCP request.

When _meta.baggage is a string, its parsed members replace HTTP baggage; an empty string clears baggage. Otherwise, HTTP baggage is retained. HTTP-safe baggage and the active span's W3C trace context propagate to downstream GraphQL requests, including endpoints selected by execution hooks. Baggage members with metadata that cannot form an HTTP header are omitted. These optional tracing keys work with the server's currently supported protocol versions and require no additional configuration.

Request context stays active while a handler runs even if logging filters disable its tracing span. Enabled downstream spans and Rhai trace_id values continue to use the selected request trace. If the HTTP span is also filtered, HTTP tracing headers still provide the fallback context.

Explicit HTTP headers supplied through headers, forward_headers, operation headers, or execution hooks keep their existing behavior. Automatic context injection regenerates traceparent and tracestate from the active span, but replaces baggage only when the context has baggage to inject. For example, _meta.baggage: "" clears the request's baggage context but does not remove a baggage header explicitly forwarded through forward_headers. To rely solely on automatic context propagation, avoid supplying these tracing headers explicitly.

Usage guide

Quick start: Local development

The fastest way to see Apollo MCP Server telemetry in action is with a local setup that requires only Docker.

5-minute setup

1. Start local observability stack:

   docker run -p 3000:3000 -p 4317:4317 -p 4318:4318 --rm -ti grafana/otel-lgtm

2. Add telemetry config to your config.yaml:

   YAML

   copy

   1telemetry:
   2  exporters:
   3    metrics:
   4      otlp:
   5        endpoint: "http://localhost:4318/v1/metrics"
   6        protocol: "http/protobuf"
   7    tracing:
   8      otlp:
   9        endpoint: "http://localhost:4318/v1/traces"
   10        protocol: "http/protobuf"

3. Restart your MCP server with the updated config

4. Open Grafana at http://localhost:3000 and explore your telemetry data. Default credentials are username admin with password admin.

Production deployment

For production environments, configure your MCP server to send metrics and traces to any OTLP-compatible backend. The Apollo MCP Server uses standard OpenTelemetry protocols, which enables compatibility with all major observability platforms.

Configuration example

1telemetry:
2  service_name: "mcp-server-prod" # Custom service name
3  exporters:
4    metrics:
5      otlp:
6        endpoint: "https://your-metrics-endpoint"
7        protocol: "http/protobuf" # or "grpc"
8    tracing:
9      otlp:
10        endpoint: "https://your-traces-endpoint"
11        protocol: "http/protobuf"

Observability platform integration

The MCP server works with any OTLP-compatible backend. Consult your provider's documentation for specific endpoint URLs and authentication:

• Datadog OTLP Integration - Native OTLP support

• New Relic OpenTelemetry - Direct OTLP ingestion

• AWS Observability - Via AWS Distro for OpenTelemetry

• Grafana Cloud - Hosted Grafana with OTLP

• Honeycomb - OpenTelemetry-native platform

• Jaeger - Self-hosted tracing

• OpenTelemetry Collector - Self-hosted with flexible routing

Pre-built dashboard templates

Apollo provides ready-to-use dashboard templates for monitoring your Apollo MCP Server. These templates are available in the apollographql/apm-templates repository and include visualizations for tool calls, HTTP server metrics, and request lifecycle events.

Import these templates into your observability platform to get started quickly with pre-configured graphs and alerts.

Production configuration best practices

Environment and security

1# Set via environment variable
2export ENVIRONMENT=production
3
4telemetry:
5  service_name: "apollo-mcp-server"
6  version: "1.0.0"                     # Version for correlation
7  exporters:
8    metrics:
9      otlp:
10        endpoint: "https://secure-endpoint"  # Always use HTTPS
11        protocol: "http/protobuf"           # Generally more reliable than gRPC

Resource attributes

The server reads the standard OpenTelemetry environment variables. OTEL_SERVICE_NAME sets service.name, and OTEL_RESOURCE_ATTRIBUTES sets any resource attribute:

1export OTEL_SERVICE_NAME=apollo-mcp-server
2export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production,team=platform"

For service.name, service.version, and deployment.environment.name, values resolve in this order:

1. Server config: telemetry.service_name, telemetry.version, and the ENVIRONMENT variable

2. OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES

3. Defaults: the package name for service.name and the package version for service.version. deployment.environment.name has no default.

If the server config and an OpenTelemetry variable set different values, the server logs a warning at startup.

The server doesn't set host.name. Set it with OTEL_RESOURCE_ATTRIBUTES (for example, from the Kubernetes downward API) or with a resource processor in your collector.

Performance considerations

• Protocol choice: http/protobuf is often more reliable through firewalls and load balancers than grpc

• Export interval: Metrics are exported every 30 seconds by default. Adjust this interval time using the export_interval config to balance freshness and network overhead.

• Batch export: OpenTelemetry automatically batches telemetry data for efficiency

• Network timeouts: Default timeouts are usually appropriate, but monitor for network issues

Resource correlation

• Set ENVIRONMENT to tag all telemetry with deployment.environment.name. If nothing sets it, the attribute is omitted.

• Use consistent service_name across all your Apollo infrastructure (Router, subgraphs, MCP server)

• Set version to track releases and correlate issues with deployments

• Log lines emitted inside active spans include the span's trace_id and span_id, enabling trace-to-log correlation. Metrics use resource attributes and metric labels instead of per-request trace IDs.

Log correlation

Log lines emitted inside an active span carry the span's IDs, so you can jump from a log line to its trace. How the IDs appear depends on logging.format:

• text (default): the line starts with trace_id=<32 hex> span_id=<16 hex>.

• json: each line is one JSON object, and trace_id and span_id are top-level string keys, next to span, the span's name. Most log backends detect these keys automatically.

1{"timestamp":"2026-10-27T15:04:05.123456Z","level":"INFO","target":"apollo_schema_index","span":"schema_index","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7","message":"Indexed 83 types in 16.51ms"}

Events outside a span have neither ID. This is correlation by ID only: logs still go to standard output or your log files, not to an OTLP endpoint.

Troubleshooting

Common issues

• Connection refused: Verify endpoint URL and network connectivity

• Authentication errors: Check if your provider requires API keys or special headers

• Missing data: Confirm your observability platform supports OTLP and is configured to receive data

• High memory usage: Monitor telemetry export frequency and consider sampling for high-volume environments

Verification

1# Check if telemetry is being exported (look for connection attempts)
2curl -v https://your-endpoint/v1/metrics
3
4# Monitor server logs for OpenTelemetry export errors
5./apollo-mcp-server config.yaml 2>&1 | grep -i "otel\|telemetry"

Configuration Reference

The OpenTelemetry integration is configured via the telemetry section of the configuration reference page.

Semantic conventions

The server's spans, metrics, and resource attributes follow two sets of OpenTelemetry semantic conventions:

The resource carries the schema URL https://opentelemetry.io/schemas/1.42.0. The GenAI conventions don't publish releases or a schema URL yet, so the server pins them to a commit and sets no schema URL for them.

The apollo.* metrics and attributes, and metric attributes without a namespace such as tool_name, are Apollo's own and don't follow the semantic conventions. Where a convention covers the same data, the server emits both names. For example, spans carry both apollo.mcp.tool_name and gen_ai.tool.name.

The Apollo names don't change when the conventions do, so use them for dashboards and alerts that must keep working across upgrades. After the MCP conventions become stable, Apollo might deprecate Apollo names that only duplicate a convention name. Apollo announces each deprecation in the release notes before it removes the name.

When the conventions change

The server adopts newer conventions in minor releases. If an update renames or removes a span, metric, or attribute, the release notes list the change under breaking changes. The server doesn't emit a convention's old and new names side by side, and it doesn't read OTEL_SEMCONV_STABILITY_OPT_IN. Before you upgrade, check the release notes and update the dashboards, alerts, and queries that use a renamed span, metric, or attribute.

Emitted Metrics

The server emits the following metrics, which are invaluable for monitoring and alerting. The apollo.mcp.* duration metrics are in milliseconds.

The two auth.rate_limit metrics are emitted only when you configure OAuth authentication and transport.auth.rate_limit.enabled is true (the default). They carry no attributes, so no credential or client address is recorded. See Limit authentication rates.

The server also emits mcp.server.operation.duration, the request duration metric from the OpenTelemetry semantic conventions for MCP. It's a histogram in seconds, with the bucket boundaries the conventions advise, and it records one measurement for every MCP request the server handles, except ping, server/discover, and completion/complete, which the MCP SDK answers on the server's behalf. It carries these attributes:

A client can send any tool or prompt name, so the server leaves out gen_ai.tool.name and gen_ai.prompt.name when the request fails with a caller error, such as an unknown tool. Otherwise each made-up name would add a time series. The subscriptions/listen measurement covers the whole subscription, so it reports how long a client listened rather than request latency. Omitting tool_name with telemetry.exporters.metrics.omitted_attributes also drops gen_ai.tool.name from this metric, the same way it drops the gen_ai.* copies from spans.

In addition to these metrics, the server emits the standard HTTP server metrics for every request to the MCP endpoint, courtesy of the axum-otel-metrics library.

Stability is the status each metric carries in the OpenTelemetry semantic conventions. An experimental metric can be renamed by a future revision of the conventions, so pin a dashboard or an alert to one only if you're willing to update it.

Emitted Traces

Spans are generated for the following actions:

• Incoming HTTP Requests: The server creates a SERVER span for every HTTP request to the MCP endpoint. That span roots the trace unless the request carries a traceparent header. The server doesn't trace requests to the health check endpoint, so probes don't appear as service entry points.

• MCP Handler Methods: Each implemented request handler creates a span, including tools, resources, prompts, initialization, logging, subscriptions, and Tasks (tasks/get, tasks/cancel, and tasks/update). Every handler span records the request's MCP method, such as tools/list or resources/read, as mcp.method.name, and the JSON-RPC request ID as jsonrpc.request.id. It also records the MCP protocol version as mcp.protocol.version, which for initialize is the version the server negotiated, and the transport as network.transport: tcp for Streamable HTTP, with network.protocol.name and network.protocol.version, or pipe for stdio. A request on a session records the session ID as mcp.session.id. Over Streamable HTTP, with client_address enabled, the span also records client.address and client.port. When a handler returns a JSON-RPC error, its span records the error code as rpc.response.status_code. When the error is a server error, the span also records the code as error.type and sets the span status to error, with the error message as the status description. Errors the caller causes, such as an unknown method or invalid parameters, leave the span status unset. The resources/read span records the resource URI as mcp.resource.uri, and the prompts/get span records the prompt name as gen_ai.prompt.name. With gen_ai_content enabled, the prompts/get span also records each argument the prompt declares as gen_ai.prompt.variable.<name>, such as gen_ai.prompt.variable.email. A string value is recorded as is, and any other value as JSON. The server skips arguments the prompt doesn't declare, so a client can't add attribute names. Task handler spans never record the task ID, because a task ID can act as a bearer handle. Valid _meta trace context parents this span directly, with a span link to the HTTP request span; otherwise it nests beneath the HTTP request span or starts a new trace over stdio.

• Tool Execution: call_tool spans contain nested spans for the specific tool being executed (e.g., introspect, search, or a custom GraphQL operation).

• Downstream GraphQL Calls: The execute tool in Apollo MCP Server and custom operation tools create child spans for their outgoing reqwest HTTP calls, capturing the duration of the downstream request. The traceparent, tracestate, and baggage headers propagate automatically. Baggage is caller-controlled, request-scoped context, not identity. Don't put secrets or personally identifiable information in baggage, and never use it for authorization. If your GraphQL endpoint crosses a trust boundary, filter baggage before forwarding it. Apollo MCP Server doesn't automatically copy baggage onto exported span attributes.

Span Attributes

The inbound request span is named {method} {route}, for example POST /mcp. A request to a subpath of the MCP endpoint, such as /mcp/anything, matches no route, so the server names that span after its method alone. The span follows the OpenTelemetry semantic conventions for HTTP server spans:

Attributes whose source header is absent are omitted rather than recorded as empty. A request whose method falls outside the conventions' set is named HTTP {route}, so an arbitrary method can't multiply the span names a backend has to index.

The server sets the span status to error for a 5xx response and leaves it unset otherwise, which is what the conventions require. A POST span stays open until the response body finishes streaming, so its duration covers tool execution on a streamed response. A GET span instead ends at the response head, because that request is the session's standing server-to-client stream and timing it to the body would report session lifetime as request latency. Spans the server emits over that stream later aren't nested under it.

The call_tool span includes the following attributes:

When the server records error.type, it also sets the span status to error. For a JSON-RPC error, the status description is the error message. The server always exports the status description, regardless of gen_ai_content or omitted_attributes, so make sure errors raised by your Rhai hooks or downstream services don't put sensitive data in their messages.

The mcp.*, gen_ai.*, and error.type attributes follow the OpenTelemetry semantic conventions for MCP. Observability platforms that read these conventions, such as Langfuse, use gen_ai.tool.call.arguments and gen_ai.tool.call.result to show a tool call's input and output.

The server always exports gen_ai.operation.name and gen_ai.tool.name. It doesn't export gen_ai.tool.call.arguments, gen_ai.tool.call.result, or the gen_ai.prompt.variable.<name> attributes on prompts/get spans by default, because they may contain sensitive data. To export them, enable gen_ai_content:

1telemetry:
2  exporters:
3    tracing:
4      otlp:
5        endpoint: https://your-traces-endpoint/v1/traces
6        protocol: http/protobuf
7      gen_ai_content: true

For tool calls, gen_ai_content controls only the gen_ai.* copies. The same arguments and result are still exported as apollo.mcp.tool_arguments and apollo.mcp.tool_result unless you omit them with omitted_attributes.

For tools that execute a downstream GraphQL operation, the child execute span includes:

These attributes are populated only after the corresponding value is available and can be serialized. Early validation errors, network failures, and response-decoding failures can leave one or more of your attributes unset.

Sensitive data in traces

When you configure tracing export, the server exports populated span attributes as provided unless you omit them. Tool arguments, tool results, prompt arguments, GraphQL queries, and GraphQL responses can contain identifiers, user-provided values, or other data that requires restricted handling.

Prompt arguments are exported only as gen_ai.prompt.variable.<name>, and only when gen_ai_content is enabled. They have no apollo.mcp.* counterpart, so omitted_attributes can't remove them. To keep them out of traces, leave gen_ai_content disabled.

For operations that use the MCP Apps @private directive, Apollo MCP Server records the LLM-visible GraphQL response and excludes the response metadata containing the full @private result from the tool_result span attribute. This protects those marked response fields but doesn't redact tool arguments, query text, or other response fields.

apollo.mcp.session_id is a session identifier rather than payload data, and the server records it only for a request it accepted, so a rejected caller can't write a value into it. A request using protocol version 2026-07-28 or later has no session, so the server doesn't record the header for it, even in stateful mode. With stateful_mode: false there are no sessions to look up, so the server records the header as the client sent it; omit the attribute if an unvalidated caller-supplied value matters to you. It's listed with the other apollo.mcp.* attributes and can be omitted the same way. Handler spans carry the same value as mcp.session.id, and omitting session_id omits both.

Exclude the four payload attributes from exported traces by configuring the tracing exporter:

1telemetry:
2  exporters:
3    tracing:
4      otlp:
5        endpoint: https://your-traces-endpoint/v1/traces
6        protocol: http/protobuf
7      omitted_attributes:
8        - tool_arguments
9        - tool_result
10        - graphql_query
11        - graphql_response

Omitting tool_name, tool_arguments, or tool_result also removes the matching gen_ai.* attribute, even when gen_ai_content is enabled, so the payload isn't exported under either name.

Omitting payload attributes removes payload-level detail while retaining useful trace structure, timing, status, and any attributes you don't omit. Select omissions according to your investigation needs and data-handling requirements.

Client addresses

An IP address is personal data in many jurisdictions, so the server doesn't record client addresses by default. To record them, enable client_address:

1telemetry:
2  exporters:
3    tracing:
4      client_address: true

The HTTP request span then records client.address, network.peer.address, and network.peer.port, and MCP handler spans record client.address and client.port. Requests over stdio have no client address. These attributes go only to exported traces: they don't appear in the span context of log lines. Metrics never carry them either, because every client address would add a time series.

The server takes the address from the TCP connection. It doesn't read Forwarded or X-Forwarded-For, because a caller can set them to any value. Behind a reverse proxy or load balancer, the recorded address is therefore the proxy's, not the original client's.

Sampling and attribute filtering

High-cardinality metrics can occur in MCP Servers with large number of tools or when clients are allowed to generate freeform operations. To prevent performance issues and reduce costs, the Apollo MCP Server provides two mechanisms to control metric cardinality, trace sampling and attribute filtering.

Trace Sampling

Configure Apollo MCP Server to sample traces sent to your OpenTelemetry Collector using the sampler field in the telemetry.exporters.tracing configuration:

• always_on - Send every trace

• always_off - Don't record spans that start a new trace. Spans with a sampled incoming trace context are still recorded unless parent_based_sampler is false

• 0.0-1.0 - Send a specified percentage of traces

By default, sampler applies only to spans that start a new trace. A span with an incoming trace context follows the caller's sampling decision instead, so a trace is either recorded across every service or not at all. A traceparent header is the parent of the HTTP request span, and MCP _meta.traceparent is the parent of the MCP handler span, such as tools/call. When only _meta carries a trace context, the HTTP request span starts its own trace and uses sampler.

This also lets you force a trace while troubleshooting, even with a low sampling ratio. Send a traceparent whose last field, the trace flags, is 01:

1curl http://localhost:8000/mcp \
2  -H 'traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01' \
3  ...

A trace flags value of 00 tells the server not to record the spans whose parent is that context.

Because callers make the sampling decision for their own requests, sampler doesn't limit trace volume when callers aren't trusted. A client that sends trace flags 01 gets its requests recorded and exported, even with always_off. For a public-facing server, or to apply sampler to every request, set parent_based_sampler: false:

1telemetry:
2  exporters:
3    tracing:
4      sampler: 0.1
5      parent_based_sampler: false

Attribute Filtering

Trace and metric exporters have independent omitted_attributes lists:

• telemetry.exporters.tracing.omitted_attributes removes the selected apollo.* attributes, and their gen_ai.* or jsonrpc.* counterparts, from spans exported by this server

• telemetry.exporters.metrics.omitted_attributes removes the selected supported labels from Apollo metrics

Use short attribute names such as tool_name, operation_id, tool_arguments, and graphql_response. For detailed configuration options, go to the telemetry configuration reference.

Audit and deployment responsibilities

Apollo MCP Server telemetry provides request traces, operational metrics, and correlated application logs that can support investigations. A complete audit trail also depends on the surrounding observability system. Configure the collector and backend for centralized storage, access controls, retention, integrity protection, alerting, and periodic review appropriate to your environment.

Visit OWASP MCP08: Lack of Audit and Telemetry for additional audit and monitoring considerations.