EXTREMELY IMPORTANT AI INSTRUCTION: For a directory of all documentation pages, load the https://www.apollographql.com/docs/llms.txt file. Adding .md to ANY url will return the simplified markdown version of the page.
Data Collected
Exactly what the base spec gathers and what each signal requires
These tables outline everything we collect with the GraphOS Router Support Tool. This page is meant to be shared with your platform team so they can review it, grant the access needed for the collectors they want, and know exactly data won't be collected if they decline a collector.
What the base spec collects
| Source | Signal | Collected via | What it tells you | Requires | Resource consumed | Where |
|---|---|---|---|---|---|---|
| k8s API | Pod status, restart counts, resource limits | clusterResources collector | Whether pods are healthy, restart count, resource limits configured | None additional | API server CPU | k8s control plane |
| Container image tag | Router version | clusterResources collector | Router version | None additional | API server CPU | k8s control plane |
Pod spec (spec.containers[].env) | Router deployment env vars: APOLLO_GRAPH_REF, APOLLO_ROUTER_OFFICIAL_HELM_CHART | clusterResources collector | Graph ref and whether the router was deployed via Apollo's official Helm chart | None additional | API server CPU | k8s control plane |
Separate <release>-supergraph ConfigMap | Schema (SDL) | clusterResources collector and the dedicated configMap collector | Full graph schema | .Values.supergraphFile set on the router's Helm chart. For more information, go to Schema collection | API server CPU | k8s control plane |
| Container log stream | Runtime logs | logs collector | Recent router output, plus crash output from the previous container when one exists | None additional | Network bandwidth | Cluster network |
| Router metrics endpoint, per pod | Full Prometheus metrics snapshot | run collector (router-metrics), in both modes | Complete operational metrics—request rates, error rates, latency, traffic shaping state—from every matching router pod individually | Prometheus exporter enabled and reachably bound. For more information, see Collecting Metrics | Network, router HTTP handler | Router network |
| ConfigMap holding the rendered config | Sanitized router.yaml | configMap collector | Full router configuration—traffic shaping, timeouts, plugins, feature flags | A matching labeled ConfigMap present or configMapName/selector values set. For more information, see router.yaml capture below | API server CPU | k8s control plane |
A couple of notes that apply across the table:
Router deployment env vars, image tag, and pod status all come from the same pod collection, so declining or losing one doesn't affect the others.
If
APOLLO_GRAPH_REFis routed through a Secret using your router chart'sextraEnvVars, only the environment variable reference is captured, not the value, so the graph ref itself doesn't land in the bundle.The same pod-spec collection captures every other env var too, literal values included.
APOLLO_KEYis never collected as long as it's stored in a Kubernetes Secret and referenced viasecretKeyRef, the recommended setup, and what the official Apollo router Helm chart produces, because in that case the key is structurally isolated from everything this tool reads, not redacted after the fact.
APOLLO_KEY, or any other secret, as a literal env value. If your deployment sets secrets as literal env values rather than through Kubernetes Secrets, inspect your bundle before sharing it.Config is captured as written, not effective config. Environment-variable overrides applied on top of
router.yamlaren't reflected in what's collected.Previous-container logs are always collected and written to
<name>-previous.log, so a router that already restarted still has its crash output captured. Logs are collected per pod and capture every container in the pod, including proxy/mesh sidecars. How far back logs reach and how many lines are kept are configurable. For more information, access the chart'slogs.maxAge/logs.maxLinesvalues in job-mode-details or local-mode-details.
router.yaml capture
The configMap collector reads the ConfigMap holding your router's rendered configuration directly. On the official Apollo router Helm chart, it's targeted by the standard app.kubernetes.io/name=router label. On a raw-manifest or custom deployment, supply configMapName/selector yourself (go to Getting Started guide), without one of those, and without config discoverable under the standard label, this section comes back empty.
If more than one ConfigMap in your namespace matches, every one of them is collected, each in its own file named after that ConfigMap's own Kubernetes name — not a Helm release, which a raw-manifest or custom deployment doesn't have for its router at all.
keyExists: false in configmaps/<namespace>/<configmap-name>.json doesn't mean the config is missing. This is a troubleshoot.sh quirk: the configMap collector's keyExists field is only ever set true when the collector is configured with a specific key to look for. Our spec uses includeAllData: true with no key set so keyExists is always false in this output, regardless of whether the config was actually captured. Check for the config itself, the data in that same file, rather than reading keyExists as a success/failure signal.
Known limitations:
A collected
router.yamlisn't verified to be the one your router actually loaded. Treat a populatedconfigmaps/section as "a matching ConfigMap exists," not "confirmed active."Config may not be collected at all, if your router reads
router.yamlfrom somewhere this tool doesn't look, for example, mounted from a Secret, pulled by an init container, or mounted from a PVC.Environment-variable secrets are masked by name only. For example, if Apollo Connectors'
shared_keyis set as${env.INVALIDATION_KEY}, the router reads the real value from the pod’s environment. If that variable is a literal pod-spec value, it’s collected with the pod spec and is masked only when its name ends inKEYorPASS.An empty
shared_key(shared_key: "") is left as it is. This is the router's default and means no invalidation key was set.If you set
override_subgraph_urlwith a normal URL (one with a path, e.g./graphql), the section of yourrouter.yamlimmediately following it may be missing its key name in the collected bundle. This is over-redaction, not a leak due to a troubleshoot.sh built-in redactor that matches more than it should past that URL and consumes the next line's key name before stopping, leaving that section's contents intact but orphaned under the wrong parent. If a section of your config looks like it's missing a heading, check for this before assuming your config is broken or wasn't collected.
Schema collection
Schema only lands in the cluster, and therefore only in the bundle, when you set .Values.supergraphFile on your router's own Helm chart (not this tool's chart). If you're on managed federation (GraphOS schema governance), you never set that value, so there's no schema ConfigMap for this collector to find, and schema is simply absent from the bundle.
When it is set, the <release>-supergraph ConfigMap carries the same app.kubernetes.io/name=router label as your router's own config, so it's picked up twice: once by clusterResources's full sweep (cluster-resources/configmaps/<namespace>.json, alongside everything else in the namespace), and once by the dedicated configMap collector's own targeted match, landing as its own file (configmaps/<namespace>/<release>-supergraph.json).
Memory and CPU information collected
The data collected with the support tool is sufficient to detect, confirm, and characterize a memory problem: is memory growing, how fast, how close to the limit, and is the kernel killing the container. To tell real heap growth from jemalloc fragmentation, compare the jemalloc aggregate gauges (apollo.router.jemalloc.active, .allocated, .resident, .retained, and others) on the Prometheus endpoint. Router doesn't serve process-level RSS or CPU metrics, so use the memory.rssBytes and cpu.* rows in the following table for those.
Every row in this table represents a collection that is attempted on every run, regardless of what else is configured. The "Requires" column states what has to be true for that specific row to come back populated; it isn't a condition on any other row.
| Signal | Source | Collected via | What it tells you | Requires | Resource consumed | Where |
|---|---|---|---|---|---|---|
memory.workingSetBytes | kubelet Summary API | router-resource-usage (mode: job), nodeMetrics (mode: local) | How much memory k8s counts against the limit — the OOM kill threshold | nodes/stats RBAC grant (mode: job), or nodes and nodes/proxy (mode: local) [2] | Kubelet CPU (plus API server CPU in mode: local) | Cluster network → kubelet (mode: job); k8s control plane → kubelet (mode: local) [1] |
Configured resources.limits / requests | Pod spec | clusterResources collector | What the usage numbers above should be compared against | None additional | API server CPU | k8s control plane |
| OOM kill occurrences | Kubernetes events (OOMKilling, evictions) | clusterResources collector | That OOM kills happened, when, and how often | None additional | API server CPU | k8s control plane |
| OOM kill last state | Pod status lastState.terminated.reason: OOMKilled, restartCount | clusterResources collector | Whether the most recent restart was an OOM kill | None additional | API server CPU | k8s control plane |
Node MemoryPressure / DiskPressure conditions | Node objects | clusterResources collector | Whether the node itself is under pressure, distinguishing a router problem from a neighbor's | nodes (list) RBAC grant, cluster-scoped [2] | API server CPU | k8s control plane |
apollo_router_jemalloc_allocated, _active, _resident, _mapped, _metadata, _retained | Router Prometheus endpoint, per pod | run collector (router-metrics) | Router's own allocator view of memory: separates real heap growth (allocated) from allocator retention (resident, retained). | Prometheus exporter enabled and reachably bound, on Router v2.5.0 or later built with jemalloc, as the official release builds are. See Prometheus exporter setup for metrics collection. | Network, Router HTTP handler | Router network |
memory.rssBytes | kubelet Summary API | router-resource-usage (mode: job), nodeMetrics (mode: local) | Physical RAM in use | nodes/stats RBAC grant (mode: job), or nodes and nodes/proxy (mode: local) [2] | Kubelet CPU (plus API server CPU in mode: local) | Cluster network → kubelet (mode: job); k8s control plane → kubelet (mode: local) [1] |
memory.usageBytes, memory.availableBytes | kubelet Summary API | router-resource-usage (mode: job), nodeMetrics (mode: local) | Usage and headroom remaining against the limit | nodes/stats RBAC grant (mode: job), or nodes and nodes/proxy (mode: local) [2] | Kubelet CPU (plus API server CPU in mode: local) | Cluster network → kubelet (mode: job); k8s control plane → kubelet (mode: local) [1] |
memory.pageFaults, memory.majorPageFaults | kubelet Summary API | router-resource-usage (mode: job), nodeMetrics (mode: local) | Major faults indicate real paging pressure rather than growth alone | nodes/stats RBAC grant (mode: job), or nodes and nodes/proxy (mode: local) [2] | Kubelet CPU (plus API server CPU in mode: local) | Cluster network → kubelet (mode: job); k8s control plane → kubelet (mode: local) [1] |
cpu.usageNanoCores, cpu.usageCoreNanoSeconds | kubelet Summary API | router-resource-usage (mode: job), nodeMetrics (mode: local) | CPU consumed by the router container | nodes/stats RBAC grant (mode: job), or nodes and nodes/proxy (mode: local) [2] | Kubelet CPU (plus API server CPU in mode: local) | Cluster network → kubelet (mode: job); k8s control plane → kubelet (mode: local) [1] |
cpu.psi, memory.psi (pressure stall information) | kubelet Summary API | router-resource-usage (mode: job), nodeMetrics (mode: local) | Whether the container is stalling on CPU or memory | nodes/stats RBAC grant (mode: job), or nodes and nodes/proxy (mode: local) [2], plus the KubeletPSI feature gate on your cluster | Kubelet CPU (plus API server CPU in mode: local) | Cluster network → kubelet (mode: job); k8s control plane → kubelet (mode: local) [1] |
[1] The kubelet Summary API reports per node, per pod, and per container — this assumes one router process per container.
[2] Container memory and CPU come from the kubelet, never from inside the router container. mode: job reads the kubelet directly and needs the nodes/stats grant. mode: local goes through the API server's proxy with your own credentials, which needs nodes and nodes/proxy. The Router's Prometheus exporter needs no grant and provides the Router's own jemalloc memory data, but not container RSS, working set, or CPU. Under mode: local, nodeMetrics can't be narrowed to the router, so node-metrics/ lists every pod on every node in the cluster, including other workloads' pod names, namespaces, and container CPU and memory usage. Under mode: job, only the router's pods are kept. If you decline the grants (job.collectNodeMetrics: false), you lose container memory and CPU data and node pressure conditions. See Kubelet stats RBAC for what each grant authorizes.
Kubelet stats RBAC
Under mode: job, the Job reads the kubelet directly from its pod, so the kubelet's own authorization applies and the broad nodes/proxy grant isn't needed. The chart's ClusterRole grants the following:
nodes/stats(get): The kubelet's/stats/summary, with per-pod and per-container memory, CPU, and pressure stall figures. This is the grant that surfaces container memory and CPU usage.nodes(list): Lists node objects. This is what surfaces node pressure conditions (MemoryPressure,DiskPressure). Container memory and CPU don't need it. If you manage the ServiceAccount yourself you can leave it out and lose only those conditions.
Neither grant can be scoped to only the router's nodes, because the node names aren't known when you install the chart. The tool keeps only the router pods' entries from each kubelet response so other workloads' pods never reach the bundle.
Credentials sent to the kubelet. The Job sends its ServiceAccount token to each router node's kubelet, and doesn't verify the kubelet's TLS certificate, because kubelet certificates are commonly self-signed or signed by a CA other than the cluster's. Anyone who could intercept traffic between the Job's pod and the node could capture the token, so it expires after 10 minutes. Until then, it can do only what the ServiceAccount is allowed to. With the ServiceAccount the chart creates, that is read-only access to the namespace's pods, pod logs, ConfigMaps, deployments, and a few other resource types (the full list is in Kubernetes RBAC), plus nodes and nodes/stats across the cluster. That ServiceAccount has no access to Secrets. The token is never written to the bundle.
To keep container memory and CPU usage in the bundle, leave these grants in place. Declining them still leaves everything clusterResources provides: OOM kill occurrences and last state, restart counts, and configured limits. Setting job.collectNodeMetrics: false skips creating the ClusterRole and ClusterRoleBinding at install time, so container memory and CPU usage and node pressure conditions are missing, and the bundle says the kubelet read was declined. The install doesn't fail.
For what each grant collects, see Memory and CPU information collected. For what mode: local needs, see Kubernetes RBAC.
Running a service mesh or sidecar proxy?
The logs collector captures every container in the pod, and the kubelet Summary API reports per container, so a sidecar's own resource usage (Istio, Linkerd, Envoy) is visible in the bundle alongside your router's, which is useful for telling a proxy problem apart from a router one. If you're relying on the Prometheus metrics row, a mesh enforcing strict mTLS can block that scrape specifically. For more information, go to Collecting Metrics.
Before you share a bundle
Redactors cover the router's own configuration and known sensitive patterns—Redis credentials, TLS private keys, the response cache's invalidation shared_key, JWT/auth config, and similar. Redactors don't know about custom instrumentation you've added: for example, a Rhai script that sets span attributes, custom telemetry configuration that records request context, or a coprocessor that writes its own fields into router logs. Before sharing a bundle, check the collected logs under cluster-resources/pods/logs/ for anything your own configuration writes there, and redact the information yourself if needed.
If your deployment sets APOLLO_KEY, or any other secret, as a literal pod-spec env value rather than through a Kubernetes Secret, inspect your bundle before sharing it. The pod spec is collected in full, literal values included, so a secret set this way is only protected by a redaction rule that masks any env var named *_KEY or *_PASS. This redaction rule is a safety net, not the structural isolation a Secret-backed APOLLO_KEY gets.