Data Collected

Exactly what the base spec gathers and what each signal requires


These tables outline everything we collect with the GraphOS Router Support Tool. This page is meant to be shared with your platform team so they can review it, grant the access needed for the collectors they want, and know exactly data won't be collected if they decline a collector.

What the base spec collects

SourceSignalCollected viaWhat it tells youRequiresResource consumedWhere
k8s APIPod status, restart counts, resource limitsclusterResources collectorWhether pods are healthy, restart count, resource limits configuredNone additionalAPI server CPUk8s control plane
Container image tagRouter versionclusterResources collectorRouter versionNone additionalAPI server CPUk8s control plane
Pod spec (spec.containers[].env)Router deployment env vars: APOLLO_GRAPH_REF, APOLLO_ROUTER_OFFICIAL_HELM_CHARTclusterResources collectorGraph ref and whether the router was deployed via Apollo's official Helm chartNone additionalAPI server CPUk8s control plane
Separate <release>-supergraph ConfigMapSchema (SDL)clusterResources collector and the dedicated configMap collectorFull graph schema.Values.supergraphFile set on the router's Helm chart. For more information, go to Schema collectionAPI server CPUk8s control plane
Container log streamRuntime logslogs collectorRecent router output, plus crash output from the previous container when one existsNone additionalNetwork bandwidthCluster network
Router metrics endpoint, per podFull Prometheus metrics snapshotrun collector (router-metrics), in both modesComplete operational metrics—request rates, error rates, latency, traffic shaping state—from every matching router pod individuallyPrometheus exporter enabled and reachably bound. For more information, see Collecting MetricsNetwork, router HTTP handlerRouter network
ConfigMap holding the rendered configSanitized router.yamlconfigMap collectorFull router configuration—traffic shaping, timeouts, plugins, feature flagsA matching labeled ConfigMap present or configMapName/selector values set. For more information, see router.yaml capture belowAPI server CPUk8s control plane

A couple of notes that apply across the table:

  • Router deployment env vars, image tag, and pod status all come from the same pod collection, so declining or losing one doesn't affect the others.

  • If APOLLO_GRAPH_REF is routed through a Secret using your router chart's extraEnvVars, only the environment variable reference is captured, not the value, so the graph ref itself doesn't land in the bundle.

  • The same pod-spec collection captures every other env var too, literal values included. APOLLO_KEY is never collected as long as it's stored in a Kubernetes Secret and referenced via secretKeyRef, the recommended setup, and what the official Apollo router Helm chart produces, because in that case the key is structurally isolated from everything this tool reads, not redacted after the fact.

caution
That guarantee doesn't extend to a deployment that sets APOLLO_KEY, or any other secret, as a literal env value. If your deployment sets secrets as literal env values rather than through Kubernetes Secrets, inspect your bundle before sharing it.
  • Config is captured as written, not effective config. Environment-variable overrides applied on top of router.yaml aren't reflected in what's collected.

  • Previous-container logs are always collected and written to <name>-previous.log, so a router that already restarted still has its crash output captured. Logs are collected per pod and capture every container in the pod, including proxy/mesh sidecars. How far back logs reach and how many lines are kept are configurable. For more information, access the chart's logs.maxAge/logs.maxLines values in job-mode-details or local-mode-details.

router.yaml capture

The configMap collector reads the ConfigMap holding your router's rendered configuration directly. On the official Apollo router Helm chart, it's targeted by the standard app.kubernetes.io/name=router label. On a raw-manifest or custom deployment, supply configMapName/selector yourself (go to Getting Started guide), without one of those, and without config discoverable under the standard label, this section comes back empty.

If more than one ConfigMap in your namespace matches, every one of them is collected, each in its own file named after that ConfigMap's own Kubernetes name — not a Helm release, which a raw-manifest or custom deployment doesn't have for its router at all.

keyExists: false in configmaps/<namespace>/<configmap-name>.json doesn't mean the config is missing. This is a troubleshoot.sh quirk: the configMap collector's keyExists field is only ever set true when the collector is configured with a specific key to look for. Our spec uses includeAllData: true with no key set so keyExists is always false in this output, regardless of whether the config was actually captured. Check for the config itself, the data in that same file, rather than reading keyExists as a success/failure signal.

Known limitations:

  • A collected router.yaml isn't verified to be the one your router actually loaded. Treat a populated configmaps/ section as "a matching ConfigMap exists," not "confirmed active."

  • Config may not be collected at all, if your router reads router.yaml from somewhere this tool doesn't look, for example, mounted from a Secret, pulled by an init container, or mounted from a PVC.

  • Environment-variable secrets are masked by name only. For example, if Apollo Connectors' shared_key is set as ${env.INVALIDATION_KEY}, the router reads the real value from the pod’s environment. If that variable is a literal pod-spec value, it’s collected with the pod spec and is masked only when its name ends in KEY or PASS.

  • An empty shared_key (shared_key: "") is left as it is. This is the router's default and means no invalidation key was set.

  • If you set override_subgraph_url with a normal URL (one with a path, e.g. /graphql), the section of your router.yaml immediately following it may be missing its key name in the collected bundle. This is over-redaction, not a leak due to a troubleshoot.sh built-in redactor that matches more than it should past that URL and consumes the next line's key name before stopping, leaving that section's contents intact but orphaned under the wrong parent. If a section of your config looks like it's missing a heading, check for this before assuming your config is broken or wasn't collected.

Schema collection

Schema only lands in the cluster, and therefore only in the bundle, when you set .Values.supergraphFile on your router's own Helm chart (not this tool's chart). If you're on managed federation (GraphOS schema governance), you never set that value, so there's no schema ConfigMap for this collector to find, and schema is simply absent from the bundle.

When it is set, the <release>-supergraph ConfigMap carries the same app.kubernetes.io/name=router label as your router's own config, so it's picked up twice: once by clusterResources's full sweep (cluster-resources/configmaps/<namespace>.json, alongside everything else in the namespace), and once by the dedicated configMap collector's own targeted match, landing as its own file (configmaps/<namespace>/<release>-supergraph.json).

Memory and CPU information collected

The data collected with the support tool is sufficient to detect, confirm, and characterize a memory problem: is memory growing, how fast, how close to the limit, and is the kernel killing the container. To tell real heap growth from jemalloc fragmentation, compare the jemalloc aggregate gauges (apollo.router.jemalloc.active, .allocated, .resident, .retained, and others) on the Prometheus endpoint. Router doesn't serve process-level RSS or CPU metrics, so use the memory.rssBytes and cpu.* rows in the following table for those.

Every row in this table represents a collection that is attempted on every run, regardless of what else is configured. The "Requires" column states what has to be true for that specific row to come back populated; it isn't a condition on any other row.

SignalSourceCollected viaWhat it tells youRequiresResource consumedWhere
memory.workingSetByteskubelet Summary APIrouter-resource-usage (mode: job), nodeMetrics (mode: local)How much memory k8s counts against the limit — the OOM kill thresholdnodes/stats RBAC grant (mode: job), or nodes and nodes/proxy (mode: local) [2]Kubelet CPU (plus API server CPU in mode: local)Cluster network → kubelet (mode: job); k8s control plane → kubelet (mode: local) [1]
Configured resources.limits / requestsPod specclusterResources collectorWhat the usage numbers above should be compared againstNone additionalAPI server CPUk8s control plane
OOM kill occurrencesKubernetes events (OOMKilling, evictions)clusterResources collectorThat OOM kills happened, when, and how oftenNone additionalAPI server CPUk8s control plane
OOM kill last statePod status lastState.terminated.reason: OOMKilled, restartCountclusterResources collectorWhether the most recent restart was an OOM killNone additionalAPI server CPUk8s control plane
Node MemoryPressure / DiskPressure conditionsNode objectsclusterResources collectorWhether the node itself is under pressure, distinguishing a router problem from a neighbor'snodes (list) RBAC grant, cluster-scoped [2]API server CPUk8s control plane
apollo_router_jemalloc_allocated, _active, _resident, _mapped, _metadata, _retainedRouter Prometheus endpoint, per podrun collector (router-metrics)Router's own allocator view of memory: separates real heap growth (allocated) from allocator retention (resident, retained).Prometheus exporter enabled and reachably bound, on Router v2.5.0 or later built with jemalloc, as the official release builds are. See Prometheus exporter setup for metrics collection.Network, Router HTTP handlerRouter network
memory.rssByteskubelet Summary APIrouter-resource-usage (mode: job), nodeMetrics (mode: local)Physical RAM in usenodes/stats RBAC grant (mode: job), or nodes and nodes/proxy (mode: local) [2]Kubelet CPU (plus API server CPU in mode: local)Cluster network → kubelet (mode: job); k8s control plane → kubelet (mode: local) [1]
memory.usageBytes, memory.availableByteskubelet Summary APIrouter-resource-usage (mode: job), nodeMetrics (mode: local)Usage and headroom remaining against the limitnodes/stats RBAC grant (mode: job), or nodes and nodes/proxy (mode: local) [2]Kubelet CPU (plus API server CPU in mode: local)Cluster network → kubelet (mode: job); k8s control plane → kubelet (mode: local) [1]
memory.pageFaults, memory.majorPageFaultskubelet Summary APIrouter-resource-usage (mode: job), nodeMetrics (mode: local)Major faults indicate real paging pressure rather than growth alonenodes/stats RBAC grant (mode: job), or nodes and nodes/proxy (mode: local) [2]Kubelet CPU (plus API server CPU in mode: local)Cluster network → kubelet (mode: job); k8s control plane → kubelet (mode: local) [1]
cpu.usageNanoCores, cpu.usageCoreNanoSecondskubelet Summary APIrouter-resource-usage (mode: job), nodeMetrics (mode: local)CPU consumed by the router containernodes/stats RBAC grant (mode: job), or nodes and nodes/proxy (mode: local) [2]Kubelet CPU (plus API server CPU in mode: local)Cluster network → kubelet (mode: job); k8s control plane → kubelet (mode: local) [1]
cpu.psi, memory.psi (pressure stall information)kubelet Summary APIrouter-resource-usage (mode: job), nodeMetrics (mode: local)Whether the container is stalling on CPU or memorynodes/stats RBAC grant (mode: job), or nodes and nodes/proxy (mode: local) [2], plus the KubeletPSI feature gate on your clusterKubelet CPU (plus API server CPU in mode: local)Cluster network → kubelet (mode: job); k8s control plane → kubelet (mode: local) [1]

[1] The kubelet Summary API reports per node, per pod, and per container — this assumes one router process per container.

[2] Container memory and CPU come from the kubelet, never from inside the router container. mode: job reads the kubelet directly and needs the nodes/stats grant. mode: local goes through the API server's proxy with your own credentials, which needs nodes and nodes/proxy. The Router's Prometheus exporter needs no grant and provides the Router's own jemalloc memory data, but not container RSS, working set, or CPU. Under mode: local, nodeMetrics can't be narrowed to the router, so node-metrics/ lists every pod on every node in the cluster, including other workloads' pod names, namespaces, and container CPU and memory usage. Under mode: job, only the router's pods are kept. If you decline the grants (job.collectNodeMetrics: false), you lose container memory and CPU data and node pressure conditions. See Kubelet stats RBAC for what each grant authorizes.

Kubelet stats RBAC

Under mode: job, the Job reads the kubelet directly from its pod, so the kubelet's own authorization applies and the broad nodes/proxy grant isn't needed. The chart's ClusterRole grants the following:

  • nodes/stats (get): The kubelet's /stats/summary, with per-pod and per-container memory, CPU, and pressure stall figures. This is the grant that surfaces container memory and CPU usage.

  • nodes (list): Lists node objects. This is what surfaces node pressure conditions (MemoryPressure, DiskPressure). Container memory and CPU don't need it. If you manage the ServiceAccount yourself you can leave it out and lose only those conditions.

Neither grant can be scoped to only the router's nodes, because the node names aren't known when you install the chart. The tool keeps only the router pods' entries from each kubelet response so other workloads' pods never reach the bundle.

Credentials sent to the kubelet. The Job sends its ServiceAccount token to each router node's kubelet, and doesn't verify the kubelet's TLS certificate, because kubelet certificates are commonly self-signed or signed by a CA other than the cluster's. Anyone who could intercept traffic between the Job's pod and the node could capture the token, so it expires after 10 minutes. Until then, it can do only what the ServiceAccount is allowed to. With the ServiceAccount the chart creates, that is read-only access to the namespace's pods, pod logs, ConfigMaps, deployments, and a few other resource types (the full list is in Kubernetes RBAC), plus nodes and nodes/stats across the cluster. That ServiceAccount has no access to Secrets. The token is never written to the bundle.

To keep container memory and CPU usage in the bundle, leave these grants in place. Declining them still leaves everything clusterResources provides: OOM kill occurrences and last state, restart counts, and configured limits. Setting job.collectNodeMetrics: false skips creating the ClusterRole and ClusterRoleBinding at install time, so container memory and CPU usage and node pressure conditions are missing, and the bundle says the kubelet read was declined. The install doesn't fail.

For what each grant collects, see Memory and CPU information collected. For what mode: local needs, see Kubernetes RBAC.

Running a service mesh or sidecar proxy?

The logs collector captures every container in the pod, and the kubelet Summary API reports per container, so a sidecar's own resource usage (Istio, Linkerd, Envoy) is visible in the bundle alongside your router's, which is useful for telling a proxy problem apart from a router one. If you're relying on the Prometheus metrics row, a mesh enforcing strict mTLS can block that scrape specifically. For more information, go to Collecting Metrics.

Before you share a bundle

caution
Every file in the bundle is redacted automatically, but redaction can fail on a single file without failing the whole run. If a bundle reports a redaction error, don't share it until you understand what it means. An error might mean redaction failed to complete on a file.

Redactors cover the router's own configuration and known sensitive patterns—Redis credentials, TLS private keys, the response cache's invalidation shared_key, JWT/auth config, and similar. Redactors don't know about custom instrumentation you've added: for example, a Rhai script that sets span attributes, custom telemetry configuration that records request context, or a coprocessor that writes its own fields into router logs. Before sharing a bundle, check the collected logs under cluster-resources/pods/logs/ for anything your own configuration writes there, and redact the information yourself if needed.

If your deployment sets APOLLO_KEY, or any other secret, as a literal pod-spec env value rather than through a Kubernetes Secret, inspect your bundle before sharing it. The pod spec is collected in full, literal values included, so a secret set this way is only protected by a redaction rule that masks any env var named *_KEY or *_PASS. This redaction rule is a safety net, not the structural isolation a Secret-backed APOLLO_KEY gets.