Collecting Metrics

Prometheus prerequisites, per-mode mechanics, and service mesh caveats


A full Prometheus metrics snapshot, from every matching router pod individually, is collected. Both modes collect it with a hostCollectors.run script named router-metrics that finds your router pods and scrapes each one. This page covers everything specific to getting that signal: what has to be true on your router's side, how each pod is reached, where the metrics land in the bundle, what to do when they're missing, and what changes if you're running a service mesh. To understand how this snapshot works alongside all other data the tool gathers, go to Data Collected.

Prerequisites on the router's side

Two settings need to be in place:

  • telemetry.exporters.metrics.prometheus.enabled: true in router.yaml: turns the exporter on.

  • telemetry.exporters.metrics.prometheus.listen in router.yaml: The bind address. Binding the exporter to loopback means an external scrape can't reach the exporter, no matter what port the collector targets.

If either is turned off or misconfigured, the bundle has no metrics files and says why instead. The rest of the bundle is unaffected. See If metrics are missing for how to read that explanation.

Which pods get scraped

selector controls which pods are scraped for metrics. It defaults to app.kubernetes.io/name=router, the official Apollo router chart's own label. Raw-manifest and custom deployments set their own selector. The same selector also decides which pods' logs are collected and, unless you set configMapName, which ConfigMap is collected. Override metricsPort if your exporter listens somewhere other than 9090.

The pods are found each time collection runs. Each run lists the pods that currently match selector in your namespace and scrapes those:

  • Collection works the same whether you installed the chart with helm install or deployed it through ArgoCD

  • Pods replaced since you installed the chart, for example by a rollout, are scraped under their new names

If your router's chart name was overridden

On the official Apollo router Helm chart, setting router.nameOverride, or installing the chart under an alias, changes the value of the pod's app.kubernetes.io/name label, so the default selector no longer matches anything. To find the label your router pods actually carry:

Bash
1kubectl get pods --namespace production --show-labels

and set selector to match it when you install this chart, for example --set selector="app.kubernetes.io/name=my-router".

mode: job

The script runs inside the Job's pod. It lists the pods matching selector with kubectl, then requests /metrics from each pod's IP directly on metricsPort, with no port-forward, Service or DNS lookup involved. It scrapes ten pods at a time and gives each request up to 30 seconds, so even when many pods don't respond, collection takes about 30 seconds per ten pods. A matched pod that has no IP yet, for example one still 'Pending', isn't scraped.

To list the pods, the Job needs get and list on pods in your namespace. The Role the chart creates for the Job already grants both, so there's nothing extra to set up. The Job image bundles kubectl and curl for this. For the full RBAC list, see mode: job Reference → Kubernetes RBAC.

mode: local

The script runs on your own machine, where pod IPs aren't reachable:

  1. Looks up every pod matching selector in your namespace (kubectl get pods -l <selector>).

  2. For each pod in turn: port-forwards straight to that pod (not a Service), scrapes /metrics to a file, then tears the port-forward down before moving to the next pod.

This needs get and list on pods, plus create on the pods/portforward subresource, in the router's namespace, all using your own kubectl credentials. Declining pods/portforward doesn't fail collection: no metrics are collected, and each pod's line in the bundle includes the port-forward error from kubectl.

Where metrics land in the bundle

Each pod's scrape is written to its own <pod-name>.txt file, in Prometheus text format. troubleshoot.sh nests a hostCollectors.run collector's output under host-collectors/run-host/<collectorName>/:

Text
1host-collectors/run-host/
2├── router-metrics-info.json
3├── router-metrics.txt
4├── router-metrics-stderr.txt
5└── router-metrics/
6    └── pods/
7        ├── <pod-name>.txt
8        └── <pod-2-name>.txt

router-metrics.txt holds the script's own output: one line for each pod it couldn't scrape, or for why it couldn't scrape any. See If metrics are missing for what each line means. router-metrics-stderr.txt holds the script's stderr, such as the error kubectl returned when it couldn't list pods. It is only present if the script writes to stderr.

note
This is a change for mode: job bundles from earlier versions. Earlier versions of the chart wrote each pod's metrics in mode: job to a top-level router-metrics-<pod-name>/result.json, wrapped in troubleshoot.sh's http collector result format. Both modes now use the host-collectors/run-host/router-metrics/pods/<pod-name>.txt path shown above, and each file holds the pod's raw Prometheus response. If you read mode: job bundles by path, in a script or manually, update them to the new path. Check a bundle's meta.json version to see which chart version produced it.

A second file, router-metrics-info.json, is fully masked. troubleshoot.sh writes this diagnostic sidecar for every hostCollectors.run collector and records the full exec invocation, including the process environment collection runs with: your own shell under mode: local (collect.sh or collect.ps1), or the Job's container under mode: job. Multiple safeguards limit what ends up in the bundle:

  • The chart sets ignoreParentEnvs: true, which narrows the captured environment to just PATH, KUBECONFIG, and PWD before the file is written.

  • A redactor masks the entire contents of the file.

If metrics are missing

When a pod has no file under router-metrics/pods/, host-collectors/run-host/router-metrics.txt says why, with one of these lines:

Line in router-metrics.txtWhat happenedWhat to do
metrics not collected: could not list pods matching selector <selector> in namespace <namespace>The tool couldn't ask Kubernetes for the pod list. Either the identity running collection lacks get/list on pods (the Job's Role under mode: job, your own credentials under mode: local), or the Kubernetes API server couldn't be reached. This isn't a selector problem.Read host-collectors/run-host/router-metrics-stderr.txt for the error kubectl returned. Under mode: job, confirm the chart's Role and RoleBinding are still in place. Under mode: local, confirm kubectl get pods --namespace <namespace> works with your credentials. If it does, your kubeconfig may need environment variables the script doesn't receive. See If the bundle says it "could not list pods" under mode: local.
metrics not collected: no pods matched selector <selector> in namespace <namespace>The pod list came back empty. selector or namespace doesn't match your router's pods.Run kubectl get pods --namespace <namespace> --show-labels and set selector (and namespace, if needed) to match. See If your router's chart name was overridden.
metrics not collected for pod <pod-name>: no response on port <metricsPort>That one pod's exporter didn't answer: it might be down, restarting, or blocked by a service mesh. Other pods' files are unaffected. Under mode: local, the line can end with — kubectl port-forward: <error>. That means the port-forward to the pod itself failed, for example with a forbidden error naming pods/portforward, or because the local port was already in use.If the line ends with a port-forward error, confirm you have create on pods/portforward in the router's namespace, and that nothing else on your machine is using port <metricsPort>. Otherwise, check that pod's status and logs in the same bundle. If every pod shows this line, read the last row.
metrics not collected for pod <pod-name>: pod has no IP yetmode: job only. The pod hadn't been given an IP yet, usually because it was still Pending or starting, so there was nothing to scrape. This isn't an exporter problem, and other pods' files are unaffected. Under mode: local, a pod in this state gets the "no response" line with a port-forward error instead.Collect again once the pod is Running. If it stays Pending, check its events in cluster-resources/events/<namespace>.json in the same bundle.
metrics not collected: no pod responded on port <metricsPort> — prometheus may not be enabledPods were scraped, but none of them answered. Under mode: local, each pod's own line above says whether its port-forward failed. This line isn't written when no matched pod had an IP to scrape; the per-pod lines already say why.If those lines end with a port-forward error, fix that first, as in the previous row. Otherwise, check the prerequisites above, that metricsPort matches the port in your listen address, and, if you run a service mesh, Running a service mesh? below.

If the bundle says it "could not list pods" under mode: local

The router-metrics script runs with a minimal environment: of your shell’s variables, it keeps only KUBECONFIG. A kubeconfig that needs other environment variables to authenticate, for example an auth plugin that reads credentials from AWS_PROFILE, cannot authenticate there. The script then writes metrics not collected: could not list pods matching selector … and saves the kubectl error in router-metrics-stderr.txt next to it.

To get metrics, point KUBECONFIG at a kubeconfig whose credentials don't depend on other environment variables, then run the collection script again.

Running a service mesh?

Neither mode: local (your own machine) nor an un-injected mode: job Job pod is inside the mesh. logs, clusterResources, nodeMetrics, router-resource-usage, and configMap all reach their data through the Kubernetes API server or the kubelet, which the mesh doesn't sit in front of, so running outside it doesn't affect them. The Prometheus scrape is the one collector that talks directly to a router port, and a mesh enforcing strict mTLS will reject it from a pod outside the mesh. To fix that, exempt the metrics port from mTLS. For Istio, PeerAuthentication: PERMISSIVE scoped to that port, or the traffic.sidecar.istio.io/excludeInboundPorts annotation. For Linkerd, config.linkerd.io/skip-inbound-ports. Apply only once to your router deployment, not a setting on this tool's chart.

If you're running mode: job in a mesh, go to Job Mode → Service mesh environments for an explanation of how the mesh sidecar itself can keep the Job's pod from ever reaching Completed, which is a separate problem from whether the scrape succeeds.