Support bundle storage and retrieval

Configuring job.storage, and getting the bundle back out


Storage only applies to and is required for mode: job. Because there is no way to get the bundle out otherwise, the chart fails the install at render time if mode: job is set and job.storage.provider is left unset.

ValueRequiredPurpose
job.storage.providerYes, for mode: jobs3, gcs, or url
job.storage.bucketRequired for s3/gcsDestination bucket for the uploaded bundle
job.storage.prefixNoKey prefix within the bucket, for example, router-bundles/
job.storage.s3.regionRequired for s3 unless implied by endpointAWS region, or the equivalent for an S3-compatible store
job.storage.s3.endpointNoOverride for a non-AWS S3-compatible endpoint (MinIO, Ceph RGW, etc.)
job.storage.s3.forcePathStyleNoMost on-prem S3-compatible stores need this set true
job.storage.s3.accessKeyId / secretAccessKeyNo, not recommendedStatic credential templated directly into the Job's container. Visible in helm get values and git history if committed. Prefer existingSecret or IRSA.
job.storage.gcs.projectNoThe GCP project gcloud bills API calls to
job.storage.gcs.credentialsJsonNo, not recommendedA GCP service-account JSON key, same exposure caveat as job.storage.s3.accessKeyId. Set via -f values.yaml or --set-json, never plain --set because Helm's --set parses commas and braces as its own syntax and silently distorts the JSON.
job.storage.url.endpointRequired for urlThe bare HTTP(S) endpoint the bundle is uploaded to
job.storage.url.methodNoPUT or POST, defaults to PUT
job.storage.url.headersNoStatic headers templated directly into the request. Same exposure caveat as job.storage.s3.accessKeyId.
job.storage.existingSecretNoName of a Secret you create yourself. Go to Credentials for what it holds by provider.

For provider: s3/gcs, existingSecret and a static credential (s3.accessKeyId/secretAccessKey or gcs.credentialsJson) are mutually exclusive. Setting both fails at helm install/helm template with an explicit error.

Examples

S3, via IRSA (recommended—no Secret to manage)

Bash
1helm install router-diagnostics oci://registry-1.docker.io/apollograph/router-diagnostics-chart \
2  --namespace production \
3  --set namespace=production \
4  --set mode=job \
5  --set job.storage.provider=s3 \
6  --set job.storage.bucket=my-router-diagnostics \
7  --set job.storage.prefix=router-bundles/ \
8  --set job.storage.s3.region=us-east-1 \
9  --set job.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::123456789012:role/router-diagnostics-s3-writer

S3, via a Secret you create yourself (no IRSA available)

Bash
1kubectl create secret generic router-diagnostics-storage \
2  --namespace production \
3  --from-literal=AWS_ACCESS_KEY_ID=<access-key-id> \
4  --from-literal=AWS_SECRET_ACCESS_KEY=<secret-access-key>
5
6helm install router-diagnostics oci://registry-1.docker.io/apollograph/router-diagnostics-chart \
7  --namespace production \
8  --set namespace=production \
9  --set mode=job \
10  --set job.storage.provider=s3 \
11  --set job.storage.bucket=my-router-diagnostics \
12  --set job.storage.prefix=router-bundles/ \
13  --set job.storage.s3.region=us-east-1 \
14  --set job.storage.existingSecret=router-diagnostics-storage

An S3-compatible store (MinIO, Ceph RGW)

This option needs forcePathStyle and an explicit endpoint, and typically has no IRSA-equivalent federation, so a Secret is the normal path:

Bash
1kubectl create secret generic router-diagnostics-storage \
2  --namespace production \
3  --from-literal=AWS_ACCESS_KEY_ID=<access-key-id> \
4  --from-literal=AWS_SECRET_ACCESS_KEY=<secret-access-key>
5
6helm install router-diagnostics oci://registry-1.docker.io/apollograph/router-diagnostics-chart \
7  --namespace production \
8  --set namespace=production \
9  --set mode=job \
10  --set job.storage.provider=s3 \
11  --set job.storage.bucket=my-router-diagnostics \
12  --set job.storage.s3.endpoint=https://minio.internal:9000 \
13  --set job.storage.s3.forcePathStyle=true \
14  --set job.storage.existingSecret=router-diagnostics-storage
Bash
1helm install router-diagnostics oci://registry-1.docker.io/apollograph/router-diagnostics-chart \
2  --namespace production \
3  --set namespace=production \
4  --set mode=job \
5  --set job.storage.provider=gcs \
6  --set job.storage.bucket=my-router-diagnostics \
7  --set job.storage.prefix=router-bundles/ \
8  --set job.storage.gcs.project=my-gcp-project \
9  --set job.serviceAccount.annotations."iam\.gke\.io/gcp-service-account"=router-diagnostics@my-gcp-project.iam.gserviceaccount.com

GCS, via a Secret holding a service-account key

For a cluster where Workload Identity isn't available—one that's not running on GKE or is running on GKE with Workload Identity not enabled for the cluster or namespace—hold a static service-account key in a Secret instead. Note -f here—--set would distort the JSON:

Bash
1kubectl create secret generic router-diagnostics-storage \
2  --namespace production \
3  --from-file=key.json=/path/to/service-account-key.json
4
5cat <<EOF > storage-values.yaml
6job:
7  storage:
8    provider: gcs
9    bucket: my-router-diagnostics
10    prefix: router-bundles/
11    gcs:
12      project: my-gcp-project
13    existingSecret: router-diagnostics-storage
14EOF
15
16helm install router-diagnostics oci://registry-1.docker.io/apollograph/router-diagnostics-chart \
17  --namespace production \
18  --set namespace=production \
19  --set mode=job \
20  -f storage-values.yaml

A bare HTTP(S) endpoint (provider: url)

This example authenticates with a bearer token via a Secret. Each key in the Secret becomes a header name:

Bash
1kubectl create secret generic router-diagnostics-storage \
2  --namespace production \
3  --from-literal=Authorization="Bearer <token>"
4
5helm install router-diagnostics oci://registry-1.docker.io/apollograph/router-diagnostics-chart \
6  --namespace production \
7  --set namespace=production \
8  --set mode=job \
9  --set job.storage.provider=url \
10  --set job.storage.url.endpoint=https://example.com/upload/support-bundle \
11  --set job.storage.url.method=PUT \
12  --set job.storage.existingSecret=router-diagnostics-storage

Or the same endpoint with a static header set directly as a value. Only use this option for a header that isn't sensitive since this is visible in helm get values and git history.

Bash
1helm install router-diagnostics oci://registry-1.docker.io/apollograph/router-diagnostics-chart \
2  --namespace production \
3  --set namespace=production \
4  --set mode=job \
5  --set job.storage.provider=url \
6  --set job.storage.url.endpoint=https://example.com/upload/support-bundle \
7  --set job.storage.url.headers.X-Upload-Source=router-diagnostics

Credentials

For s3/gcs, bundle uploads go through the provider's own CLI (aws s3 cp / gcloud storage cp), so any credential source the CLI already knows how to resolve works. (provider: url doesn't use a CLI at all; it's a plain curl -X <method> -T <bundle> <endpoint>, so its only credential mechanism is the HTTP headers described further on.)

  • IRSA (AWS) / Workload Identity (GCP): Recommended. No secrets to store or rotate. Requires setting job.serviceAccount.annotations with eks.amazonaws.com/role-arn or iam.gke.io/gcp-service-account. This is an annotation on the ServiceAccount, not a Kubernetes RBAC grant.

  • Static credentials via a Secret: For clusters without IRSA/Workload Identity (on-prem, or an S3-compatible store like MinIO/Ceph with no equivalent federation). Set job.storage.existingSecret to a Secret you create yourself, with AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY keys for provider: s3 or a GCP service-account JSON key for provider: gcs. The chart mounts the Secret into the Job's container so we never see or template the values themselves.

  • Static credentials via values.: Not recommended. job.storage.s3.accessKeyId/secretAccessKey or job.storage.gcs.credentialsJson set directly. The credential is exposed in shell history, helm get values, and git history if committed. Prefer other options whenever you have any way to manage a Secret directly.

For provider: url, whatever the endpoint needs (a bearer token, a signed-URL header) is supplied as an HTTP header, two ways, combinable:

  • Through job.storage.existingSecret: A Secret you create with no, or more arbitrary, keys. Each key becomes a header name, with its value the header's value. Mounted as a volume.

  • Through job.storage.url.headers: Templated directly into the request, with the same exposure caveat as other static-values paths.

Headers from both sources are sent together when both are set (existingSecret first, then url.headers). A header name appearing in both is sent twice.

Cloud IAM for storage

Whichever credential path you use, the minimum cloud permissions are the same:

  • S3: s3:PutObject scoped to arn:aws:s3:::<bucket>/<prefix>* (or the equivalent bucket policy for an S3-compatible store without IAM ARNs)

  • GCS: storage.objects.create on the bucket (the predefined roles/storage.objectCreator role covers this)

Neither needs read, list, or delete on the bucket. Instead, the Job only ever writes one object per run. This doesn't apply to provider: url. There, authentication is whatever the receiving endpoint enforces using headers.

Retention on the destination is governed by the bucket. The tool doesn't build or operate a retention mechanism of its own.

Retrieval

Once the Job completes and uploads succeed, the bundle is retrieved the same way you'd retrieve anything else from that destination:

  • S3 — aws s3 cp s3://<bucket>/<prefix>support-bundle-<timestamp>.tar.gz ., or browse the bucket in the console. The uploading identity (IRSA role or static credential) only has PutObject, so a separate identity with read access is normally what your platform team uses to pull it down.

  • GCS — gcloud storage cp gs://<bucket>/<prefix>support-bundle-<timestamp>.tar.gz ., or browse the bucket in the console. Same IAM note as S3: the uploading identity can't read the bucket back.

  • A bare endpoint (provider: url) — retrieval is entirely up to whatever the receiving service does with the upload; this tool's role ends once the PUT/POST succeeds.

Before you share a bundle

caution
Every file in the bundle is redacted automatically, but redaction can fail on a single file without failing the whole run. If a bundle reports a redaction error, don't share it until you understand what it means. An error might mean redaction failed to complete on a file.

Redactors cover Router's own configuration and known sensitive patterns—Redis credentials, TLS private keys, JWT/auth config, and similar. The redactors don't know about custom instrumentation you've added: for example, a Rhai script that sets span attributes, custom telemetry configuration that records request context, or a coprocessor that writes its own fields into router logs. Before sharing a bundle, check the collected logs under cluster-resources/pods/logs/ for anything your own configuration writes there, and redact the information yourself if needed.

If your deployment sets APOLLO_KEY, or any other secret, as a literal pod-spec env value rather than through a Kubernetes Secret, inspect your bundle before sharing it. The pod spec is collected in full, literal values included, so a secret set this way is only protected by a redaction rule that masks any env var named *_KEY or *_PASS. This redaction rule is a safety net, not the structural isolation a Secret-backed APOLLO_KEY gets.