Skip to content

Router Envoy logs request credentials #2205

Closed

Description

@git286

The router's Envoy writes request secrets to its container log. On GKE this goes to Cloud Logging and stays there for 30 days. Anyone with log read access on the project can see it.

Two ways it leaks:

  1. Debug logging is on in the base manifest. manifests/ate-install/atenet-router.yaml starts Envoy with --component-log-level upstream:debug,router:debug,ext_proc:debug. At this level Envoy dumps every request header for every request, so Authorization and Cookie go into the log as-is. This is not a kind-only setting. It ships in the base install.

  2. The access log prints the query string. The router's HCMs use StdoutAccessLog{} with no format (cmd/atenet/internal/router/xds.go). Envoy's default format logs the full path, including the query string. So ?api_key=... and similar tokens are logged even with debug off.

Repro on kind with main:

kubectl -n ate-system port-forward svc/atenet-router 18099:80 &
curl -X POST "http://localhost:18099/?api_key=QUERYSECRET" \
  -H "ate-target-actor: <atespace>/<running actor>" \
  -H "Authorization: Bearer FAKE-TOKEN" \
  -H "Cookie: session=FAKECOOKIE" -d '{}'
kubectl -n ate-system logs deploy/atenet-router -c envoy | grep -E 'FAKE-TOKEN|FAKECOOKIE|QUERYSECRET'

All three show up in the log.

The debug flag also makes the log noisy. On an almost idle GKE cluster the router's Envoy writes more than 1000 debug lines an hour.

Nothing in the repo reads these debug lines. The flag came in with #711 and looks like it was left over from local debugging.

Proposed fix:

  • Drop --component-log-level from the router manifest. Envoy goes back to info. If someone needs debug, they can turn it on for a while through the admin port with POST /logging?router=debug. No restart needed.
  • Set an explicit access log format: Envoy's default, but with %PATH(NQ:ORIG_OR_PATH)% so the query is never logged.
  • Add a test that every router HCM uses that format.

Related: #103 (log sanitization), #1743 (proto redaction).

Activity

  1. added this to the M3 milestone on Oct 5, 2026
  2. added a commit that references this issue on Oct 6, 2026
    2806695
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions