Add an internal ateapi statusz page

[Adhita Selvaraj (swiftdiaries)]

Description

• Added a statusz page for the ateapi
• Used the existing ate-router page as reference

Addresses #1458

Review Notes

Things I'm not sure about:

• I'm not aggregating metrics across replicas, is this needed?
• On the metrics themselves, could I please get a review on what's needed/how they're represented?

Screenshot

---

AI-generated PR description

Adds an internal ateapi /statusz page for inspecting the serving replica. HTML and JSON expose build/version, uptime, live readiness, listener/drain settings, and safely redacted startup configuration, alongside known workers grouped by namespace/pool, PostgreSQL client-pool occupancy, and recent failed authenticated unary Control RPCs.

The dedicated --status-port listener defaults to 4040, can be disabled with a nonpositive value, and stays available through gRPC drain. It is exposed through the internal API Service. Configuration uses a closed display policy to redact credentials, environment-resolved values, and sensitive paths.

Each request reads diagnostics directly from existing in-memory readers; there is no diagnostic response cache or background sampler. The browser offers Manual / 5s / 10s default / 30s full-page refresh, matching the router. The selection resets to 10 seconds on each page load.

Worker counts reflect this replica's cached view of shared inventory and must not be summed across replicas; the page displays at most 100 groups with totals and truncation information. Pool statistics perform no SQL, connection acquisition, or connectivity probe. Failure history retains the newest 100 matching completions with bounded method/principal text, status, and elapsed time, without payloads, credentials, or raw errors. It resets on restart and is not an error-rate metric.

Includes both the endpoint foundation and runtime diagnostics.

Validation: focused ateapi/statusz race tests passed on the combined head a1b96d9f302766f4f744693e06bd442a2c57da3f; diff checks passed. Earlier validation covered PostgreSQL-backed pool tests, browser refresh behavior, GolangCI-Lint, kube-api-linter, and applicable static checks. The full make verify run encountered inherited color/telemetry environment failures; those affected packages passed when rerun without the conflicting variables. Some static wrappers required a clean worktree and were not run during the uncommitted refactor. No single full-verification pass, live-cluster deployment, or E2E validation is claimed.

[Adhita Selvaraj (swiftdiaries)]

cc Da Huang (@git286)

[Bowei Du (bowei)]

+1 I love statusz pages :-)

[Adhita Selvaraj (swiftdiaries)]

Bowei Du (@bowei) haiyanmeng added a statusz page based on the issue description. Generated a screenshot with some synthetic seed data to visualize what it'd look like.

Could I please a get review on the representation of the metrics and if replica-wide aggregation is needed here? Tried to keep it simple for the first pass.

[Da Huang (git286)]

The page is served by a new HTTP listener on 4040. It is separate from the gRPC server, so the auth interceptors on 443 do not apply to it. It binds to all interfaces and the manifest also puts 4040 on the api Service, so any pod in any namespace can open http://api.ate-system.svc:4040/statusz?format=json.

Even with redaction, the page still shows build version and commit, listener addresses, flag names, PG pool counts, and the last 100 failed Control RPCs. For a platform that runs untrusted agent workloads, I think this is too much to expose with no check.

I suggest closing the network path instead of adding more redaction:

• Bind the status listener to 127.0.0.1:4040. kubectl port-forward still works. The router already does this for ext_proc
• Remove the status port from the api Service. The documented workflow only uses pod port-forward, so nothing needs it.
• Update docs/observability.md. "Only exposed through the cluster-internal Service" includes every tenant pod, so it reads safer than it is.

With this, access control becomes Kubernetes RBAC (pods/portforward), which is the same model kube-apiserver uses for its own /statusz and /flagz (KEP-4827, KEP-4828). If we later need an in-cluster reader, that should be a separate change behind the existing authenticated listener, not 4040 on the Service.

Please keep the redaction anyway. It is still useful when people paste the page into tickets or chat.

[Adhita Selvaraj (swiftdiaries)]

addressed review comments and rebased, could you please re-review? Da Huang (@git286)