ate-api-server: add /statusz (feature parity with atenet router, KEP-4827 convention) #1458

Summary

Feature request: give ate-api-server a /statusz endpoint following the pattern the atenet router already ships (--status-port, default 4040, see cmd/atenet/internal/router/status.go). Today the two control-plane components are inconsistent: the router serves a rich HTML status dashboard, while ateapi has nothing on 4040 (connection refused) and its only HTTP port (9090) serves just /metrics and /healthz.

This is also the direction upstream Kubernetes is taking for its own control-plane components via KEP-4827 component statusz (with /flagz in KEP-4828), so following it here means adopting an emerging convention rather than inventing one.

Motivation — three real debugging sessions that needed it

During a bug bash on a live GKE install (substrate rev 0d85e85), every ateapi-side incident required log spelunking that a status page would have shortcut:

1. Wedged actors after template deletion. Deleting an ActorTemplate with live actors left them unable to suspend ("actor template not found") or delete (FailedPrecondition). ateapi's own metrics counted these failures (ate_actor_lifecycle_operation_duration_seconds{error_type="FailedPrecondition"}), but there was no place to see recent failing RPCs. The router's "Recent 100 Processing Inquiries" table is exactly the right pattern; ateapi needs the control-plane equivalent.
2. Silent golden-snapshot failure. A template with an unpullable image (MANIFEST_UNKNOWN) retried AteomHerder/Run every ~20s for 45 minutes while kubectl ate get actor-template showed an empty ERROR column and status: {}. A statusz section for in-flight template/golden work would have surfaced the retry loop and its last error immediately.
3. Postgres invisibility. DB behavior is currently invisible in traces (ateapi: postgres store has no trace instrumentation - DB time is invisible in every trace #1455) and there is no page showing pool utilization, connectivity, or outbox depth — the two gaps compound.

The router page also proved its worth in the same bash: an operator flag change (--route-timeout=180s) was immediately visible under "Global Arguments & Configuration".

Suggested content

• Build revision, uptime, resolved flags (KEP-4827/4828 style)
• PostgreSQL: pool stats (in-use/idle/max), connectivity health check with ok/err counters + last-known-good, outbox depth/lag
• Worker cache: known workers per pool, eligible-worker counts
• Recent N control-plane RPCs with method, caller identity, status, latency (router-table analogue), or at minimum recent non-OK RPCs
• Golden-snapshot / template work in flight, with last error per template
• Drain state (--drain-delay etc. are already flags)

Two cautions

1. Redaction: the router page dumps full os.Args and resolved flags verbatim. ateapi's flags include --postgres-connection-string (password-free on GKE installs thanks to mTLS cert auth, but other deployments may embed credentials in it). Connection strings and env-sourced values must be redacted on the page.
2. Separate port: like the router, serve it on a dedicated --status-port, cluster-internal only — never on the 443 gRPC listener.

Related

• atenet router /statusz: Service IP panel is forbidden by the router's own shipped RBAC #1457 — the router's existing statusz has an RBAC bug in its Service IP panel; worth keeping in mind so a shared status-page framework (extracting the router's implementation into an internal package) doesn't inherit it.
• ateapi: postgres store has no trace instrumentation - DB time is invisible in every trace #1455 — postgres trace instrumentation gap; the DB section above is the status-page half of the same visibility problem.