Repository navigation
ate.workerpool.workers is double counted when ate-api-server runs more than one replica #3055
Description
Every ate-api-server replica reports ate.workerpool.workers for the whole fleet. The default install runs 2 replicas, so anything that adds the metric up, like a sum(), a dashboard, or an alert, shows twice the real number of workers.
On a default GKE install (v0.4.0-gke.0) with one pool of 3 workers:
| Query | Result |
|---|---|
kubectl get workerpool counter |
3 desired, 3 ready |
sum by (instance) ({"ate.workerpool.workers", cluster="<cluster>"}) |
3 from each ate-api-server pod |
sum({"ate.workerpool.workers", cluster="<cluster>"}) |
6 |
Possible fixes
- Report the metric from one place only: atecontroller, which already reports
desired_workers/ready_workers, or the API server leader. - Or document that users must de-duplicate the metric, e.g.
max by ("ate.workerpool.name", "ate.worker.state") (...).
Activity
docs: ate.workerpool.workers is reported by every ateapi replica
git286 commentedon Oct 21, 2026
cc Julian Gutierrez Oschmann (@juli4n)
juli4n commentedon Oct 21, 2026
I think this is WAI. Each
ate-apiserverreplica keeps a full view of the workers and reports the whole fleet as a metric. The metrics hasinstance(andpod) labels so each replica's metric series can by grouped together.This is not different than how kube-apiserver works. Every replica in an HPA setup reports the object counts for the whole cluster.
That being said, I agree it's worth a paragraph in the documentation about observability / otel.
docs: ate.workerpool.workers is reported by every ateapi replica (age…