Repository navigation
docs: ate.workerpool.workers is reported by every ateapi replica - #3056
Jeff Luo (JeffLuoo) merged 2 commits into
Conversation
Each ateapi replica reports the whole fleet on ate.workerpool.workers, and the default install runs two, so a sum across replicas counts every worker twice. The registry told readers they could add the counts together. Say that a sum across replicas double counts, and to take the max across replicas by pool namespace, pool, state and sandbox class before adding. Refs agent-substrate#3055
Jeff Luo (JeffLuoo)
left a comment
There was a problem hiding this comment.
Thanks for this mitigation. I checked the claim against RegisterWorkerCount and the ate-api-server manifest, and it is correct. I have one comment on the table row in observability.md.
| | `ate.workerpool.ready_workers` | atecontroller | up/down counter | number of worker pods currently ready for a WorkerPool, from `status.readyReplicas` (labels | ||
| `ate.workerpool.namespace`, `ate.workerpool.name`) | | ||
| | `ate.workerpool.workers` | ateapi | up/down counter | live worker count per pool, split by state (`idle`/`partial`/`at_capacity`/`unschedulable`) and sandbox class to provide fleet capacity and saturation at a glance | | ||
| | `ate.workerpool.workers` | ateapi | up/down counter | live worker count per pool, split by state (`idle`/`partial`/`at_capacity`/`unschedulable`) and sandbox class to provide fleet capacity and saturation at a glance. Every ateapi replica reports the whole fleet, so take the `max` across replicas before you `sum` ([#3055](https://github.com/agent-substrate/substrate/issues/3055)) | |
There was a problem hiding this comment.
Please name the grouping here, as the registry note does. "Take the max across replicas before you sum" does not say which labels to keep. A literal sum(max(...)) returns the largest single series, not the fleet. Suggested text: "Every ateapi replica reports the whole fleet. Take the max by pool namespace, pool, state, and sandbox class first, then sum." An example query helps more than prose:
sum by ("ate.workerpool.namespace", "ate.workerpool.name") (
max by ("ate.workerpool.namespace", "ate.workerpool.name", "ate.worker.state", "ate.sandbox.class") (
{"ate.workerpool.workers"}
)
)
There was a problem hiding this comment.
Good catch, thanks. Fixed in 278ba7d. The row now names the grouping, and I added your query in a note below the table (a code block cannot go in a table cell). I also added a line to keep cluster in both groupings when one backend gets several clusters, otherwise same-named pools merge. I ran the query on a GKE cluster: it gives 3 for a 3-worker pool, while the plain sum gives 6.
The table row said to take the max across replicas before the sum but did not say which labels to keep, and a literal sum(max(...)) returns the largest single series. Name the grouping in the row and add a note below the table with a per-pool query, and say to add cluster when one backend receives several clusters.
…gent-substrate#3193) The aggregation note from agent-substrate#3056 tells readers to take the `max` across ate-api-server replicas and then `sum`. As agent-substrate#3162 shows, the replicas read separate worker caches and can disagree for a few seconds: one reports `{ate_worker_state="partial"} 3` while the other still reports `{ate_worker_state="idle"} 3`, so `max by (ate_worker_state)` keeps both series and the pool sums to 6 workers. Take the average across replicas instead. It keeps the sum across the states equal to the pool size; the counts can be fractional for those few seconds. Updates the registry note and the observability guide. No code change: the locust collector already reads a single process's series, and nothing else aggregates this instrument. Fixes agent-substrate#3162 AI use: Claude Code analyzed the issue and wrote the change and this description. I reviewed the diff. Codex rebased the branch onto main and resolved the documentation conflicts. --------- Signed-off-by: cloudonly <enrondg@gmail.com>
…3193) The aggregation note from #3056 tells readers to take the `max` across ate-api-server replicas and then `sum`. As #3162 shows, the replicas read separate worker caches and can disagree for a few seconds: one reports `{ate_worker_state="partial"} 3` while the other still reports `{ate_worker_state="idle"} 3`, so `max by (ate_worker_state)` keeps both series and the pool sums to 6 workers. Take the average across replicas instead. It keeps the sum across the states equal to the pool size; the counts can be fractional for those few seconds. Updates the registry note and the observability guide. No code change: the locust collector already reads a single process's series, and nothing else aggregates this instrument. Fixes #3162 AI use: Claude Code analyzed the issue and wrote the change and this description. I reviewed the diff. Codex rebased the branch onto main and resolved the documentation conflicts. --------- Signed-off-by: cloudonly <enrondg@gmail.com> (cherry picked from commit 6949841) Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Each ateapi replica reports the whole fleet on ate.workerpool.workers, and the default install runs two, so a sum across replicas counts every worker twice. The registry told readers they could add the counts together. Say that a sum across replicas double counts, and to take the max across replicas by pool namespace, pool, state and sandbox class before adding.
To mitigate #3055