Repository navigation
docs: average ate.workerpool.workers across replicas before summing - #3193
Merged
Jeff Luo (JeffLuoo) merged 2 commits intoOct 29, 2026
Merged
Jeff Luo (JeffLuoo) merged 2 commits into
Jeff Luo (JeffLuoo) merged 2 commits into
Conversation
Da Huang (git286)
approved these changes
Oct 27, 2026
Jeff Luo (JeffLuoo)
previously approved these changes
Oct 27, 2026
Jeff Luo (JeffLuoo)
enabled auto-merge
October 27, 2026 13:57
Bowei Du (bowei)
added
area/observability
kind/docs
Contributor
Author
|
The only failing check is Could a maintainer re-run the failed job and merge? |
Each replica reads its own worker cache, and the caches can disagree for a few seconds. A worker that changed state then sits under its old state on one replica and its new state on the other, so max by state counts it twice and a summed pool outgrows its real size. avg keeps the sum across states equal to the pool size. Signed-off-by: cloudonly <enrondg@gmail.com>
Shorten the metrics table row back to a pointer at the example query, and wrap the long line in the registry note. Signed-off-by: cloudonly <enrondg@gmail.com>
auto-merge was automatically disabled
October 29, 2026 05:04
Head branch was pushed to by a user without write access
Luomu (cloudonly)
force-pushed
the
docs-workerpool-workers-avg
branch
from
October 29, 2026 05:04
59a4989 to
373e884
Compare
Jeff Luo (JeffLuoo)
approved these changes
Oct 29, 2026
1 of 2 tasks
Dmitry Berkovich (dberkov)
pushed a commit
that referenced
this pull request
Nov 2, 2026
…3193) The aggregation note from #3056 tells readers to take the `max` across ate-api-server replicas and then `sum`. As #3162 shows, the replicas read separate worker caches and can disagree for a few seconds: one reports `{ate_worker_state="partial"} 3` while the other still reports `{ate_worker_state="idle"} 3`, so `max by (ate_worker_state)` keeps both series and the pool sums to 6 workers. Take the average across replicas instead. It keeps the sum across the states equal to the pool size; the counts can be fractional for those few seconds. Updates the registry note and the observability guide. No code change: the locust collector already reads a single process's series, and nothing else aggregates this instrument. Fixes #3162 AI use: Claude Code analyzed the issue and wrote the change and this description. I reviewed the diff. Codex rebased the branch onto main and resolved the documentation conflicts. --------- Signed-off-by: cloudonly <enrondg@gmail.com> (cherry picked from commit 6949841) Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The aggregation note from #3056 tells readers to take the
maxacross ate-api-server replicas and thensum. As #3162 shows, the replicas read separate worker caches and can disagree for a few seconds: one reports{ate_worker_state="partial"} 3while the other still reports{ate_worker_state="idle"} 3, somax by (ate_worker_state)keeps both series and the pool sums to 6 workers.Take the average across replicas instead. It keeps the sum across the states equal to the pool size; the counts can be fractional for those few seconds. Updates the registry note and the observability guide. No code change: the locust collector already reads a single process's series, and nothing else aggregates this instrument.
Fixes #3162
AI use: Claude Code analyzed the issue and wrote the change and this description. I reviewed the diff. Codex rebased the branch onto main and resolved the documentation conflicts.