Skip to content

docs: ate.workerpool.workers is reported by every ateapi replica - #3056

Merged
Jeff Luo (JeffLuoo) merged 2 commits into
agent-substrate:mainfrom
git286:docs-workerpool-workers-replicas
Oct 21, 2026
Merged

Jeff Luo (JeffLuoo) merged 2 commits into
agent-substrate:mainfrom
git286:docs-workerpool-workers-replicas

Conversation

@git286

@git286 Da Huang (git286) commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Each ateapi replica reports the whole fleet on ate.workerpool.workers, and the default install runs two, so a sum across replicas counts every worker twice. The registry told readers they could add the counts together. Say that a sum across replicas double counts, and to take the max across replicas by pool namespace, pool, state and sandbox class before adding.

To mitigate #3055

  • Tests pass
  • Docs updated

Each ateapi replica reports the whole fleet on ate.workerpool.workers,
and the default install runs two, so a sum across replicas counts every
worker twice. The registry told readers they could add the counts
together. Say that a sum across replicas double counts, and to take the
max across replicas by pool namespace, pool, state and sandbox class
before adding.

Refs agent-substrate#3055

@JeffLuoo Jeff Luo (JeffLuoo) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this mitigation. I checked the claim against RegisterWorkerCount and the ate-api-server manifest, and it is correct. I have one comment on the table row in observability.md.

Comment thread docs/observability.md Outdated
| `ate.workerpool.ready_workers` | atecontroller | up/down counter | number of worker pods currently ready for a WorkerPool, from `status.readyReplicas` (labels
`ate.workerpool.namespace`, `ate.workerpool.name`) |
| `ate.workerpool.workers` | ateapi | up/down counter | live worker count per pool, split by state (`idle`/`partial`/`at_capacity`/`unschedulable`) and sandbox class to provide fleet capacity and saturation at a glance |
| `ate.workerpool.workers` | ateapi | up/down counter | live worker count per pool, split by state (`idle`/`partial`/`at_capacity`/`unschedulable`) and sandbox class to provide fleet capacity and saturation at a glance. Every ateapi replica reports the whole fleet, so take the `max` across replicas before you `sum` ([#3055](https://github.com/agent-substrate/substrate/issues/3055)) |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please name the grouping here, as the registry note does. "Take the max across replicas before you sum" does not say which labels to keep. A literal sum(max(...)) returns the largest single series, not the fleet. Suggested text: "Every ateapi replica reports the whole fleet. Take the max by pool namespace, pool, state, and sandbox class first, then sum." An example query helps more than prose:

sum by ("ate.workerpool.namespace", "ate.workerpool.name") (
  max by ("ate.workerpool.namespace", "ate.workerpool.name", "ate.worker.state", "ate.sandbox.class") (
    {"ate.workerpool.workers"}
  )
)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks. Fixed in 278ba7d. The row now names the grouping, and I added your query in a note below the table (a code block cannot go in a table cell). I also added a line to keep cluster in both groupings when one backend gets several clusters, otherwise same-named pools merge. I ran the query on a GKE cluster: it gives 3 for a 3-worker pool, while the plain sum gives 6.

The table row said to take the max across replicas before the sum but
did not say which labels to keep, and a literal sum(max(...)) returns
the largest single series. Name the grouping in the row and add a note
below the table with a per-pool query, and say to add cluster when one
backend receives several clusters.
@JeffLuoo
Jeff Luo (JeffLuoo) added this pull request to the merge queue Oct 21, 2026
Merged via the queue into agent-substrate:main with commit 1d42774 Oct 21, 2026
15 checks passed
@JeffLuoo Jeff Luo (JeffLuoo) added kind/docs Improvements or additions to documentation area/observability labels Oct 21, 2026
Davanum Srinivas (dims) pushed a commit to dims/substrate that referenced this pull request Oct 29, 2026
…gent-substrate#3193)

The aggregation note from agent-substrate#3056 tells readers to take the `max` across
ate-api-server replicas and then `sum`. As agent-substrate#3162 shows, the replicas
read separate worker caches and can disagree for a few seconds: one
reports `{ate_worker_state="partial"} 3` while the other still reports
`{ate_worker_state="idle"} 3`, so `max by (ate_worker_state)` keeps both
series and the pool sums to 6 workers.

Take the average across replicas instead. It keeps the sum across the
states equal to the pool size; the counts can be fractional for those
few seconds. Updates the registry note and the observability guide. No
code change: the locust collector already reads a single process's
series, and nothing else aggregates this instrument.

Fixes agent-substrate#3162

AI use: Claude Code analyzed the issue and wrote the change and this
description. I reviewed the diff. Codex rebased the branch onto main and
resolved the documentation conflicts.

---------

Signed-off-by: cloudonly <enrondg@gmail.com>
Dmitry Berkovich (dberkov) pushed a commit that referenced this pull request Nov 2, 2026
…3193)

The aggregation note from #3056 tells readers to take the `max` across
ate-api-server replicas and then `sum`. As #3162 shows, the replicas
read separate worker caches and can disagree for a few seconds: one
reports `{ate_worker_state="partial"} 3` while the other still reports
`{ate_worker_state="idle"} 3`, so `max by (ate_worker_state)` keeps both
series and the pool sums to 6 workers.

Take the average across replicas instead. It keeps the sum across the
states equal to the pool size; the counts can be fractional for those
few seconds. Updates the registry note and the observability guide. No
code change: the locust collector already reads a single process's
series, and nothing else aggregates this instrument.

Fixes #3162

AI use: Claude Code analyzed the issue and wrote the change and this
description. I reviewed the diff. Codex rebased the branch onto main and
resolved the documentation conflicts.

---------

Signed-off-by: cloudonly <enrondg@gmail.com>
(cherry picked from commit 6949841)
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/observability kind/docs Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants