Skip to content

fix: count the state once per request in the context-limit guard - #16

Merged
planetf1 merged 1 commit into
mainfrom
fix/count-state-once-per-request
Oct 25, 2026
Merged

planetf1 merged 1 commit into
mainfrom
fix/count-state-once-per-request

Conversation

@planetf1

Copy link
Copy Markdown
Collaborator

A request with many questions over a long state was refused with a 400 even when it was well under the limit. For example, 37 short questions over a 5,000-token state are about 5,200 tokens, but the guard counted 185,185 and rejected them at the 64,000-token cap. After this change they are accepted.

jev's limits are per request (the state plus all questions) and per question (the state plus that one question). _enforce_context_limits added the state to every question and summed the results, so the state was counted once per question. The per-question check was already right and is unchanged. The per-request total is now the state once plus the questions.

Callers affected are those sending many questions over a long state. Requests that really exceed the cap are still rejected, with the same message.

Tested:

  • pytest: 228 passed, 4 skipped. Two new tests: 37 questions over a 5,000-token state are accepted (this one failed on the old code with the 185,185 figure), and a request of 70,000 real tokens is still rejected.
  • ruff check and ruff format --check pass.
  • Counted with a real tokenizer, 25 requests from a public benchmark suite that the old guard refused are all accepted by the new one. Whether the engine then answers them well is checked separately.

Not covered: jev's own service was not called to confirm its counting rule. The rule comes from the wording on its models page ("the state plus all questions combined").

jev's limits are per request (the state plus all questions) and per question (the state plus
one question). The guard added the state to every question and summed those, so a request
with many questions over a long state was counted many times over: 37 short questions over a
5,000-token state came to 185,185 tokens and were refused with a 400 at the 64,000 cap, where
jev counts about 5,200.

The per-question check is unchanged. The per-request total is now the state once plus the
questions. A request that really exceeds the cap is still refused.

Assisted-by: Claude Code
Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
@planetf1
planetf1 marked this pull request as ready for review October 25, 2026 23:02
@planetf1
planetf1 merged commit 54b6383 into main Oct 25, 2026
3 checks passed
@planetf1
planetf1 deleted the fix/count-state-once-per-request branch October 25, 2026 23:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant