Repository navigation
fix: count the state once per request in the context-limit guard - #16
Merged
Merged
Conversation
jev's limits are per request (the state plus all questions) and per question (the state plus one question). The guard added the state to every question and summed those, so a request with many questions over a long state was counted many times over: 37 short questions over a 5,000-token state came to 185,185 tokens and were refused with a 400 at the 64,000 cap, where jev counts about 5,200. The per-question check is unchanged. The per-request total is now the state once plus the questions. A request that really exceeds the cap is still refused. Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A request with many questions over a long state was refused with a 400 even when it was well under the limit. For example, 37 short questions over a 5,000-token state are about 5,200 tokens, but the guard counted 185,185 and rejected them at the 64,000-token cap. After this change they are accepted.
jev's limits are per request (the state plus all questions) and per question (the state plus that one question).
_enforce_context_limitsadded the state to every question and summed the results, so the state was counted once per question. The per-question check was already right and is unchanged. The per-request total is now the state once plus the questions.Callers affected are those sending many questions over a long state. Requests that really exceed the cap are still rejected, with the same message.
Tested:
pytest: 228 passed, 4 skipped. Two new tests: 37 questions over a 5,000-token state are accepted (this one failed on the old code with the 185,185 figure), and a request of 70,000 real tokens is still rejected.ruff checkandruff format --checkpass.Not covered: jev's own service was not called to confirm its counting rule. The rule comes from the wording on its models page ("the
stateplus all questions combined").