Repository navigation
server: fix the off-by-two in the SWA reuse threshold - #30580
Merged
Merged
Conversation
The first evaluated token attends to the positions above pos - n_swa, so a SWA state starting at pos - n_swa + 1 holds the whole window, but the reuse threshold asked for two more cells. A saved SWA state keeps only the cells inside its window, so a state read back from a slot file or from the RAM prompt cache was always rejected and the slot rolled back to a checkpoint. The SWA threshold now matches the window, while a state without SWA keeps the exact bound. The test harness gains override_kv, used to give the SWA test model a window smaller than the test context.
Contributor
|
@ggerganov isn't this the same I mentioned to you earlier? i.e. SWA state-restore is not bit-exact |
Member
|
No, the thing we discussed was about producing the same logits. This here is about reusing a valid saved state in a situation that |
ggerganov
approved these changes
Oct 15, 2026
ggerganov
added
the
merge ready
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Found on master while testing #30574: after a slot restore on Gemma 4, the next turn reprocessed 71 tokens instead of 23, even though the restored state holds everything the turn needs.
Since #23981 a saved SWA state keeps only the cells inside its window, so a state read back from a slot file or from the RAM prompt cache starts at pos_max - n_swa + 1. The first evaluated token attends to the positions above pos - n_swa, so this is enough, but the reuse threshold asks for pos_min < pos - n_swa, two cells more than the window. The state is rejected and the slot rolls back to the previous checkpoint, which can be far behind or missing.
The SWA threshold now matches the window. A state without SWA keeps the exact bound, since the recurrent state must end right before the evaluated token.
Additional information
Gemma 4 E4B, tokens reprocessed on the turn after the state comes back:
test_slot_restore_swa_window and test_prompt_cache_swa_window give tinygemma3 a 64 token window through the new override_kv harness field, and fail without this change. The rest of the server suite has the same results as master.
Requirements