Repository navigation
Conversation
The row-count sort of a fully resident layer's experts skipped native-layout packs, though slot/src/off are built the same way for both layouts. 4x GP100 (sm_60): 1.2K prompt 7.27 -> 5.32 s, 35K 63.3 -> 54.1 s, the same bits. The switch stays opt-in.
This was referenced Oct 9, 2026
Maddoxkkm
added a commit
to Maddoxkkm/Strata
that referenced
this pull request
Oct 10, 2026
…1221#1520 (release staged pages), Niko1221#1107 (grouped gather on short prompts) Three of the candidates we read apply and are not superseded by upstream: - Niko1221#1660 lets the MMQ resident sort take native-layout packs (slot/src/off are built the same way for both layouts; the PR measures 1.2K prompt 7.27 -> 5.32 s and 35K 63.3 -> 54.1 s on 4xGP100, same bits). - Niko1221#1520 extends STRATA_FILE_RELEASE to prefill staging copies (opt-in; the PR's own numbers: +11.7 GiB available RAM for 3.7-4.4% prefill - a trade worth testing on this 32 GB box). - Niko1221#1107 stages the short-prompt walk through a grouped ring (STAGE_GRP=32) instead of one slot at a time. Not applied, and why: Niko1221#1101 and Niko1221#1237 are already upstream in newer form (upstream has the stager sleep plus a stager_sleep helper and a batched copy_blobs loop, and pinned stage buffers behind STAGE_PIN_DEFAULT=0 - the PR's older always-pin revision would re-arm the IQ3_S decode corruption the release gate found).
Maddoxkkm
added a commit
to Maddoxkkm/Strata
that referenced
this pull request
Oct 10, 2026
…batch Niko1221#1416 is superseded by upstream 0.1.41 (which carries STRATA_PREFILL_CPU_SHARE_MAX and a newer two-argument arm_cpu_share the PR lacks); Niko1221#1101 and Niko1221#1237 are upstream already, and Niko1221#1237's older always-pin revision would re-arm the IQ3_S decode corruption the 0.1.41 gate found. Applied instead: Niko1221#1660 (native-layout MMQ resident sort), Niko1221#1520 (release staged pages, opt-in) and Niko1221#1107 (grouped gather on short prompts) - +4.7% prefill at 4K, +4.3% at 16K, with STRATA_FILE_RELEASE measured at -6.3% and left off. Adds pr-triage.py, pr-check.py, the report and the cached candidate diffs.
Owner
|
Ported with your authorship (still opt-in). On a 3060 with IQ3_XXS the output with STRATA_MMQ_RESIDENT_SORT_NE=1 is bit identical at 4096 and 20000 tokens, but no layer there is fully resident so we could not see the speedup; your 4xGP100 numbers stand as the evidence. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The row-count sort of a fully resident layer's experts (c6a0d40) skips native-layout packs, though slot/src/off are built the same way for both layouts and nothing reads off[e + 1] as a count. This drops that one condition; the switch stays opt-in.
Measured on 4× GP100 (2× PH402, sm_60), Swift 1.5 IQ3_XXS served native with Q8_0 dense shards,
--layer-split 12,24,36, every expert resident. Switch off → on, the same build, replies identical bit for bit:The expert GEMMs take 44% less time; nothing else moves, and decode is unchanged. Only sm_60 was checked for identical output; whether newer architectures' MMQ (stream-k) keeps the bits under the reordered groups is not measured.