Skip to content

prefill: use active QSA top-k bounds on Turing with large KV capacity - #512

Closed
imanu86 wants to merge 2 commits into
Niko1221:mainfrom
imanu86:prefill-topk-active-bound
Closed

imanu86 wants to merge 2 commits into
Niko1221:mainfrom
imanu86:prefill-topk-active-bound

Conversation

@imanu86

@imanu86 imanu86 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

With --max-context 262144, QSA score rows have a stride of 65538 blocks. A prefill batch at 131K actual context fits the register top-k kernel's 33792-block limit, but CUDA dispatch currently tests the allocated capacity and selects the reference kernel.

This enables the already supplied active_blocks bound only on sm_75. It keeps the score-row stride, selection arithmetic and kernel bodies unchanged. Other CUDA architectures retain the capacity rule, including the RTX 5070 case documented in the current source as a 1–3% regression. HIP retains its existing policy. Decode omits the bound, so captured graphs keep the capacity rule as their context grows. Invalid bounds and contexts beyond the register limit fall back to the existing dispatch. Presence of STRATA_TOPK_CAPACITY_GUARD restores the CUDA capacity rule for A/B runs.

The change reuses upstream's existing API and prefill caller rather than importing the separate API from our 0.1.31 fork. A CUDA parity target covers 16 cases with stride 65538: partial tails, zero/tied scores, batch sizes 1/8/255/256/257, invalid bounds, and both sides of the 33792-block limit.

Retained measurements on our previous daily

Modified RTX 2080 Ti 22 GB, Ryzen 7 5800X, 96 GB RAM, Windows, CUDA 12.6, IQ3_XXS, KV262144/int8/resident32768, automatic elastic expert cache and MTP. Same-executable capacity/active/capacity A/B/A, two 1024-token outputs per arm; fresh prompt131248, second prompt131239 with131072 reused.

Dispatch Fresh prefill tok/s TTFT seconds Aggregate decode tok/s
Capacity A1 831.20 158.509 38.251
Active B 886.38 148.664 39.272
Capacity A2 828.26 159.092 39.293

Prefill +6.83% relative to the mean reference; reference spread0.35%. TTFT saves10.14s. No decode gain attributed to this change. The kernel fixture passed16 bitwise selected-ID cases. At8192 context the register kernel was slower in the microbenchmark; no universal short-prompt improvement is claimed. Elastic slots and MTP output varied, and whole-model logits were not bitwise even between the reference arms.

Full measurement record, fixture cases and limitations.

Validation status of this upstream port

Updated against upstream 36fa455e579b23a9c909c2c6fe1bddd9e51cb8ca (0.1.36), preserving the new sm_90+ cluster dispatch. The branch contains a merge of upstream main without rewriting its published history. The CUDA port and harness compile with CUDA 12.6 for sm_75, and all 16 selected-ID parity cases pass bitwise on the modified RTX 2080 Ti. The full SM75 fork engine build and its CMake-built top-k harness now pass; the fork contains additional changes and its model timings are not attributed to this narrow PR. The harness also passes MSVC C++17 syntax-only compilation; the existing-file diffs pass the whitespace check. The measurements above belong to the separately validated 0.1.31 daily, not this rebased tree. Keep this PR draft until an isolated full-engine model A/B/A has been repeated on this upstream base. No installer, workflow, elastic-cache patch or other fork optimization is included.

imanu86 and others added 2 commits October 2, 2026 18:44
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Niko1221 pushed a commit that referenced this pull request Oct 3, 2026
With --max-context 262144 the QSA score rows have a stride of 65538 blocks; a prefill batch at 131K actual context
fits the register top-k kernel's 33792-block limit, but CUDA dispatch tested the allocated capacity and chose the
reference kernel. The supplied active_blocks bound is now used on sm_75 only; other CUDA cards keep the capacity
rule, HIP its own policy, and decode (no bound) is unchanged. STRATA_TOPK_CAPACITY_GUARD restores the capacity rule.
qsa_topk_active_parity: 16 cases at stride 65538, selected ids bitwise.

Squashed from #512 (its commit and its merge of upstream 0.1.36).

Co-Authored-By: Codex <noreply@openai.com>
Niko1221 added a commit that referenced this pull request Oct 3, 2026
…ard (#512 follow-up)

The engine takes the bound on Turing only, so on other cards the test compared the capacity rule with itself.
STRATA_TOPK_ACTIVE_ANY=1 (set by the test) takes the Turing dispatch on any card; the kernels are the same on every
architecture. The CMake block moves above the qsa_select_bench comment it had split.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
@Niko1221

Niko1221 commented Oct 3, 2026

Copy link
Copy Markdown
Owner

Released in 0.1.38 (squashed; the merge commit left out), Turing only as proposed. We added STRATA_TOPK_ACTIVE_ANY=1, which the parity test sets, so the active-bound path is checked on any CUDA card (it passes on an RTX 5070). Merged under your name (GitHub doesn't show it as merged because it went in by hand). Thank you!

@Niko1221 Niko1221 closed this Oct 3, 2026
imanu86 pushed a commit to imanu86/Strata-2080Ti that referenced this pull request Oct 3, 2026
…ds with a histogram per warp (CUDA) - 9-12x at 262K-524K cells, a 243K-token prompt +26% on an RTX 3060

On CUDA the register kernel holds 33,792 blocks (~135K cells). Past it qsa_block_topk took the original kernel:
256 threads, each reading its own stretch of the scores on all six passes, one shared histogram updated with
atomics. The dispatch follows the capacity (--max-context), so with a context past ~135K that was every prompt
batch and every decode window on a card without the sm_75 active-bound dispatch (Niko1221#512), whatever the prompt's
length, and on sm_75 every prompt batch past 33,792 blocks and every decode window.

block_topk_wide_kernel is the register kernel's layout (1,024 threads, a histogram per warp, warp scans) with the
keys read from memory:
- the four radix passes: thread t reads blocks t, t + 1024, ... - a histogram has no order, so a warp reads 32
  neighbours at a time;
- the count and the output: warp w holds `per` rows of 32 consecutive blocks, and the cells are written in the
  order (warp, row, lane); a row without a selected block is skipped.
The selection rule is the original kernel's (radix threshold, ties to the lowest index, cells ascending): identical
ids. One launch per call, no scratch memory, no capacity limit.

Dispatch (CUDA; HIP is unchanged and does not compile the kernel): where the blocks do not fit the registers the
new kernel, except a call that names fewer than 4,608 blocks (a prompt batch below ~18K cells), which keeps the
original kernel: its fixed cost is lower there. A decode window (no block count) takes the new kernel at any
context. The register kernel's range, the sm_75 active-bound dispatch and the sm_90+ cluster path are unchanged.
STRATA_TOPK_OLD=1 keeps the original kernel, as before.

qsa_topk_parity (new, with the kernel parity tests): the dispatched top-k against qsa_block_topk_ref on both sides
of the register kernel's reach, at short contexts under a long capacity and up to 524,288 cells, called as decode
and as the prompt path call it, with continuous scores, 64 score levels and equal scores.

RTX 3060 12 GB (sm_86), 256 queries with their block count under a 262,144-cell capacity (524,288 for the last
two), original -> dispatched:
  16K cells    0.199 -> 0.199 ms (the original kernel)
  32K          0.845 -> 0.328
  131K         4.70  -> 0.97
  262K        16.09  -> 1.71
  393K        26.02  -> 2.49
  524K        37.53  -> 3.24
4 queries without a count (a decode window): 0.031 -> 0.028 ms at 8K, 0.176 -> 0.083 at 131K, 0.309 -> 0.127 at
262K.

IQ3_XXS on that card (--resident-experts, --max-context 262144, --kv int8 --kv-resident 32768),
tools/needle_bench.py, STRATA_TOPK_OLD=1 against the default: tok/s of the prompt, and the prompt path's selection
phase (STRATA_PREFILL_TIMING):
  16,471 tokens    1004.5, 1016.2 ->  989.0, 1013.6       195 ->    195 ms
  31,970           1043.3, 1055.5 -> 1060.1, 1060.5       900 ->    731
  61,333            932.9         -> 1013.8             4,061 ->  2,428
  122,755           904.7,  904.2 ->  992.7,  989.6    20,019 ->  9,034
  185,729           819.7         ->  954.7            50,055 -> 20,761
  243,176           743.3         ->  933.7            96,326 -> 35,544
The needles are found in every run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Niko1221 pushed a commit that referenced this pull request Oct 4, 2026
…ds with a histogram per warp (CUDA) - 9-12x at 262K-524K cells, a 243K-token prompt +26% on an RTX 3060

On CUDA the register kernel holds 33,792 blocks (~135K cells). Past it qsa_block_topk took the original kernel:
256 threads, each reading its own stretch of the scores on all six passes, one shared histogram updated with
atomics. The dispatch follows the capacity (--max-context), so with a context past ~135K that was every prompt
batch and every decode window on a card without the sm_75 active-bound dispatch (#512), whatever the prompt's
length, and on sm_75 every prompt batch past 33,792 blocks and every decode window.

block_topk_wide_kernel is the register kernel's layout (1,024 threads, a histogram per warp, warp scans) with the
keys read from memory:
- the four radix passes: thread t reads blocks t, t + 1024, ... - a histogram has no order, so a warp reads 32
  neighbours at a time;
- the count and the output: warp w holds `per` rows of 32 consecutive blocks, and the cells are written in the
  order (warp, row, lane); a row without a selected block is skipped.
The selection rule is the original kernel's (radix threshold, ties to the lowest index, cells ascending): identical
ids. One launch per call, no scratch memory, no capacity limit.

Dispatch (CUDA; HIP is unchanged and does not compile the kernel): where the blocks do not fit the registers the
new kernel, except a call that names fewer than 4,608 blocks (a prompt batch below ~18K cells), which keeps the
original kernel: its fixed cost is lower there. A decode window (no block count) takes the new kernel at any
context. The register kernel's range, the sm_75 active-bound dispatch and the sm_90+ cluster path are unchanged.
STRATA_TOPK_OLD=1 keeps the original kernel, as before.

qsa_topk_parity (new, with the kernel parity tests): the dispatched top-k against qsa_block_topk_ref on both sides
of the register kernel's reach, at short contexts under a long capacity and up to 524,288 cells, called as decode
and as the prompt path call it, with continuous scores, 64 score levels and equal scores.

RTX 3060 12 GB (sm_86), 256 queries with their block count under a 262,144-cell capacity (524,288 for the last
two), original -> dispatched:
  16K cells    0.199 -> 0.199 ms (the original kernel)
  32K          0.845 -> 0.328
  131K         4.70  -> 0.97
  262K        16.09  -> 1.71
  393K        26.02  -> 2.49
  524K        37.53  -> 3.24
4 queries without a count (a decode window): 0.031 -> 0.028 ms at 8K, 0.176 -> 0.083 at 131K, 0.309 -> 0.127 at
262K.

IQ3_XXS on that card (--resident-experts, --max-context 262144, --kv int8 --kv-resident 32768),
tools/needle_bench.py, STRATA_TOPK_OLD=1 against the default: tok/s of the prompt, and the prompt path's selection
phase (STRATA_PREFILL_TIMING):
  16,471 tokens    1004.5, 1016.2 ->  989.0, 1013.6       195 ->    195 ms
  31,970           1043.3, 1055.5 -> 1060.1, 1060.5       900 ->    731
  61,333            932.9         -> 1013.8             4,061 ->  2,428
  122,755           904.7,  904.2 ->  992.7,  989.6    20,019 ->  9,034
  185,729           819.7         ->  954.7            50,055 -> 20,761
  243,176           743.3         ->  933.7            96,326 -> 35,544
The needles are found in every run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Niko1221 pushed a commit that referenced this pull request Oct 6, 2026
With --max-context 262144 the QSA score rows have a stride of 65538 blocks; a prefill batch at 131K actual context
fits the register top-k kernel's 33792-block limit, but CUDA dispatch tested the allocated capacity and chose the
reference kernel. The supplied active_blocks bound is now used on sm_75 only; other CUDA cards keep the capacity
rule, HIP its own policy, and decode (no bound) is unchanged. STRATA_TOPK_CAPACITY_GUARD restores the capacity rule.
qsa_topk_active_parity: 16 cases at stride 65538, selected ids bitwise.

Squashed from #512 (its commit and its merge of upstream 0.1.36).

Co-Authored-By: Codex <noreply@openai.com>
Niko1221 added a commit that referenced this pull request Oct 6, 2026
…ard (#512 follow-up)

The engine takes the bound on Turing only, so on other cards the test compared the capacity rule with itself.
STRATA_TOPK_ACTIVE_ANY=1 (set by the test) takes the Turing dispatch on any card; the kernels are the same on every
architecture. The CMake block moves above the qsa_select_bench comment it had split.
Niko1221 pushed a commit that referenced this pull request Oct 6, 2026
…ds with a histogram per warp (CUDA) - 9-12x at 262K-524K cells, a 243K-token prompt +26% on an RTX 3060

On CUDA the register kernel holds 33,792 blocks (~135K cells). Past it qsa_block_topk took the original kernel:
256 threads, each reading its own stretch of the scores on all six passes, one shared histogram updated with
atomics. The dispatch follows the capacity (--max-context), so with a context past ~135K that was every prompt
batch and every decode window on a card without the sm_75 active-bound dispatch (#512), whatever the prompt's
length, and on sm_75 every prompt batch past 33,792 blocks and every decode window.

block_topk_wide_kernel is the register kernel's layout (1,024 threads, a histogram per warp, warp scans) with the
keys read from memory:
- the four radix passes: thread t reads blocks t, t + 1024, ... - a histogram has no order, so a warp reads 32
  neighbours at a time;
- the count and the output: warp w holds `per` rows of 32 consecutive blocks, and the cells are written in the
  order (warp, row, lane); a row without a selected block is skipped.
The selection rule is the original kernel's (radix threshold, ties to the lowest index, cells ascending): identical
ids. One launch per call, no scratch memory, no capacity limit.

Dispatch (CUDA; HIP is unchanged and does not compile the kernel): where the blocks do not fit the registers the
new kernel, except a call that names fewer than 4,608 blocks (a prompt batch below ~18K cells), which keeps the
original kernel: its fixed cost is lower there. A decode window (no block count) takes the new kernel at any
context. The register kernel's range, the sm_75 active-bound dispatch and the sm_90+ cluster path are unchanged.
STRATA_TOPK_OLD=1 keeps the original kernel, as before.

qsa_topk_parity (new, with the kernel parity tests): the dispatched top-k against qsa_block_topk_ref on both sides
of the register kernel's reach, at short contexts under a long capacity and up to 524,288 cells, called as decode
and as the prompt path call it, with continuous scores, 64 score levels and equal scores.

RTX 3060 12 GB (sm_86), 256 queries with their block count under a 262,144-cell capacity (524,288 for the last
two), original -> dispatched:
  16K cells    0.199 -> 0.199 ms (the original kernel)
  32K          0.845 -> 0.328
  131K         4.70  -> 0.97
  262K        16.09  -> 1.71
  393K        26.02  -> 2.49
  524K        37.53  -> 3.24
4 queries without a count (a decode window): 0.031 -> 0.028 ms at 8K, 0.176 -> 0.083 at 131K, 0.309 -> 0.127 at
262K.

IQ3_XXS on that card (--resident-experts, --max-context 262144, --kv int8 --kv-resident 32768),
tools/needle_bench.py, STRATA_TOPK_OLD=1 against the default: tok/s of the prompt, and the prompt path's selection
phase (STRATA_PREFILL_TIMING):
  16,471 tokens    1004.5, 1016.2 ->  989.0, 1013.6       195 ->    195 ms
  31,970           1043.3, 1055.5 -> 1060.1, 1060.5       900 ->    731
  61,333            932.9         -> 1013.8             4,061 ->  2,428
  122,755           904.7,  904.2 ->  992.7,  989.6    20,019 ->  9,034
  185,729           819.7         ->  954.7            50,055 -> 20,761
  243,176           743.3         ->  933.7            96,326 -> 35,544
The needles are found in every run.
anon761 pushed a commit to anon761/Strata-upstream that referenced this pull request Oct 6, 2026
With --max-context 262144 the QSA score rows have a stride of 65538 blocks; a prefill batch at 131K actual context
fits the register top-k kernel's 33792-block limit, but CUDA dispatch tested the allocated capacity and chose the
reference kernel. The supplied active_blocks bound is now used on sm_75 only; other CUDA cards keep the capacity
rule, HIP its own policy, and decode (no bound) is unchanged. STRATA_TOPK_CAPACITY_GUARD restores the capacity rule.
qsa_topk_active_parity: 16 cases at stride 65538, selected ids bitwise.

Squashed from Niko1221#512 (its commit and its merge of upstream 0.1.36).

Co-Authored-By: Codex <noreply@openai.com>
anon761 pushed a commit to anon761/Strata-upstream that referenced this pull request Oct 6, 2026
…ard (Niko1221#512 follow-up)

The engine takes the bound on Turing only, so on other cards the test compared the capacity rule with itself.
STRATA_TOPK_ACTIVE_ANY=1 (set by the test) takes the Turing dispatch on any card; the kernels are the same on every
architecture. The CMake block moves above the qsa_select_bench comment it had split.
anon761 pushed a commit to anon761/Strata-upstream that referenced this pull request Oct 6, 2026
…ds with a histogram per warp (CUDA) - 9-12x at 262K-524K cells, a 243K-token prompt +26% on an RTX 3060

On CUDA the register kernel holds 33,792 blocks (~135K cells). Past it qsa_block_topk took the original kernel:
256 threads, each reading its own stretch of the scores on all six passes, one shared histogram updated with
atomics. The dispatch follows the capacity (--max-context), so with a context past ~135K that was every prompt
batch and every decode window on a card without the sm_75 active-bound dispatch (Niko1221#512), whatever the prompt's
length, and on sm_75 every prompt batch past 33,792 blocks and every decode window.

block_topk_wide_kernel is the register kernel's layout (1,024 threads, a histogram per warp, warp scans) with the
keys read from memory:
- the four radix passes: thread t reads blocks t, t + 1024, ... - a histogram has no order, so a warp reads 32
  neighbours at a time;
- the count and the output: warp w holds `per` rows of 32 consecutive blocks, and the cells are written in the
  order (warp, row, lane); a row without a selected block is skipped.
The selection rule is the original kernel's (radix threshold, ties to the lowest index, cells ascending): identical
ids. One launch per call, no scratch memory, no capacity limit.

Dispatch (CUDA; HIP is unchanged and does not compile the kernel): where the blocks do not fit the registers the
new kernel, except a call that names fewer than 4,608 blocks (a prompt batch below ~18K cells), which keeps the
original kernel: its fixed cost is lower there. A decode window (no block count) takes the new kernel at any
context. The register kernel's range, the sm_75 active-bound dispatch and the sm_90+ cluster path are unchanged.
STRATA_TOPK_OLD=1 keeps the original kernel, as before.

qsa_topk_parity (new, with the kernel parity tests): the dispatched top-k against qsa_block_topk_ref on both sides
of the register kernel's reach, at short contexts under a long capacity and up to 524,288 cells, called as decode
and as the prompt path call it, with continuous scores, 64 score levels and equal scores.

RTX 3060 12 GB (sm_86), 256 queries with their block count under a 262,144-cell capacity (524,288 for the last
two), original -> dispatched:
  16K cells    0.199 -> 0.199 ms (the original kernel)
  32K          0.845 -> 0.328
  131K         4.70  -> 0.97
  262K        16.09  -> 1.71
  393K        26.02  -> 2.49
  524K        37.53  -> 3.24
4 queries without a count (a decode window): 0.031 -> 0.028 ms at 8K, 0.176 -> 0.083 at 131K, 0.309 -> 0.127 at
262K.

IQ3_XXS on that card (--resident-experts, --max-context 262144, --kv int8 --kv-resident 32768),
tools/needle_bench.py, STRATA_TOPK_OLD=1 against the default: tok/s of the prompt, and the prompt path's selection
phase (STRATA_PREFILL_TIMING):
  16,471 tokens    1004.5, 1016.2 ->  989.0, 1013.6       195 ->    195 ms
  31,970           1043.3, 1055.5 -> 1060.1, 1060.5       900 ->    731
  61,333            932.9         -> 1013.8             4,061 ->  2,428
  122,755           904.7,  904.2 ->  992.7,  989.6    20,019 ->  9,034
  185,729           819.7         ->  954.7            50,055 -> 20,761
  243,176           743.3         ->  933.7            96,326 -> 35,544
The needles are found in every run.
anon761 pushed a commit to anon761/Strata-upstream that referenced this pull request Oct 6, 2026
With --max-context 262144 the QSA score rows have a stride of 65538 blocks; a prefill batch at 131K actual context
fits the register top-k kernel's 33792-block limit, but CUDA dispatch tested the allocated capacity and chose the
reference kernel. The supplied active_blocks bound is now used on sm_75 only; other CUDA cards keep the capacity
rule, HIP its own policy, and decode (no bound) is unchanged. STRATA_TOPK_CAPACITY_GUARD restores the capacity rule.
qsa_topk_active_parity: 16 cases at stride 65538, selected ids bitwise.

Squashed from Niko1221#512 (its commit and its merge of upstream 0.1.36).

Co-Authored-By: Codex <noreply@openai.com>
anon761 pushed a commit to anon761/Strata-upstream that referenced this pull request Oct 6, 2026
…ard (Niko1221#512 follow-up)

The engine takes the bound on Turing only, so on other cards the test compared the capacity rule with itself.
STRATA_TOPK_ACTIVE_ANY=1 (set by the test) takes the Turing dispatch on any card; the kernels are the same on every
architecture. The CMake block moves above the qsa_select_bench comment it had split.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
anon761 pushed a commit to anon761/Strata-upstream that referenced this pull request Oct 6, 2026
…ds with a histogram per warp (CUDA) - 9-12x at 262K-524K cells, a 243K-token prompt +26% on an RTX 3060

On CUDA the register kernel holds 33,792 blocks (~135K cells). Past it qsa_block_topk took the original kernel:
256 threads, each reading its own stretch of the scores on all six passes, one shared histogram updated with
atomics. The dispatch follows the capacity (--max-context), so with a context past ~135K that was every prompt
batch and every decode window on a card without the sm_75 active-bound dispatch (Niko1221#512), whatever the prompt's
length, and on sm_75 every prompt batch past 33,792 blocks and every decode window.

block_topk_wide_kernel is the register kernel's layout (1,024 threads, a histogram per warp, warp scans) with the
keys read from memory:
- the four radix passes: thread t reads blocks t, t + 1024, ... - a histogram has no order, so a warp reads 32
  neighbours at a time;
- the count and the output: warp w holds `per` rows of 32 consecutive blocks, and the cells are written in the
  order (warp, row, lane); a row without a selected block is skipped.
The selection rule is the original kernel's (radix threshold, ties to the lowest index, cells ascending): identical
ids. One launch per call, no scratch memory, no capacity limit.

Dispatch (CUDA; HIP is unchanged and does not compile the kernel): where the blocks do not fit the registers the
new kernel, except a call that names fewer than 4,608 blocks (a prompt batch below ~18K cells), which keeps the
original kernel: its fixed cost is lower there. A decode window (no block count) takes the new kernel at any
context. The register kernel's range, the sm_75 active-bound dispatch and the sm_90+ cluster path are unchanged.
STRATA_TOPK_OLD=1 keeps the original kernel, as before.

qsa_topk_parity (new, with the kernel parity tests): the dispatched top-k against qsa_block_topk_ref on both sides
of the register kernel's reach, at short contexts under a long capacity and up to 524,288 cells, called as decode
and as the prompt path call it, with continuous scores, 64 score levels and equal scores.

RTX 3060 12 GB (sm_86), 256 queries with their block count under a 262,144-cell capacity (524,288 for the last
two), original -> dispatched:
  16K cells    0.199 -> 0.199 ms (the original kernel)
  32K          0.845 -> 0.328
  131K         4.70  -> 0.97
  262K        16.09  -> 1.71
  393K        26.02  -> 2.49
  524K        37.53  -> 3.24
4 queries without a count (a decode window): 0.031 -> 0.028 ms at 8K, 0.176 -> 0.083 at 131K, 0.309 -> 0.127 at
262K.

IQ3_XXS on that card (--resident-experts, --max-context 262144, --kv int8 --kv-resident 32768),
tools/needle_bench.py, STRATA_TOPK_OLD=1 against the default: tok/s of the prompt, and the prompt path's selection
phase (STRATA_PREFILL_TIMING):
  16,471 tokens    1004.5, 1016.2 ->  989.0, 1013.6       195 ->    195 ms
  31,970           1043.3, 1055.5 -> 1060.1, 1060.5       900 ->    731
  61,333            932.9         -> 1013.8             4,061 ->  2,428
  122,755           904.7,  904.2 ->  992.7,  989.6    20,019 ->  9,034
  185,729           819.7         ->  954.7            50,055 -> 20,761
  243,176           743.3         ->  933.7            96,326 -> 35,544
The needles are found in every run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants