Repository navigation
Conversation
…ing scorer (bitwise) A verify window's 1-8 queries scored and selected on a handful of SMs; at 420K cells on an RTX 2080 Ti the two took ~6 ms of a ~52 ms window. - block_scores_halve_kernel: the decode scorer's warp sums as one recursive-halving reduction (each value pairs lane l with l ^ o at offsets 16..1, the butterfly's additions operand for operand), the lane's query values in registers (shared-memory reads were the limit, ncu mio_throttle), 5-8 queries as two calls of up to 4. Same bits (qsa_select_bench QSA_BENCH_DECODE=1 hashes, 32K-524K, 1-8 queries). 420K: 1 / 3 / 4 / 6 / 8 queries 117 / 265 / 347 / 510 / 774 -> 101 / 118 / 125 / 238 / 255 us. STRATA_SCORES_HALVE=0 keeps the old kernel. - qsa_block_topk_split: the decode top-k over chunks of 2,048 blocks in six launches (four radix passes into per-query global histograms, a count, the emit); every CTA takes each digit from the complete histogram with block_topk_kernel's rule, so the ids are identical (qsa_topk_parity --selftest: continuous, 64-level and equal scores, 2K-524K). Decode calls at 420K: 0.21 -> 0.033-0.064 ms; short contexts unchanged within 0.01 ms. STRATA_TOPK_SPLIT=0 keeps the wide kernel (=1: on any card, for tests). Both on sm_75 only by default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tical ids) block_topk_wide_kernel reads a query's scores six times (four radix passes, a count, the emit): past ~115K cells that is DRAM traffic. block_topk_sample_kernel takes a bound from every 32nd block (the KS-th largest of those keys), reads all blocks once keeping those at or above it, and when they hold at least `width` cells computes block_topk_kernel's threshold, `above` and tie budget over them and emits them in block order (bitonic sort): every block at or above the threshold is a candidate, so the ids are the reference's. Otherwise the query is flagged and block_topk_wide_kernel<true> runs it. qsa_topk_parity --selftest and qsa_topk_active_parity: identical ids (continuous, 64-level, equal scores - the last ones take the fallback). RTX 2080 Ti, 256 prompt queries under a 512K capacity: 131K 0.51 -> 0.45 ms, 262K 0.95 -> 0.59, 400K 1.48 -> 0.71, 520K 1.84 -> 0.79. Used above 28,672 active blocks; STRATA_TOPK_SAMPLE=0 keeps the wide kernel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
xrip
added a commit
to xrip/Strata
that referenced
this pull request
Oct 12, 2026
…mpt read, first logits byte-identical Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #2072 (it sits next to that PR's split top-k in
qsa_select.cu); the first commit here is #2072's, the second is this one.Past ~115K cells the prompt path's top-k is DRAM traffic on Turing:
block_topk_wide_kernelreads a query's scores six times (four radix passes, a count, the emit), 400 KB a query at 400K, 256 queries a call.block_topk_sample_kernelreads them once:widthcells, the width-th largest cell key (block_topk_kernel's threshold) is ≥ LB, so every block at or above the threshold is a candidate: the threshold,aboveand the tie budget computed over the candidates with block_topk_kernel's rule are the reference's, and the candidates sorted by block (bitonic) emit the reference's cells in its order;block_topk_wide_kernel<true>runs it - the all-equal-scores case of the parity test takes that path.Used for prompt calls above 28,672 active blocks (below, the wide kernel is as fast);
STRATA_TOPK_SAMPLE=0keeps the wide kernel.Setup
RTX 2080 Ti 22 GB (sm_75), i7-6700, CUDA 13.0, driver 595, SM 1650 MHz, 240 W;
v0.1.43+ #2072,-DSTRATA_BUILD_TESTS=ON. Model Q2_0,--max-context 524288,--kv int8 --kv-resident 32768,STRATA_SELECT_SIMT=2(#2071).qsa_topk_parity CTX 256 20 524288 1(a prompt call), dispatched timeqsa_topk_parity --selftest: PASS (continuous, 64-level and equal scores; identical ids).qsa_topk_active_parity(STRATA_TOPK_ACTIVE_ANY=1): 20 cases, identical.Engine
First logits after a 196K prompt: bitwise identical to the run without it (
STRATA_DUMP_FIRST_LOGITS). The gain sits at depth: with this, #2071 and #2091 and #2092, appending 16K tokens to a 390K conversation (--serve) went 15.9 → 12.9 s here.🤖 Generated with Claude Code