Repository navigation
RTX 5090, 1M context: --prefill auto:32768 reads a 598K prompt 21% faster, and above 135K cells the reference top-k costs more than the block scores #669
Description
Measurements of the prompt path at a 1M context on an RTX 5090, with what I tried and what did not help. No request attached: two observations you may want to act on, and three dead ends so that nobody repeats them.
Setup
Windows 11, RTX 5090 32 GB, Ryzen 9 9950X3D, 128 GB of RAM. Engine built from v0.1.38 (99f3dbd), Release, sm_120, CUDA 13.4, MSVC 14.44. Qwen3.8-Flash-Next GSQ-RCO IQ3_S, served by serve/server.py over /v1/messages.
Engine arguments: --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp <rt> --max-context 1048576 --kv int8 --kv-resident 32768 --rope-scaling yarn --rope-scale 4 --conversation-cache-mib 16384 --conversation-cache-slots 4. The expert cache has 11,702 slots and 205 MiB of VRAM stay free, in every run below.
Two prompts, always in this order after a fresh start: 58,342 tokens read from 0, then 597,889 tokens whose first tokens are the first prompt (so it resumes from a checkpoint). Synthetic log lines, one question about a line in the middle; the answer was right in every run.
1. --prefill auto:32768 reads 21% faster here
--prefill |
58,342 tokens | the long prompt |
|---|---|---|
auto (8,192) |
9.5 s | 548,737 tokens in 128.0 s, 4,287 tok/s |
auto:16384 |
8.4 s | 548,737 tokens in 112.6 s, 4,874 tok/s |
auto:32768 |
8.0 s | 565,121 tokens in 108.6 s, 5,201 tok/s |
Same expert slots, same free VRAM. The cost I saw: checkpoints land on chunk boundaries, so the long prompt resumed from 32,768 tokens instead of 49,152.
Where the time goes (STRATA_PREFILL_TIMING=1, the long prompt, ms):
| phase | auto (8,192), 548,732 tokens |
auto:32768, 565,116 tokens |
|---|---|---|
| GPU timeline | 141,014 | 111,759 |
| qsa select | 38,223 (27.1%) | 37,272 (33.4%) |
| dequant | 18,129 (12.9%) | 4,257 (3.8%) |
| gemm gate/up | 15,673 | 10,202 |
| gemm down | 8,269 | 5,850 |
| kv stage | 4,929 | 1,322 |
| wait copy | 1,702 | 423 |
| gdn recurrence | 9,567 | 9,906 |
| host staging | 19,946 | 4,896 |
Most of the gain is the dequant phase (the gather of each expert into its group slot, once per chunk) and the KV staging. On the 58K prompt qsa select is 3 to 5% of the read; at 565K it is a third.
2. Above ~135K cells the reference top-k costs more than the block scores
qsa_select_bench is only built for HIP in CMakeLists.txt. I built it for CUDA (target_link_libraries(qsa_select_bench PRIVATE strata_kernels CUDA::cudart)) and ran it with 256 queries and a 1,048,576-cell capacity (ms per call):
| context | block scores (TF32) | top-k |
|---|---|---|
| 32,768 | 0.087 | 0.063 |
| 131,072 | 0.289 | 0.229 (0.168 with STRATA_TOPK_ACTIVE_ANY=1) |
| 400,000 | 0.835 | 1.06 |
| 598,000 | 1.256 | 1.38 |
| 1,000,000 | 2.269 | 3.13 |
The scores are linear in the context. The top-k is not, and it passes the scores from 400K on: fit is TK_T * TK_PER = 33,792 blocks on NVIDIA, so beyond ~135K cells qsa_block_topk takes qsa_block_topk_ref whatever the active bound is (STRATA_TOPK_ACTIVE_ANY=1 changed nothing at 400K and 598K). That kernel reads each query's score row six times (four radix passes, the count, the write) with 256 threads per query.
So at a 1M context roughly half of qsa select is the reference top-k. I did not write a replacement.
What did not help
- More key tiles per CTA in
block_scores_tc_kernel(TC_ITER16 or 64 instead of 4): at 16, +2% at 598K and +5% at 1M; at 64, -2% and +7%. Both lose 8 to 64% at 131K and below. Scores bitwise identical. - The query tile stored as its TF32 hi and lo parts in shared memory, split once per CTA instead of once per key tile: 12 to 39% slower, at every context. Scores bitwise identical.
STRATA_TOPK_ACTIVE_ANY=1in the engine, default chunk: 9.53 s and 128.0 s, the same as without.
"Bitwise identical" is an FNV-1a over the scores of the tested scorer, added to the bench, equal to the unchanged kernel's at 2,048, 32,768, 131,072, 400,000, 598,000 and 1,000,000 cells.
Two things the bench shows on unchanged v0.1.38 code
- The FP64 gate prints
FAILat 2,048 and 131,072 cells on this card: the tensor-core scorer's error is 1.0e-06 of the score scale, right at the floor (max(4 * err_old, 1e-6 * scale)), andPASSat 32,768 and 598,000 with 9.8e-07. If the bench becomes a CUDA ctest, that floor will flip. - At 400,000 cells, 253 of 256 selections are identical between the warp scorer and the tensor-core one (256 of 256 at the other five contexts). That is the "near-tie can select differently" the kernel's comment describes.
Raw logs are available if you want them.
Activity
T-Crypt commentedon Oct 4, 2026
RTX 4090 with 31 GB RAM:
auto:32768reads long prompts about 3x slower, because the lent slots' experts no longer fit in RAMA data point from the other end of the RAM range, which I think qualifies observation 1 above rather than contradicting it.
Setup. RTX 4090 24 GB, i9-14900K, 31 GB RAM, Linux. Engine built from main 99f3dbd (sm_89). Qwen3.8-Flash-Next IQ2_XS,
--resident-experts --expert-cache auto --spec 4 --spec-min-p 0.5 --mtp <rt> --kv q4_0 --kv-resident 32768. Each pair below differs only in--prefill, with one engine start per run. The prompts are synthetic log lines, read from 0 (0 reused).--prefillauto:32768autoauto:32768autoThe engine's own line explains it:
FileExpertSource: 3699 of the prompt path's 5991 lendable slots keep their experts in RAM too (the others are read from the file when lent: not enough RAM for them). With the bigger chunk, about 2,300 to 4,000 of the lent experts come back from the SSD on every chunk, and that outweighs the dequant and staging savings you measured. With 128 GB they all stay in RAM, so only the gain shows.Decode is unaffected (within run-to-run noise).
Suggestion. When the prompt path picks a chunk above 8,192, it could cap the chunk at the size whose lent slots still all fit in RAM (the count it already logs), or fall back to 8,192 when they don't. The RAM budgeting in #658 looks like the natural place for that. I have not tried it.
Caveat: the
autorows came from a build that also carries an unrelated KV change (#711, which only touches--kv k8v4), not from 99f3dbd itself. The q4_0 paths are the same source.gputier commentedon Oct 5, 2026
Thanks, that's the case I couldn't see: my runs were on 128 GB, and I didn't log the
FileExpertSourceline, so I have no lent-slot count to compare with yours. Observation 1 should read "on a PC whose RAM holds the lent experts".Capping the chunk at the size whose lent slots all stay in RAM sounds right to me, since the engine already counts them.
For anyone reading this on 0.1.39:
--prefill autonow picks the chunk from the expert ring's size in bytes (#583), and the top-k past the register kernel's reach uses a histogram per warp (#603). I haven't checked whether the new chunk choice looks at that RAM count.brenoperucchi commentedon Oct 5, 2026
On the question of whether 0.1.39 changes observation 1: on this PC it doesn't. RTX 5090 32 GB, Ryzen 9 5950X, 96 GB DDR4, release 0.1.39, ISTA-DASLab IQ3_S at a 65,536-token context,
--expert-cache auto --spec 4 --spec-min-p 0.70 --kv int8 --pcie-frac 0.55, one engine start per setting, 1 warm-up + 3 runs, every prompt read from 0:--prefill auto--prefill auto:32768That is the same gain as on 0.1.38 here (+14% and +25%, from 5,520 and 5,495 tok/s). With 96 GB the lent slots' experts should stay in RAM (I didn't log the count), which is gputier's case; it says nothing about T-Crypt's 31 GB one. Swift IQ3_XXS at a 32K context shows the same pattern against setup 0.1.39's defaults (+10% and +18% at 14.7K and 28.9K), though that pair also differs in
--pcie-fracand--spec-min-p. The runs, configs and engine logs are in #433.engine: a big prompt chunk whose lent slots do not fit in RAM is warn…
Niko1221 commentedon Oct 7, 2026
0.1.40.2: when a prompt chunk above 8,192 tokens lends more cache slots than the RAM can keep copies for, the engine warns at start with the counts and recommends --prefill auto. Not capped: with the RAM for them the big chunk is the faster one.
engine: a big prompt chunk whose lent slots do not fit in RAM is warn…
エンジンの小さな修正を移す（プロファイルの版、密な重みのシャード、警告と --help）
gputier commentedon Oct 10, 2026
Thanks @Niko1221. The 0.1.40.2 warning covers T-Crypt's case: when the RAM cannot keep the lent slots' experts, the engine says so at start and points to --prefill auto, and with the RAM for them the big chunk keeps its gain. Closing, 0.1.40.2 answers it.