[None][feat] Deterministic mode for the self-sampling GVR V2 top-K decode (fp32) by longcheng-nv · Pull Request #20100 · NVIDIA/TensorRT-LLM

longcheng-nv · 2026-10-11T08:45:31Z

Summary

Opt-in deterministic mode for the fp32 self-sampling GVR V2 top-K decode kernel (gvr_topk_decode_self_sampling): deterministic=True gives a repeatable index set and a repeatable output order, sorting="ascending" additionally sorts each row. The default arguments run the unchanged original module (bit-identical to main).

Why: the production kernel reserves output positions through atomics, so the output order follows arrival order even when every score is distinct, and when scores tie at the K-th boundary the selected set itself follows arrival order. Bitwise reproducibility of the indexer (and anything downstream) is impossible without a defined rule.

What changes:

run_varlen / run / run_ws / warmup_varlen and the prefill entry points run_prefill / prefill_ready / warmup_prefill gain deterministic: bool = False and sorting: str | None = None; TopK(...) gains gvr_deterministic / gvr_sorting and forwards them on both its decode and prefill self-sampling branches. Nothing else in the default path changes.
Integration: DeepSeekSparseAttentionConfig.gvr_deterministic_topk (default False, also on the V4 config) reaches DSAParams / DSAMetadataParams; the DSA indexer constructs TopK(gvr_deterministic=...) from it and the DSA metadata warmup pre-compiles the deterministic decode and prefill engines, so CUDA-graph batches keep their warmed launchers. YAML: sparse_attention_config: {enable_heuristic_topk: true, gvr_deterministic_topk: true}.
The deterministic result lives in the same device module behind cutlass.const_expr(self.det) branches (a det constructor flag on the four kernel classes, plus fused_k, the compile-time top-K, on the main kernel): selected set = {score > tau} plus the smallest indices among {score == tau}; output order is a fixed function of the input per kernel route (slot order on the register routes, per-warp staging segments in program order on main R=1 and clus, a bitmap / bitonic canonicalizer on the SPLIT merge CTA). With det=False the branches fold away and the original code path is compiled; the only additions on that path are three typed predeclarations (idv, bq, xv in the main kernel) that the DSL needs once the names are also assigned in the other branch.
Default path: cache keys and compile calls are unchanged when the flags stay at their defaults; results are identical to the pre-merge module (45-cell parity run: identical index sets on unique data, identical value multisets on tied data, same kernel-family routing) and the default-path kernel time is unchanged within noise (old-vs-new cold A/B table below).
On the main R=1 route at K=2048 the deterministic module stages 20480 candidates instead of 16384 (dynamic smem 164 -> 197 KB, still one CTA per SM) so that the clustered candidate lists of real indexer rows do not overflow the per-warp segments; the sampling ladder keeps its original capacity constant. An overflow re-sweeps the row once into exact segments (same order, no sort).

Scope: the self-sampling GVR engine (use_self_sampling_topk=True, the default under enable_heuristic_topk) for decode and prefill. The temporal-hint engine (use_self_sampling_topk=False) and the radix paths are untouched, so gvr_deterministic_topk is a no-op there; the QSA indexer only uses GVR for prefill and is covered by the same prefill path.

Cost (B200, cold L2, added kernel time over the original): synthetic rows geomean +10.2 % / median +1.26 us / max +4.29 us over 84 cells; real captured logits geomean +10.5 % / median +0.91 us / max +3.00 us over 100 cells. Per-route table and per-cell tables below.

Semantics

All modes select the K largest scores and use the same boundary tie-break: at equal scores the smaller index wins (index-min). sorting orders the output indices, not the scores. Repeatability assumes identical inputs, shape and software.

`deterministic`	`sorting`	selected set	output order
False (default)	None (default)	exact when the boundary scores are unique; arrival order decides among ties	arrival order, may differ between launches
True	None	fixed (index-min among ties)	fixed per kernel route, same on every launch, not sorted
True	"ascending"	same	ascending per row, `-1` pads at the tail
False	"ascending"		rejected: `ValueError: sorting requires deterministic=True` (sorting a changing set would only hide it)

Example 1, distinct scores, K=4 (adapted from the GVR README; the set {1, 3, 6, 7} is unique, only the order differs):

index:   0    1    2    3    4    5    6    7
score:  4.0  9.0  5.0 10.0  3.0  2.0  6.0  8.0

mode	launch 1	launch 2
default	`[7, 1, 6, 3]`	`[3, 6, 1, 7]` (may differ)
`deterministic=True`	`[7, 1, 6, 3]`	`[7, 1, 6, 3]` (a fixed permutation, not necessarily this one)
`deterministic=True, sorting="ascending"`	`[1, 3, 6, 7]`	`[1, 3, 6, 7]`

Example 2, boundary tie, K=4: indices 1, 3, 7 fill three slots; indices 2, 4, 6 tie at 7.0 for the last one. Index-min selects 2, so every deterministic launch returns the set {1, 2, 3, 7} (ascending [1, 2, 3, 7]); the default mode returns 2, 4 or 6 depending on arrival.

index:   0    1    2    3    4    5    6    7
score:  4.0  9.0  7.0  9.0  7.0  2.0  7.0  8.0

Measured, N=32768, K=512, bf16-rounded scores (129 ties at tau, 447 strictly above, 65 slots for the ties), B=64, three launches each: the default returns three different index sets (25795c, 3f2a9f, feceaf as set hashes) in different orders; deterministic=True returns the same set (1722e9) in the same order every time (8222, 66, 12359, ..., the main route's per-warp segment order); with sorting="ascending" the same set comes back as 66, 180, 306, 312, ....

Overhead

B200, nsys GPU spans, 24 reps per cell, L2 flushed before every rep ("cold"). "Added" is deterministic kernel time minus original kernel time on the same input; the original against itself (A/A) is within about +/-1 %. Synthetic rows: one unique fp32 row per N (B in 1/8/64/128, N 4K .. 1M). Real rows: captured fp32 indexer decode logits, 8 layers per capture, one capture per ISL, batch cycles the 8 rows, 16-aligned buffer stride with lengths carrying the valid N.

Added time per K and kernel route

K / data	route	cells	original kernel	added, mean	added, max	geomean
512, synthetic	reg	8	4.6 - 6.6 us	+0.54 us	+0.77 us	+10.1 %
512, synthetic	reg_clus	5	7.7 - 12.6 us	+0.65 us	+1.16 us	+6.9 %
512, synthetic	clus	4	15.4 - 67.6 us	+0.80 us	+1.16 us	+3.2 %
512, synthetic	main	11	10.6 - 113.1 us	+1.62 us	+1.98 us	+8.9 %
1024, synthetic	reg	8	5.1 - 6.8 us	+0.75 us	+1.00 us	+13.1 %
1024, synthetic	reg_clus	5	7.9 - 13.5 us	+0.63 us	+0.98 us	+6.2 %
1024, synthetic	clus	4	15.8 - 66.6 us	+0.98 us	+1.24 us	+3.3 %
1024, synthetic	main	11	11.4 - 114.5 us	+2.69 us	+3.52 us	+14.8 %
2048, synthetic	reg	8	4.8 - 6.3 us	+0.22 us	+0.70 us	+3.4 %
2048, synthetic	main	20	9.7 - 108.7 us	+2.41 us	+4.29 us	+14.8 %
512, Flash captures	reg	20	4.1 - 9.2 us	+0.55 us	+1.12 us	+8.5 %
512, Flash captures	reg_clus	7	7.4 - 10.4 us	+0.65 us	+1.20 us	+7.7 %
512, Flash captures	clus	3	12.4 - 23.9 us	+0.63 us	+1.14 us	+4.5 %
512, Flash captures	main	6	10.0 - 40.2 us	+1.67 us	+2.04 us	+10.4 %
1024, Pro captures	reg	20	3.9 - 9.3 us	+0.56 us	+1.26 us	+8.3 %
1024, Pro captures	reg_clus	7	7.7 - 10.3 us	+0.76 us	+1.30 us	+8.5 %
1024, Pro captures	clus	3	12.3 - 24.2 us	+1.18 us	+1.46 us	+7.1 %
1024, Pro captures	main	6	10.7 - 41.8 us	+2.50 us	+3.00 us	+14.9 %
2048, V3.2 captures	reg	8	5.5 - 7.4 us	+0.56 us	+0.76 us	+8.9 %
2048, V3.2 captures	main	20	9.6 - 29.3 us	+2.21 us	+2.78 us	+17.3 %

Reading the table: the register routes (reg, reg_clus) cost 0.2 - 0.8 us; clus 0.6 - 1.2 us; main 1.6 - 2.7 us on average with a maximum of +4.29 us (synthetic, B=1 N=1M K=2048, where the SPLIT merge CTA sorts the 2048 indices because the bitmap canonicalizer is bounded to N <= 256K). Percentages above 20 % only occur on 9 - 13 us kernels (B <= 8 on the main route). On the long-row cells (B >= 64, N >= 256K, 20 - 110 us) the added time is within a few percent. Warm L2 on the synthetic cells: geomean +10.8 %, median +1.35 us.

Without the staging change the V3.2 rows at B >= 64 would have cost +4 to +47 us (warp-segment overflow -> scalar re-sweep + sort); with it they are +0.51 to +2.78 us.

Synthetic rows, K = 512: geomean +8.1 %, median added +0.76 us, max +1.98 us (cell: original us -> added us (added %), route)

N	B=1	B=8	B=64	B=128
4,096	4.8 -> +0.46 (+9.5 %), reg	4.6 -> +0.77 (+17.0 %), reg	5.1 -> +0.49 (+9.6 %), reg	5.6 -> +0.45 (+8.1 %), reg
8,192	5.3 -> +0.30 (+5.6 %), reg	5.3 -> +0.66 (+12.6 %), reg	6.0 -> +0.56 (+9.3 %), reg	6.6 -> +0.63 (+9.6 %), reg
32,768	7.7 -> +0.57 (+7.4 %), reg_clus	8.0 -> +0.69 (+8.6 %), reg_clus	9.7 -> +1.16 (+11.9 %), reg_clus	11.4 -> +1.62 (+14.2 %), main
131,072	12.2 -> +0.44 (+3.6 %), reg_clus	12.6 -> +0.41 (+3.2 %), reg_clus	15.4 -> +1.16 (+7.5 %), clus	24.6 -> +1.28 (+5.2 %), main
262,144	10.6 -> +1.58 (+14.8 %), main	13.4 -> +1.65 (+12.3 %), main	24.2 -> +0.68 (+2.8 %), clus	40.7 -> +1.57 (+3.9 %), main
524,288	11.7 -> +1.55 (+13.2 %), main	15.8 -> +1.41 (+8.9 %), main	39.2 -> +0.62 (+1.6 %), clus	68.6 -> +1.98 (+2.9 %), main
1,048,576	14.2 -> +1.88 (+13.3 %), main	20.2 -> +1.95 (+9.7 %), main	67.6 -> +0.74 (+1.1 %), clus	113.1 -> +1.33 (+1.2 %), main

Synthetic rows, K = 1024: geomean +11.0 %, median added +0.99 us, max +3.52 us (cell: original us -> added us (added %), route)

N	B=1	B=8	B=64	B=128
4,096	5.4 -> +0.62 (+11.4 %), reg	5.1 -> +0.75 (+14.7 %), reg	5.7 -> +0.62 (+10.8 %), reg	5.6 -> +0.55 (+9.7 %), reg
8,192	5.7 -> +0.70 (+12.4 %), reg	5.4 -> +1.00 (+18.5 %), reg	6.2 -> +0.90 (+14.6 %), reg	6.8 -> +0.89 (+13.2 %), reg
32,768	7.9 -> +0.45 (+5.7 %), reg_clus	8.1 -> +0.47 (+5.8 %), reg_clus	10.1 -> +0.98 (+9.7 %), reg_clus	14.2 -> +1.94 (+13.7 %), main
131,072	12.2 -> +0.63 (+5.2 %), reg_clus	13.5 -> +0.62 (+4.5 %), reg_clus	15.8 -> +0.66 (+4.2 %), clus	27.2 -> +2.00 (+7.4 %), main
262,144	12.2 -> +3.52 (+28.7 %), main	14.5 -> +3.06 (+21.1 %), main	23.7 -> +1.24 (+5.2 %), clus	41.6 -> +2.17 (+5.2 %), main
524,288	11.4 -> +2.94 (+25.8 %), main	16.4 -> +3.36 (+20.4 %), main	39.6 -> +0.84 (+2.1 %), clus	68.3 -> +2.07 (+3.0 %), main
1,048,576	14.1 -> +3.35 (+23.7 %), main	20.3 -> +3.29 (+16.2 %), main	66.6 -> +1.16 (+1.7 %), clus	114.5 -> +1.94 (+1.7 %), main

Synthetic rows, K = 2048: geomean +11.4 %, median added +1.81 us, max +4.29 us (cell: original us -> added us (added %), route)

N	B=1	B=8	B=64	B=128
4,096	4.8 -> -0.17 (-3.5 %), reg	4.8 -> -0.22 (-4.6 %), reg	5.1 -> -0.13 (-2.5 %), reg	5.5 -> -0.20 (-3.6 %), reg
8,192	5.4 -> +0.61 (+11.4 %), reg	5.3 -> +0.70 (+13.1 %), reg	5.7 -> +0.57 (+10.1 %), reg	6.3 -> +0.58 (+9.2 %), reg
32,768	9.7 -> +2.20 (+22.8 %), main	9.8 -> +2.23 (+22.8 %), main	12.6 -> +2.06 (+16.4 %), main	14.0 -> +2.09 (+14.9 %), main
131,072	9.8 -> +2.07 (+21.1 %), main	11.5 -> +1.89 (+16.5 %), main	19.6 -> +1.61 (+8.2 %), main	25.5 -> +1.74 (+6.8 %), main
262,144	9.8 -> +2.40 (+24.4 %), main	12.6 -> +2.30 (+18.3 %), main	31.0 -> +1.74 (+5.6 %), main	40.5 -> +1.67 (+4.1 %), main
524,288	10.6 -> +3.99 (+37.5 %), main	14.6 -> +4.25 (+29.1 %), main	49.9 -> +1.73 (+3.5 %), main	65.2 -> +2.04 (+3.1 %), main
1,048,576	15.9 -> +4.14 (+26.0 %), main	22.3 -> +4.29 (+19.3 %), main	81.9 -> +1.72 (+2.1 %), main	108.7 -> +1.94 (+1.8 %), main

DSv4 Flash, K=512 (9 captures): geomean +8.3 %, median added +0.73 us, max +2.04 us (cell: original us -> added us (added %), route)

ISL / N	B=1	B=8	B=64	B=128
4k / 1,027	4.1 -> +0.01 (+0.2 %), reg	4.4 -> -0.34 (-7.7 %), reg	4.6 -> -0.15 (-3.2 %), reg	4.7 -> -0.20 (-4.3 %), reg
8k / 2,051	4.3 -> +0.09 (+2.2 %), reg	4.4 -> +0.50 (+11.4 %), reg	4.8 -> +0.33 (+6.8 %), reg	5.1 -> +0.30 (+5.8 %), reg
16k / 4,099	4.8 -> +0.64 (+13.3 %), reg	5.2 -> +0.61 (+11.7 %), reg	5.4 -> +0.90 (+16.7 %), reg	5.8 -> +0.87 (+14.9 %), reg
32k / 8,195	5.6 -> +1.02 (+18.3 %), reg	5.9 -> +0.72 (+12.2 %), reg	6.1 -> +1.01 (+16.6 %), reg	7.0 -> +1.12 (+16.0 %), reg
64k / 16,387	6.9 -> +0.91 (+13.2 %), reg	7.6 -> +0.62 (+8.2 %), reg	8.3 -> +0.89 (+10.7 %), reg	9.2 -> +1.05 (+11.5 %), reg
128k / 32,771	7.4 -> +0.67 (+9.1 %), reg_clus	8.0 -> +0.57 (+7.1 %), reg_clus	9.5 -> +1.20 (+12.7 %), reg_clus	11.7 -> +1.81 (+15.5 %), main
256k / 65,538	7.5 -> +0.76 (+10.2 %), reg_clus	8.5 -> +0.75 (+8.8 %), reg_clus	12.4 -> +1.14 (+9.3 %), clus	15.0 -> +2.04 (+13.6 %), main
512k / 131,075	9.8 -> +0.29 (+3.0 %), reg_clus	10.4 -> +0.34 (+3.3 %), reg_clus	15.6 -> +0.57 (+3.7 %), clus	24.4 -> +1.54 (+6.3 %), main
1024k / 262,127	10.0 -> +1.40 (+14.0 %), main	13.7 -> +1.17 (+8.6 %), main	23.9 -> +0.17 (+0.7 %), clus	40.2 -> +2.04 (+5.1 %), main

DSv4 Pro, K=1024 (9 captures): geomean +9.3 %, median added +0.91 us, max +3.00 us (cell: original us -> added us (added %), route)

ISL / N	B=1	B=8	B=64	B=128
4k / 1,027	3.9 -> -0.35 (-8.8 %), reg	4.0 -> +0.02 (+0.6 %), reg	4.5 -> -0.14 (-3.1 %), reg	4.6 -> -0.16 (-3.5 %), reg
8k / 2,051	4.7 -> +0.28 (+5.9 %), reg	4.7 -> +0.14 (+3.1 %), reg	5.1 -> +0.36 (+7.1 %), reg	5.0 -> +0.25 (+4.9 %), reg
16k / 4,099	4.9 -> +0.37 (+7.5 %), reg	5.0 -> +0.98 (+19.4 %), reg	5.6 -> +0.74 (+13.2 %), reg	6.2 -> +0.69 (+11.1 %), reg
32k / 8,195	6.0 -> +0.91 (+15.2 %), reg	6.1 -> +1.26 (+20.6 %), reg	6.3 -> +1.04 (+16.4 %), reg	6.8 -> +1.07 (+15.7 %), reg
64k / 16,387	7.0 -> +1.04 (+14.8 %), reg	7.7 -> +0.99 (+12.8 %), reg	8.8 -> +0.91 (+10.4 %), reg	9.3 -> +0.84 (+9.1 %), reg
128k / 32,771	7.7 -> +0.60 (+7.8 %), reg_clus	8.1 -> +0.74 (+9.1 %), reg_clus	10.1 -> +1.30 (+12.9 %), reg_clus	14.5 -> +2.30 (+15.9 %), main
256k / 65,539	8.1 -> +0.91 (+11.2 %), reg_clus	8.9 -> +1.08 (+12.2 %), reg_clus	12.3 -> +0.92 (+7.5 %), clus	17.4 -> +1.84 (+10.6 %), main
512k / 131,075	9.7 -> +0.06 (+0.6 %), reg_clus	10.3 -> +0.63 (+6.2 %), reg_clus	16.0 -> +1.46 (+9.1 %), clus	25.5 -> +2.72 (+10.7 %), main
1024k / 262,127	10.7 -> +2.67 (+24.8 %), main	13.4 -> +3.00 (+22.4 %), main	24.2 -> +1.15 (+4.7 %), clus	41.8 -> +2.50 (+6.0 %), main

DeepSeek-V3.2, K=2048 (7 captures): geomean +14.8 %, median added +2.08 us, max +2.78 us (cell: original us -> added us (added %), route)

ISL / N	B=1	B=8	B=64	B=128
4k / 4,111	5.7 -> +0.47 (+8.3 %), reg	5.5 -> +0.30 (+5.5 %), reg	5.7 -> +0.51 (+8.8 %), reg	6.1 -> +0.56 (+9.2 %), reg
8k / 8,207	6.4 -> +0.68 (+10.6 %), reg	6.5 -> +0.52 (+8.0 %), reg	7.0 -> +0.71 (+10.2 %), reg	7.4 -> +0.76 (+10.4 %), reg
16k / 16,399	9.6 -> +1.91 (+19.9 %), main	9.8 -> +2.06 (+21.0 %), main	10.8 -> +2.78 (+25.8 %), main	11.8 -> +2.51 (+21.2 %), main
32k / 32,783	9.8 -> +2.13 (+21.7 %), main	10.3 -> +1.93 (+18.8 %), main	13.0 -> +2.28 (+17.6 %), main	14.5 -> +2.17 (+15.0 %), main
64k / 65,551	9.7 -> +2.10 (+21.6 %), main	10.4 -> +1.91 (+18.4 %), main	15.9 -> +1.89 (+11.9 %), main	17.9 -> +2.06 (+11.5 %), main
128k / 131,087	9.8 -> +2.35 (+24.0 %), main	11.9 -> +2.31 (+19.4 %), main	20.9 -> +2.31 (+11.0 %), main	26.3 -> +2.30 (+8.7 %), main
256k / 163,775	10.2 -> +2.34 (+22.9 %), main	12.2 -> +2.49 (+20.4 %), main	22.9 -> +2.17 (+9.5 %), main	29.3 -> +2.11 (+7.2 %), main

Default path: pre-merge module vs this PR with `deterministic=False`

Same cold protocol, B 1/8/64/128; old = the device module before this PR, baseline = this PR's default path on the same inputs. Positive means the old module took longer. All differences are inside the +/-1-2 % A/A band (the -0.17 us minimum is a 3.5 us register kernel).

data	cells	this PR default, range	old - new, mean	old - new, min / max	geomean
synthetic K=512	28	4.0 - 109.7 us	+0.03 us	-0.13 / +0.16 us	+0.09 %
synthetic K=1024	28	4.1 - 110.8 us	+0.01 us	-0.38 / +0.27 us	+0.01 %
synthetic K=2048	28	4.8 - 108.8 us	+0.03 us	-0.13 / +0.24 us	+0.15 %
Flash captures K=512	36	3.8 - 38.1 us	+0.01 us	-0.16 / +0.17 us	+0.09 %
Pro captures K=1024	36	3.2 - 38.6 us	+0.01 us	-0.17 / +0.19 us	+0.09 %
V3.2 captures K=2048	28	4.7 - 29.4 us	+0.03 us	-0.08 / +0.25 us	+0.31 %

Validation

Prefill battery (run_prefill, deterministic): rows in {1, 8, 149, 297, 400} (all three row tiers, BLK 1024 / 512 / 256), N in {4K, 32K, 128K}, K in {512, 1024, 2048}, unique / bf16-rounded / all-equal / 8-value plateau scores, random windows and one short row per batch: 180 cases, 0 failures (oracle set in the local frame, identical output across launches, ascending option). This battery found and fixed one bug in the BLK=256 deterministic emission (output written in the global frame instead of column - ks).
696-record battery across 11 data patterns (unique, bf16-rounded, all-equal, boundary ties, plateaus, mixed lengths), eager and CUDA-graph replay, K in {256, 512, 1024, 2048}: 0 mismatches against a stable-sort oracle, 0 order differences across repeated launches. 88-case battery re-run on the shipped module: 0 failures.
Real captures (25 files, 3 models): exact set against the oracle and identical output across launches on every row; the original kernel's order differs between launches on the same rows.
Default path with the two arguments at their defaults: identical results to the pre-merge module on a 45-cell parity run (unique, bf16-rounded and all-equal scores; same kernel-family routing) and unchanged cold kernel time (table above).
sorting="ascending": rows sorted, same set as the unsorted deterministic output.

Test Coverage

tests/unittest/_torch/attention/sparse/test_cute_dsl_gvr_topk_deterministic.py: oracle set + repeat identity on one shape per kernel route x 4 data patterns (unique, bf16-rounded, all-equal, boundary ties), ascending rows, default-path argument validation, and a prefill case per row tier (1 / 149 / 297 rows) with random windows.
Existing test_cute_dsl_gvr_topk_decode.py / test_cute_dsl_gvr_topk_tiers.py keep covering the unchanged default path.

PR Checklist

PR title follows [None][feat] ...
Tests added
Default behavior unchanged; no new config / API surface beyond the opt-in keyword arguments

🤖 Generated with Claude Code

Dev Engineer Review

The PR adds opt-in deterministic selection to self-sampling GVR V2 Top-K. The new options flow through TopK, the GVR host APIs, and DeepSeek sparse-attention configuration. The reported defaults preserve the existing path; deterministic selection is intended for self-sampling GVR, not temporal-hint or radix paths.

The available diff for llm_args.py shows import-formatting changes, but does not show the reported gvr_deterministic_topk addition. The supplied change summary describes the feature wiring; I could not verify all implementation details against the available diff.

QA Engineer Review

The PR adds tests/unittest/_torch/attention/sparse/test_cute_dsl_gvr_topk_deterministic.py and modifies tests/unittest/_torch/modules/test_top_k.py. The new tests cover repeatability, tie-breaking, ascending output, prefill selection, and rejection of sorting without deterministic mode. The TopK test checks option forwarding.

The test-list search returned no matching entries under tests/integration/test_lists/. The supplied test summary reports hardware-gated tests, but no test-run results were independently verified here. Coverage verdict: needs follow-up.

Per-File QA Perspective

tensorrt_llm/_torch/attention/backends/sparse/dsa/indexer.py — Verify that the deterministic configuration reaches TopK without changing decode or prefill implementation selection.
tensorrt_llm/_torch/attention/backends/sparse/dsa/metadata.py — Verify that metadata warmup uses deterministic mode only when configured.
tensorrt_llm/_torch/attention/backends/sparse/dsa/params.py — Verify that the new configuration fields default to False and propagate through runtime and metadata parameters.
tensorrt_llm/_torch/cute_dsl_kernels/blackwell/top_k/gvr_topk_decode_self_sampling_host.py — Verify deterministic cache-key handling, output ordering, padding behavior, and the error for sorting without deterministic mode.
tensorrt_llm/_torch/modules/top_k.py — Verify option forwarding for self-sampling decode and prefill, and unchanged behavior with default arguments.
tensorrt_llm/llmapi/llm_args.py — The reported configuration adds gvr_deterministic_topk; verify its propagation to DSA parameters. The available diff shows formatting changes, not that addition.
tests/unittest/_torch/attention/sparse/test_cute_dsl_gvr_topk_deterministic.py — Tests deterministic selection, tie-breaking, ordering, prefill, and invalid sorting options. No integration test-list entry was found.
tests/unittest/_torch/modules/test_top_k.py — Tests that TopK forwards deterministic and sorting options. No integration test-list entry was found.

longcheng-nv · 2026-10-11T13:38:01Z

/bot run

tensorrt-cicd · 2026-10-11T13:44:48Z

PR_Github #77151 [ run ] triggered by Bot. Commit: 2807acd Link to invocation

tensorrt-cicd · 2026-10-11T14:04:33Z

PR_Github #77151 [ run ] completed with state FAILURE. Commit: 2807acd
/LLM/main/L0_MergeRequest_PR pipeline #63605 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

longcheng-nv · 2026-10-11T14:09:30Z

/bot run

tensorrt-cicd · 2026-10-11T14:17:03Z

PR_Github #77153 [ run ] triggered by Bot. Commit: a2b51d5 Link to invocation

longcheng-nv · 2026-10-11T14:18:49Z

/bot run

tensorrt-cicd · 2026-10-11T14:25:46Z

PR_Github #77155 [ run ] triggered by Bot. Commit: 37056a8 Link to invocation

tensorrt-cicd · 2026-10-11T14:29:59Z

PR_Github #77153 [ run ] completed with state ABORTED. Commit: a2b51d5

Link to invocation

tensorrt-cicd · 2026-10-11T17:14:48Z

PR_Github #77155 [ run ] completed with state SUCCESS. Commit: 37056a8
/LLM/main/L0_MergeRequest_PR pipeline #63608 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

longcheng-nv · 2026-10-12T00:55:25Z

/bot run

tensorrt-cicd · 2026-10-12T01:01:25Z

PR_Github #77168 [ run ] triggered by Bot. Commit: cb88296 Link to invocation

tensorrt-cicd · 2026-10-12T03:53:19Z

PR_Github #77168 [ run ] completed with state FAILURE. Commit: cb88296
/LLM/main/L0_MergeRequest_PR pipeline #63619 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

longcheng-nv · 2026-10-12T05:10:03Z

/bot run

tensorrt-cicd · 2026-10-12T05:16:05Z

PR_Github #77204 [ run ] triggered by Bot. Commit: 7fb624c Link to invocation

tensorrt-cicd · 2026-10-12T08:04:22Z

PR_Github #77204 [ run ] completed with state SUCCESS. Commit: 7fb624c
/LLM/main/L0_MergeRequest_PR pipeline #63647 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…code Opt-in `deterministic` / `sorting` arguments on the fp32 self-sampling host API (run, run_ws, run_varlen, warmup_varlen, run_prefill, prefill_ready, warmup_prefill), the TopK module (decode and prefill branches) and the DeepSeek sparse attention config (`gvr_deterministic_topk`, plumbed through DSAParams / DSAMetadataParams to the DSA indexer and its warmup). With `deterministic=True` the kernels compile their `cutlass.const_expr(self.det)` branches and compute a data-defined result: the selected set is {score > tau} plus the smallest indices among the ties, and the output order is a fixed per-route order (no sort); `sorting= "ascending"` additionally sorts the rows. Default arguments compile the original code path (same cache keys, identical results to main). The deterministic main R=1 kernel stages 20480 candidates at K=2048 (was 16384; the sampling ladder keeps the original capacity) so that real indexer rows, whose candidate lists are clustered, do not overflow the per-warp segments; an overflow re-sweeps the row once into exact segments. Cold-L2 added kernel time (B200): synthetic rows geomean about +10 %, median +0.9 us; real captured logits (Flash / Pro / V3.2) +0.2 .. +3.3 us per cell. Battery of 696 + 88 records, 0 mismatches. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>

longcheng-nv · 2026-10-12T08:07:16Z

/bot run

tensorrt-cicd · 2026-10-12T08:14:31Z

PR_Github #77233 [ run ] triggered by Bot. Commit: d0bab4a Link to invocation

tensorrt-cicd · 2026-10-12T12:31:50Z

PR_Github #77233 [ run ] completed with state SUCCESS. Commit: d0bab4a
/LLM/main/L0_MergeRequest_PR pipeline #63669 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

longcheng-nv · 2026-10-12T12:33:44Z

/bot run --stage-list "CPU-Generic-arm-1"

tensorrt-cicd · 2026-10-12T12:42:22Z

PR_Github #77271 [ run ] triggered by Bot. Commit: d0bab4a Link to invocation

tensorrt-cicd · 2026-10-12T14:02:29Z

PR_Github #77271 [ run ] completed with state SUCCESS. Commit: d0bab4a
/LLM/main/L0_MergeRequest_PR pipeline #63700 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

mikeiovine

Stamp on behalf of runtime devs, delegating proper review to @NVIDIA/trt-llm-topk-devs; please ping me if you think this is not accurate

longcheng-nv · 2026-10-13T01:27:36Z

/bot run

tensorrt-cicd · 2026-10-13T01:34:43Z

PR_Github #77366 [ run ] triggered by Bot. Commit: d0bab4a Link to invocation

tensorrt-cicd · 2026-10-13T02:03:07Z

PR_Github #77366 [ run ] completed with state SUCCESS. Commit: d0bab4a
/LLM/main/L0_MergeRequest_PR pipeline #63778 completed with status: 'SUCCESS'

CI Report

Link to invocation

yuxianq

Attention part LGTM

lfr-0531

LGTM~

github-actions · 2026-10-13T05:40:32Z

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

yuxianq · 2026-10-13T05:41:03Z

@yuxianq, this PR is already assigned to you as trt-llm-torch-attention-devs based on your earlier review. Could you help with the renewed review request? Thanks~

Assignment explanation

Components	backends/sparse (60%); integration/attention-tests (40%)
Tags	algorithm:sparse (100%); specific-algorithm:dsa (100%); phase:decode (50%); phase:prefill (50%); features:cuda-graphs (100%)
PR effort	8 points — Review requires deep reasoning about top-K tie selection, repeatable ordering across five intended kernel routes, prefill-local indices and warmup/dispatch agreement. Eight points is provisional because the core implementation patches are absent.
Importance	Important — The indexer exposes a sensitive sparse-token selection contract: tests require smaller-index boundary tie resolution and repeatable output across decode and prefill. Metadata also selects deterministic warmup variants.
Affinity (0–1)	1.000 component/tag match; 1.000 after PIC preference
Outstanding points at selection	97 → 97 (+0)
Points already reserved on this PR	8
Effective weight	1.5
Completed review points (preceding 168 hours)	37
Combined points after assignment	134
Demand-derived target	70.50 points (soft share)
Additional fair share before assignment	0.00 points
Share explanation	Existing owner confirmation; no new routing decision or additional obligation.
Selection rule	existing review owner; renewed attention team request

Affinity and workload are snapshots at selection, not live counters. Existing reservations on this PR are counted once.

…atches_committed_golden Re-enable the premerge privacy gate and regenerate the golden manifest, which drifted again while the waiver hid the test: NVIDIA#20100 added sparse_attention_config.gvr_deterministic_topk to llm_args.py without regenerating. Signed-off-by: Faraz Khoubsirat <farazkh80@users.noreply.github.com>

…atches_committed_golden Re-enable the premerge privacy gate and regenerate the golden manifest, which drifted again while the waiver hid the test: NVIDIA#20100 added sparse_attention_config.gvr_deterministic_topk to llm_args.py without regenerating. Signed-off-by: Faraz Khoubsirat <58580514+farazkh80@users.noreply.github.com>

github-actions Bot assigned longcheng-nv Oct 11, 2026

longcheng-nv force-pushed the det/gvr-v2-deterministic branch from 7cd8c9c to 2807acd Compare October 11, 2026 13:30

longcheng-nv force-pushed the det/gvr-v2-deterministic branch from 2807acd to a2b51d5 Compare October 11, 2026 14:09

longcheng-nv force-pushed the det/gvr-v2-deterministic branch from a2b51d5 to 37056a8 Compare October 11, 2026 14:18

longcheng-nv force-pushed the det/gvr-v2-deterministic branch from 37056a8 to cb88296 Compare October 12, 2026 00:55

longcheng-nv force-pushed the det/gvr-v2-deterministic branch from cb88296 to 7fb624c Compare October 12, 2026 05:09

longcheng-nv force-pushed the det/gvr-v2-deterministic branch from 7fb624c to d0bab4a Compare October 12, 2026 08:07

longcheng-nv marked this pull request as ready for review October 12, 2026 14:03

yuxianq removed the request for review from kris1025 October 12, 2026 14:13

mikeiovine approved these changes Oct 12, 2026

View reviewed changes

longcheng-nv added the api-compatible Accepted LLM API contract change that is backwards-compatible label Oct 13, 2026

longcheng-nv requested review from a team and yweng0828 October 13, 2026 02:35

yuxianq approved these changes Oct 13, 2026

View reviewed changes

longcheng-nv requested review from lfr-0531 and limin2021 October 13, 2026 05:31

longcheng-nv mentioned this pull request Oct 13, 2026

[https://nvbugs/6811958][chore] Unwaive test_build_capture_manifest_matches_committed_golden #20057

Merged

5 tasks done

longcheng-nv requested review from a team, kris1025 and pengbowang-nv October 13, 2026 05:36

lfr-0531 approved these changes Oct 13, 2026

View reviewed changes

trtllm-agent added the ci: full pre-merge approved label Oct 13, 2026

yuxianq requested review from yuxianq and removed request for kris1025 and pengbowang-nv October 13, 2026 05:41

limin2021 approved these changes Oct 13, 2026

View reviewed changes

yweng0828 approved these changes Oct 13, 2026

View reviewed changes

lfr-0531 merged commit b22283a into NVIDIA:main Oct 13, 2026
23 of 25 checks passed

This was referenced Oct 13, 2026

[None][feat] Expose the deterministic self-sampling GVR Top-K knob on CSA2 #20206

Closed

[None][fix] Reduce deterministic GVR V2 top-K overflow overhead #20290

Merged

farazkh80 mentioned this pull request Oct 15, 2026

[https://nvbugs/6811958][chore] Unwaive test_build_capture_manifest_matches_committed_golden #20419

Merged

5 tasks done

Conversation

longcheng-nv commented Oct 11, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Semantics

Overhead

Added time per K and kernel route

Default path: pre-merge module vs this PR with deterministic=False

Validation

Test Coverage

PR Checklist

Dev Engineer Review

QA Engineer Review

Per-File QA Perspective

Uh oh!

longcheng-nv commented Oct 11, 2026

Uh oh!

tensorrt-cicd commented Oct 11, 2026

Uh oh!

tensorrt-cicd commented Oct 11, 2026

Uh oh!

longcheng-nv commented Oct 11, 2026

Uh oh!

tensorrt-cicd commented Oct 11, 2026

Uh oh!

longcheng-nv commented Oct 11, 2026

Uh oh!

tensorrt-cicd commented Oct 11, 2026

Uh oh!

tensorrt-cicd commented Oct 11, 2026

Uh oh!

tensorrt-cicd commented Oct 11, 2026

Uh oh!

longcheng-nv commented Oct 12, 2026

Uh oh!

tensorrt-cicd commented Oct 12, 2026

Uh oh!

tensorrt-cicd commented Oct 12, 2026

Uh oh!

longcheng-nv commented Oct 12, 2026

Uh oh!

tensorrt-cicd commented Oct 12, 2026

Uh oh!

tensorrt-cicd commented Oct 12, 2026

Uh oh!

longcheng-nv commented Oct 12, 2026

Uh oh!

tensorrt-cicd commented Oct 12, 2026

Uh oh!

tensorrt-cicd commented Oct 12, 2026

Uh oh!

longcheng-nv commented Oct 12, 2026

Uh oh!

tensorrt-cicd commented Oct 12, 2026

Uh oh!

tensorrt-cicd commented Oct 12, 2026

Uh oh!

mikeiovine left a comment

Choose a reason for hiding this comment

Uh oh!

longcheng-nv commented Oct 13, 2026

Uh oh!

tensorrt-cicd commented Oct 13, 2026

Uh oh!

tensorrt-cicd commented Oct 13, 2026

Uh oh!

yuxianq left a comment

Choose a reason for hiding this comment

Uh oh!

lfr-0531 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 13, 2026

Uh oh!

yuxianq commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

longcheng-nv commented Oct 11, 2026 •

edited by coderabbitai Bot

Loading

Default path: pre-merge module vs this PR with `deterministic=False`

yuxianq commented Oct 13, 2026 •

edited

Loading