sm_75: run the prompt path's BF16 products on the FP16 tensor cores (+15-18 % prefill on RTX 2080 Ti)

[rafatxf]

What

Turing (sm_75) and Volta (sm_70) have FP16 tensor cores but no BF16 ones, so cublasGemmEx on CUDA_R_16BF inputs falls back to a SIMT fp32 kernel (magma_sgemmEx). On an RTX 2080 Ti that was the largest kernel of the prompt path, 20 % of a 4K prompt (nsys). It runs every Gemm::bf16 product: the hyper-connection down / up projections, the router and indexer rows, and PLE key / value.

On compute capability 7.x, Gemm::bf16 now:

• converts the weight and the activations BF16 → FP16, the activations in row slices of at most 32 MiB;
• runs the existing Gemm::f16 product (fp32 accumulate) on the tensor cores.

BF16 → FP16 is exact for every value inside FP16's range, because the 7-bit mantissa fits in 10 bits.

• Scope. Only compute capability 7.x (Volta, Turing) by default; every other card runs exactly what it ran before. HIP is untouched (#if !defined(__HIPCC__)).
• Override. STRATA_BF16_TC=0/1 forces the path off or on.
• Single-row products (N = 1) stay on cuBLAS's GEMV, which is faster than the conversions.
• Scratch. It belongs to the Gemm instance (there is one, in Prefill::Impl) and is freed with it. With Qwen3.8-Flash-Next it peaks at about 45 MiB: the 2560×2560 PLE value weight plus one activation slice.
• Allocation failure. If the scratch allocation fails, the call falls back to the cuBLAS BF16 path.

Accuracy

New test gemm_bf16_parity. It compares Gemm::bf16 with cuBLAS's own BF16 product on the same inputs. GPU, synthetic, no model; --bench adds timings. It covers:

• the prompt path's shapes;
• a T large enough to slice the activations;
• beta = 1 accumulation (the bf16x2 low parts);
• an output row stride wider than N.

On the 2080 Ti the worst relative difference is 1.7e-5. With STRATA_BF16_TC=0 it is 0, so the test also passes unchanged on other cards.

Real products. I also checked every beta = 0 product of a real 32K prompt (400 products) against the cuBLAS BF16 result, with a temporary diagnostic that is not in this PR:

• worst relative difference 3.5e-5;
• no activation outside FP16 range (max |X| 75.5);
• no non-finite values.

Greedy output. It matches the unpatched engine for the first 52 (4K) and 105 (32K) tokens, then diverges the usual way for a change in summation order.

gr_parity still passes.

Numbers

Setup:

• RTX 2080 Ti 22 GB (PCIe 3.0 x16), i7-6700, 32 GB DDR4-2133, Linux.
• Qwen3.8-Flash-Next Q2_0, --resident-experts, --spec 4.
• Fixed-token prompts, greedy, same binary with STRATA_BF16_TC=0 vs 1.

prompt	prefill t/s, before	after
4K	877	1020	+16 %
16K	1134	1337	+18 %
32K	1153	1366	+18 %
64K	1116	1312	+18 %
128K	1009	1166	+15 %

Decode does not use Gemm::bf16 and is unchanged within noise.

Per product (gemm_bf16_parity --bench):

product	cuBLAS BF16	this PR
hc down, T 4096 × N 320 × K 10240	4428 µs	1668 µs
hc up, T 4096 × N 10240 × K 320	3468 µs	613 µs
PLE value, T 777 × N 2560 × K 2560	1440 µs	325 µs

After the change, the conversion kernel is 3.8 % of a 32K prefill. Writing FP16 directly from the producers would take that back, but it touches more code, so I left it out of this PR.

Measured on sm_75 only (Volta was added in review, unmeasured). Other cards keep the default off, so their path is the same code as before.

🤖 Generated with Claude Code

[eelgaev]

sm_70 can benefit from this too, Volta (V100) cards introduced Tensor core FP16 operations.

[rafatxf]

@eelgaev Good point. Volta has FP16 tensor cores and no BF16 ones too, so cuBLAS takes the same SIMT path there. 7b2dcd3 widens the default to compute capability 7.x.

I can't measure on a V100. If you can, ./build/gemm_bf16_parity --bench would confirm it there (it builds with -DSTRATA_BUILD_TESTS=ON), and STRATA_BF16_TC=0 / =1 A/Bs a prompt.

[Niko1221]

Merged for the next release, with your commits and gemm_bf16_parity, and two changes: on by default on Volta only, and opt-in on RTX 20 (STRATA_BF16_TC=1). We have no sm_75 card to gate it on, and the ready-made engine's output stays the same until then. #540's safeguards were added: finite values beyond FP16's range clamp to 65504, and a beta = 1 product (the BF16X2 remainder, in FP16's subnormal band) keeps cuBLAS. A run of the parity test and an A/B on your 2080 Ti with the merged code would help us turn it on for RTX 20.

[rafatxf]

Thanks for taking it. Here are the parity test and an A/B on the RTX 2080 Ti with the merged code: the v0.1.39 tag, built from source with -DSTRATA_BUILD_TESTS=ON. Setup: sm_75, CUDA 13.0, driver 595, RTX 2080 Ti 22 GB (PCIe 3.0 x16), i7-6700, DDR4-2133. Model: Q2_0, --resident-experts, --spec 4.

gemm_bf16_parity

With the default, the FP16 path is off on sm_75: 0 failures, every difference 0.

With STRATA_BF16_TC=1: 0 failures, worst relative difference 1.7e-5.

product	cuBLAS BF16	FP16 path
hc down, T 4096 × N 320 × K 10240	4428 µs	1693 µs	2.62x
hc up, T 4096 × N 10240 × K 320	3467 µs	620 µs	5.59x
PLE value, T 777 × N 2560 × K 2560	1440 µs	327 µs	4.40x
router / indexer q, T 512 × N 512 × K 2560	233 µs	69 µs	3.38x
hc inject, T 300 × N 4 × K 10240	758 µs	53 µs	14.25x
single row (N = 1, stays cuBLAS)	22.3 µs	22.2 µs	1.00x
row stride wider than N	193 µs	29 µs	6.73x

Speed: same binary, STRATA_BF16_TC 0 vs 1

Fixed-token prompts, greedy, 256 new tokens, runs interleaved off / on / on / off. Means of two runs:

prompt	prefill off	prefill on		decode off → on
4K	904	1028	+13.7 %	49.0 → 49.0
32K	1156	1371	+18.6 %	49.3 → 48.9
128K	1010	1168	+15.7 %	45.6 → 46.3

Run to run, prefill spread is under 2 %.

Quality: teacher-forced, the two paths against each other

Free-running greedy text is not a usable gate here. In the one-shot engine, two runs with the same setting already part ways after 115-190 tokens. So I compared the two paths teacher-forced, the way docs/UNSLOTH_Q4.md describes:

• a --serve engine with --adapt-every 100000;
• the first P tokens of a document read by the batched prompt path (the path this changes), as one request;
• a second request with the next 2,048 tokens, which resumes at P (RESUME P) and reads them through the verify windows with --short-read;
• STRATA_LOGPOS with STRATA_LOGPOS_TOPK=20 writing every position.

The text is raw prose (Don Quixote, Spanish), without a chat header.

prefix read batched	positions	argmax agreement	KL (off ‖ on, top 20 + rest)	perplexity off / on
8,192	2,047	95.2 %	0.011	3.995 / 4.038
30,720	2,047	96.8 %	0.0085	2.146 / 2.159

• Noise floor: two runs with the path off write identical files (KL 0, every position the same), so every difference above is the FP16 path.
• Near-ties: the disagreements are almost all near-ties. Where the off run's two most likely tokens are at least 0.5 nats apart, the argmax agrees at 1745 of 1757 positions (8K) and 1858 of 1875 (30K).
• Perplexity: it moves by +1.1 % and +0.6 %. A chat-formatted run of the same 30K prompt went the other way (3.500 → 3.454, without <|im_end|>).
• For scale: both KLs are below the 0.02-0.05 that DETAILS.md lists for other prompt-path changes against the FP16 path.

The server here now runs 0.1.39 with STRATA_BF16_TC=1.