kv: --kv k8v4 streams with --kv-resident

[T-Crypt]

What

--kv k8v4 (INT8 K + rotated Q4_0 V, 816 B/cell) now streams with --kv-resident N, like fp16, int8 and q4_0. Both refusals are gone (generate.cpp and the backstop in qsa_state_init).

How

The streaming machinery already moves a block as a list of byte runs (runs_of in kv_stream.cu), so K8V4 is a fourth format with three runs, and everything that moves blocks (resolve, copy, ring restore, prompt-path staging) takes it unchanged.

• KvFormat gains kKvHybrid = 3, the number conversation snapshots already use for K8V4. Its runs are K codes, K scales and V q4_0, and kv_block_bytes knows its size. qsa_kv_format returns it instead of exiting.
• Writers: the hybrid appends already fold each half onto one pool (kv_append_q8_step(k_q, k_q, ...), kv_append_q4_step(v_q4, v_q4, ...)). Two helpers in kv_stream.hpp, kv_hybrid_k_half and kv_hybrid_v_half, give each half's host copy (and staging pool) in the shape those kernels expect. Both fields of a half point at the same array on purpose: the kernels test the K-side pointer for presence (if (host.k_q4 != nullptr)) and then pick K or V per lane, so a V half with only v_q4 set would silently skip the host write. The decode (layer.cpp), verify-window (verify.cpp) and prompt (prefill.cpp) appends pass them.
• qsa_state_init allocates the host copy in the three runs, and take_stage gives the prompt path's staging pool the K8V4 layout (pools_of then reads it as mode 3).
• Conversation snapshots read a streamed K8V4 state's host copy, as they already do for the other formats. A ring is still refused.

setup.py is unchanged and still keeps K8V4 unstreamed: a prebuilt engine without this refuses the pair, so turning it on there needs a MIN_ENGINE that includes it. I left that for you to time.

Tests

• kv_stream_parity now runs K8V4 too, with the engine's folded appends. Streamed and resident attention are bitwise equal: 306 batches, 702,021 block lookups, 70,449 misses, 0 failures, and the ring restore is identical. The other three formats are unchanged and still pass.
• Negative control: with the V half's K-side pointer left null (the trap above), the test fails from the first batch (k8v4 pos 130 n_q 8: streamed attention differs).
• kv_hybrid_parity: all pass.

Measured

RTX 4090 24 GB (sm_89), i9-14900K, 31 GB RAM, Linux, CUDA 13. Qwen3.8-Flash-Next IQ2_XS, --max-context 262144 --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp <rt> --resident-experts, one engine start per row. Long prompts are synthetic log lines with one planted line in the middle, and the answer was right in every row. "Decode at depth" is a ~300-token second turn on the cached prompt.

--kv	streamed	GPU expert slots	short decode	32K: read / decode	119K: read / decode	238K: read / decode
k8v4	no	11,208	117.7	3,879 / 92.8	3,870 / 114.2	3,658 / 111.0
k8v4	--kv-resident 32768	12,953	126.2	3,935 / 99.8	3,960 / 116.4	3,659 / 108.2
q4_0	--kv-resident 32768	13,033	117.8	4,090 / 99.9	4,045 / 108.2	3,796 / 120.1

Streamed K8V4 holds 2.39 GiB of KV in pinned RAM, 95.7% of its KV block reads hit VRAM, and its freed VRAM went to 1,745 more expert slots (11,208 -> 12,953). Prompt-cache checkpoints were restored on the streamed K8V4 state (114,688 and 229,376 tokens reused) and decoded normally.

Single runs, so read the decode differences of a few percent as noise, except the short-decode gain, which comes from the larger expert cache.

Not tested

• HIP: not built here. The change is in shared C++ and in kv_stream.cu, so a HIP build should pick it up the same way it does for the other formats.
• Multi-GPU (--gpus). A port of this change on 2x V100 with a layer split reports wrong answers on short stateless prompts under k8v4 that int8 does not show (tools: a stateless prompt probe (k8v4 on 2x V100: dropped prompt tokens and stale-context replies) jmnargi/Strata-V100#34). Not reproduced on sm_89 yet; see the thread below.

Tested by others

• @midhatn, RTX 4070 Laptop 8 GB, Windows, Swift 1.5 IQ3_XXS, 64K context, --kv k8v4 --kv-resident 20480, 2 GiB two-slot conversation cache: a 17-request smoke test passed, including a 57,745-token conversation restore after an unrelated request and an image prefix restore. kv_stream_parity and kv_hybrid_parity pass on that GPU. Built on v0.1.39 with Overlap streamed KV uploads with prefill computation #732, Pipeline CPU expert work without a batch-wide phase barrier #733 (opt-ins off) and Respect Windows commit capacity #749.

[midhatn]

I exercised the conversation-cache path listed as untested here on Windows: Ryzen 9 7940HS, RTX 4070 Laptop 8 GB, 64 GB RAM, Swift 1.5 IQ3_XXS, 65,536 context, --kv k8v4 --kv-resident 20480, and a 2 GiB / two-slot conversation cache.

The 17-request smoke test passed, including:

• A cold 57,750-token lookup, a separate auxiliary request, then restoration of 57,745 tokens with the correct answer.
• A red-square image, an auxiliary request, then restoration of the 1,086-token image prefix with the correct color. Subsequent green and blue images were read afresh and answered correctly.
• A tool-call round trip and a generated function checked independently on 155 interval-merging cases plus invalid-input, generator and input-mutation checks.

kv_stream_parity and kv_hybrid_parity also passed on this GPU. This is a local v0.1.39 build with this PR plus #732/#733 (both opt-ins disabled) and #749, so it is not a standalone-PR performance result. I haven't compared complete model-state fingerprints, tested cancellation here, or tested beyond 64K.

On this small card the automatic expert cache increased from 993 to 1,028 entries versus INT8 KV. I didn't see a speed benefit in this workload, but the bounded RAM conversation-cache and image-history checks worked.

[T-Crypt]

@midhatn thanks for this. I hadn't tested the conversation-cache restore at all, and you covered the two cases I worried about most: a 57K-token restore after an unrelated request, and an image prefix that comes back with the right color. I've added your rig to the description under "Tested by others".

There's one open report against this change. @fishlikeX runs a port of #711 on 2x V100 with a layer split and sees two failures under --kv k8v4 --kv-resident. Short stateless prompts lose tokens ("reply with 10086" gets "1086"), and once prefix reuse engages, replies come from an unrelated parked conversation. The same build on --kv int8 passes all 16 checks (jmnargi#34). That rig goes through the Volta prompt attention in the CUDA 12 engine and the layer split. Your 4070 and my 4090 use neither.

I'll run their probe (tools/stateless_prompt_probe.py, with --parked) on the 4090 against this branch, k8v4 and int8 back to back, and post the result here. A clean 4090 run points at the Volta or layer-split path. A failure gets fixed on this PR before anyone merges it.

[fishlikeX]

Update from the fork side (fishlikeX here — I run the V100 port where I reported the two failures): root cause found and fixed — jmnargi#36 — and it changes the expected outcome of your 4090 run.

The failures are not the Volta prompt attention or the layer split. This fork's verify path writes K/V through a captured-graph batch replay (kv_append_q8_batch / kv_append_q4_batch — fork machinery so one launch covers a window over any slot), and the k8v4 case in that branch passed nullptr for the host mirror while every other format passes &st.host and the per-step k8v4 branch mirrors through the halves helpers. Every decode-written cell therefore skipped the host copy, and this fork's resume path trusts the invariant ("every writer writes both") and skips re-reading — so parking and restores served stale bytes: fresh prompts lost tokens, and reused prefixes returned other conversations.

The fix is five lines — fold the hybrid halves and pass the host pools in the batch branch, the same pattern as the per-step branch. The exact diff: https://github.com/jmnargi/Strata-V100/pull/36/files. On the same rig the probe goes from 1 ok / 11 bad / 4 inconclusive ("4096" answered "2048", echoes truncated, six reused-prefix requests answering from unrelated conversations) to 14 ok / 0 bad / 2 inconclusive, with the previously-failing reused-prefix rounds (cached_tokens 55-57) all correct.

Two notes for this PR:

• Your 4090 run against this branch should come back clean: the per-step writes here mirror correctly (the k8v4 step branch carries the halves helpers, and every other format passes &st.host). So a clean 4090 would not mean the failures are Volta- or layer-split-specific — they were the fork's batched replay layer, which this repo does not have.
• If the batched replay form ever moves upstream, the hybrid mirror has to move with it — that is the one line this port initially missed.

The probe lives in the fork at https://github.com/jmnargi/Strata-V100/blob/main/tools/stateless_prompt_probe.py (merged via jmnargi#34) if you want it for the back-to-back run. Thanks for taking the report seriously and adding the rig to the description — and to @midhatn for the 4070 restore coverage.

[Niko1221]

Fixed in 0.1.40: K8V4 now streams with --kv-resident (your commit, kept as it is). Thank you for the kv_stream_parity cases.

[gravitomagnetic]

Field report: --kv k8v4 on RTX 5090 (sm_120), 0.1.40 — +30-33% decode, but a deterministic repetition artifact on sequential generation. Ran the A/B on both of our quants (IQ3_S and Q2_0), same harness, int8 baseline vs k8v4, fresh engine start per leg.

Setup: Qwen3.8-Flash-Next (qwen4exp MoE, 12 QSA layers), --kv-resident 32768, 1M ctx YaRN×4, --mtp --spec 4, --batch 2 --batch-mtp, vision on. Driver flips --kv, waits for a real completion, then probes: cold prefill ~200K tokens, decode 256 tokens, needle-in-haystack at ~200K.

lane	leg	KV pinned RAM	expert slots	engine free	cold prefill	decode	needle @200k	counting probe
IQ3_S	int8	12.38 GiB	8,139	490 MiB	3,369 t/s	98 t/s	✅ exact	clean 1 2 3 4 5…
IQ3_S	k8v4	9.56 GiB	8,238	489 MiB	3,276 t/s	130 t/s	✅ exact	⚠ repeats
Q2_0	int8	12.38 GiB	12,598	544 MiB	3,516 t/s	151 t/s	✅ exact	clean 1 2 3 4 5…
Q2_0	k8v4	9.56 GiB	12,815	544 MiB	3,385 t/s	197 t/s	✅ exact	⚠ repeats

What works:

• KV pinned RAM −22.8% exactly as the 816 vs 1,056 B/cell math predicts, and the engine auto-converts the freed space into expert slots (--expert-cache behaves as a floor, not a cap — nice).
• Decode +33% / +30% on the two quants. Consistent and large.
• Needle recall at 200K tokens: untouched. Retrieval survives 4-bit values completely.

The artifact: the "count from 1 to 300" decode probe repeats each integer under k8v4:

1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 10 10 11 11 11 11 12 12 12 12 13 13 13 1…

The striking part: the corrupted output is byte-identical between IQ3_S and Q2_0 — two substrates sharing nothing about their weights (1.94 vs 1.32 MiB experts, different quantization entirely). The only shared component is the k8v4 KV path, so the repetition is introduced by the value quantization itself, deterministically, at temperature 0. That signature reads more like a systematic grid/scale bias in the Q4_0 dequant than random distortion — possibly fixable rather than inherent.

It's also exactly the asymmetry the design predicts: K stays INT8-exact (pointer → retrieval fine, needle survives), V is rotated Q4_0 (payload → short-range recency signal blurs, and counting depends on "what did I just emit").

Recommendation from our side: k8v4 looks clearly right for chat/voice seats (decode latency dominates, enumeration rare); for agent seats that count, enumerate, or emit structured output, int8 is the safer default until the repetition is understood. Single run per leg, and we've only characterized the counting probe — other structured outputs (code, JSON) untested. Happy to run more legs on our box if useful.