Repository navigation
enable --kv-grow with --batch - #1232
BlueKingMuch wants to merge 4 commits into
Conversation
… up front; its table uploads wait for the copy With --kv-grow the K/V is made elastic at session init and maps 16,384 cells. Growth needs more than init checks: a device residency table, cache slots and a loan above the floor. Without them (--no-pool, --no-token-graph, the dump modes, --expert-cache auto resolving to 0) kvg_start left the growth off, nothing mapped more, and a prompt past 16K wrote K/V into unmapped memory: an illegal address. That run now maps the whole window from new memory at start, or stops with a message when it does not fit. The growth's and the trim's residency-table uploads also wait for their own copy (a pageable cudaMemcpy may return before its DMA lands; the streams that read the table are non-blocking). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio
… of switching --kv-grow off
With --batch N every slot carves a session for the whole --max-context, used or not, and --kv-grow switched itself
off beside --batch ("the batch slots carve their own K/V"). Now --kv-grow makes every slot session elastic too:
- the elastic pools come in groups: the main session and its drafter are group 0, batch slot b's session group b + 1
(qsa_set_kv_elastic_group while it is initialized). qsa_kv_elastic_{cells,need,grow,shrink} take the group; every
existing caller is group 0, so one session runs exactly as before;
- with --batch-mtp each slot's drafter K/V joins its slot's group, so it grows and shrinks with the slot;
- kvg_ensure / kvg_trim take the group too: all groups draw on the same loan of the expert cache (the slots just below
the prompt path's) and give their chunks back to it;
- a slot grows before every write: at its admission (giving back what it held beyond first), at a BYIELD, and before
each batch window (its position + 1 + 64). A slot starts with 4,096 cells mapped (STRATA_KV_GROW_SLOT_INIT);
- a slot's indexer keys (the pooled rows) are an array of its elastic range too, one row per page. The main
session's stay whole: the prompt path rounds all of them for its block scores, and it runs on the main session only.
--kv-grow stays off beside --vram-elastic and --peer-device, as before. Without --kv-grow nothing changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Correction to the description: the reason it gives for keeping the main session's indexer keys whole is wrong. The prompt path does not read all pooled rows; its block scores stop at each query's completed blocks, as the windows do (qsa_block_scores_tc reads up to the highest n_bid of its queries). Only the debug check STRATA_IDX_FP16_CHECK reads them all. The same sentence is in the commit message and in the comment above pooled_elastic. The change itself is unaffected: the main session's keys stay whole, as before. |
generate.cpp: kvg_trim takes both Niko1221#1128's hold (STRATA_KV_GROW_HOLD=1, the cells of the longest parked conversation) and the group: kvg_trim(cells, hold, grp). The main session's trim keeps the hold; a slot's trim at its admission holds only its own conversation (parked conversations are restored into the main session), kvg_trim(need, 0, admit_slot + 1). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FKZxC8sUotEzdWXws2FTUC
…s stay whole, instead of a wrong reason The comment said the prompt path rounds all of a state's pooled rows for its block scores. It reads them only up to its queries' completed blocks (qsa_block_scores_tc stops at the highest n_bid), as the windows do; only the debug check STRATA_IDX_FP16_CHECK reads them all. The code is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FKZxC8sUotEzdWXws2FTUC
|
I've updated this PR for 0.1.40.3: a merge of v0.1.40.3 and one commit on top.
Measured on 0.1.40.3, this against #1231 on 0.1.40.3, with the setup from the description (RTX 4080 SUPER 32 GB, PCIe 3.0, Windows 11, IQ3_S, 262K int8, 4 slots, conversation cache 12,288 MiB; 1 orchestrator + 4 subagents, two rounds, the orchestrator working on its own between them, 6 turns a round):
With STRATA_KV_GROW_HOLD=1 as well (346 s, no ERR), the main session keeps its cells as that setting intends (no trim, 14 without it), and the slots trim at their admissions as without it (8 times in both runs). Tests: |
|
Thanks. This is a memory feature (smaller slot sessions, more experts in VRAM) that builds on #1231 and no longer applies cleanly to the current main, so it is outside the speed pass. The 375 -> 358 s on a 32 GB card is a nice side effect; we keep it for the cache review. |
|
Thanks for the pull request. We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected. To reopen it takes about two minutes:
The rules are in CONTRIBUTING.md. |
2 participants
Builds on #1231 (the first commit); this PR is the second.
With
--batch Nevery slot carves a session for the whole--max-context, used or not: 3.65 GiB each at 262K with int8 K/V, or 0.95 GiB with--kv-resident 32768plus ~3.1 GiB of pinned RAM.--kv-growswitches itself off beside--batch("the batch slots carve their own K/V"). This makes every slot session elastic under--kv-grow.What it does
qsa_set_kv_elastic_groupwhile it is initialized).qsa_kv_elastic_{cells,need,grow,shrink}take the group; every existing caller is group 0, so one session runs exactly as before.--batch-mtpeach slot's drafter K/V joins its slot's group, so it grows and shrinks with the slot.kvg_ensure/kvg_trimtake the group too: all groups draw on the same loan of the expert cache (the slots just below the prompt path's) and give their chunks back to it.copy_to_slot, trimming what it held beyond first), at a BYIELD, and before each batch window (its position + 1 + 64). A slot starts with 4,096 cells mapped (STRATA_KV_GROW_SLOT_INIT).upto.--kv-growstays off beside--vram-elasticand--peer-device, as before. Without--kv-grownothing changes.Tests
tools/batch_test.py, four slots,--pcie-frac 0 --adapt-every 1000000 --no-prefill-borrow --prefill 1024: without and with--kv-grow4 of 4 IDENTICAL; with--batch-mtp, without and with--kv-grow, 4 of 4 IDENTICAL (with the one-line fix from fix(mtp): --batch-mtp exits on the first admission with a draft-vocab head (slot drafters never copy its type) #1063: the MTP directory here has adraft_vocab.bin).tools/batch_interleave_test.pywith--kv-growand the slots pre-grown (STRATA_KV_GROW_SLOT_INIT=16384): 8 of 8 IDENTICAL.vmm_test,conversation_snapshot_test,conversation_cache_test,conversation_file_testpass.Measured
On 0.1.40.1, this against 0.1.40.1: RTX 4080 SUPER 32 GB, PCIe 3.0, Windows 11, IQ3_S, 262K int8, 4 slots,
conversation cache 12,288 MiB; 1 orchestrator + 4 subagents, two rounds, the orchestrator working on its own between them (6 turns a round):
--kv-resident 32768--kv-growEach subagent's slot grew to 16,384 cells and took 86-88 expert slots for it; a slot that held the orchestrator grew to 49,152 cells (227-314 slots) and gave 223 back when it went on with a subagent.
The same pair with the VRAM of a 16 GB card, emulated on this card (
--vram-reserve-mib 17084: the same PCIe link, 16 GiB less for the cache):--kv-resident 32768--kv-growcc @sergqwer - this extends your elastic K/V (#378) to the batch slots.