Repository navigation
Q4_1 acceleration - #193
Merged
Merged
Conversation
Contributor
Author
|
The generation at 13B Q4_1 feels a bit iffy to me. Note the immediate misspelling (it changes the assistant's name to "Allice" in over half of the chats earlier or later), sheer drunkenness (fantasy words, believing St Basil's Cathedral to be a bridge) and final repetition loop despite the repetition penalty. I'll try and see if eliminating the RMS patch makes it better. |
Contributor
Author
|
Some limited evidence that the RMS patch may be problematic. I cherry-picked it away, getting the following: Same seed: Different seed: |
Closed
Contributor
Author
|
Requesting review; it seems that it wouldn't actually let me review and merge myself (...even though I possibly could have merged locally and pushed straight to master?) |
ggerganov
approved these changes
Mar 17, 2023
Member
|
@blackhole89 Thank you! Great work as usual 🦙 |
anzz1
referenced
this pull request
in anzz1/alpaca.cpp
Mar 17, 2023
* Add AVX2 version of ggml_vec_dot_q4_1 * Small optimisations to q4_1 dot product (@Const-me) * Rearrange Q4_1 quantization to work for multipart models. (Fix antimatter15#152) * Fix ggml_vec_mad_q4_1 too * Fix non-vectorised q4_1 vec mul
mudler
pushed a commit
to go-skynet/llama
that referenced
this pull request
Mar 17, 2023
* Add AVX2 version of ggml_vec_dot_q4_1 * Small optimisations to q4_1 dot product (@Const-me) * Rearrange Q4_1 quantization to work for multipart models. (Fix ggml-org#152) * Fix ggml_vec_mad_q4_1 too * Fix non-vectorised q4_1 vec mul
Contributor
|
Thanks for doing that! I successfully quantised 30B to 4_1 size, here's my test.
system_info: n_threads = 16 / 16 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | main: prompt: ' Transcript of a conversation between a User and an AI Assistant named Judy. Judy is a helpful, kind, honest and never fails to answer the User's requests immediately and in great detail. Judy knows the answer to any question.' main: interactive mode on. sampling parameters: temp = 0.700000, top_k = 40, top_p = 0.500000, repeat_last_n = 64, repeat_penalty = 1.176470 == Running in interactive mode. ==
main: mem per token = 43600900 bytes |
Contributor
Author
|
Thank you! @FNsi, is this the Q4_1 quantification you are testing? The filename says models/30B/ggml-model-q4_0.bin. How is the performance at 30B? |
Contributor
|
Sorry I just find out the problem I made. I already changed the line to 4_1 bin, and resubmit the current response.
|
Seunghhon
pushed a commit
to Seunghhon/llama.cpp
that referenced
this pull request
Apr 26, 2026
* Add AVX2 version of ggml_vec_dot_q4_1 * Small optimisations to q4_1 dot product (@Const-me) * Rearrange Q4_1 quantization to work for multipart models. (Fix ggml-org#152) * Fix ggml_vec_mad_q4_1 too * Fix non-vectorised q4_1 vec mul
ljubomirj
pushed a commit
to ljubomirj/llama.cpp
that referenced
this pull request
May 6, 2026
* Add AVX2 version of ggml_vec_dot_q4_1 * Small optimisations to q4_1 dot product (@Const-me) * Rearrange Q4_1 quantization to work for multipart models. (Fix ggml-org#152) * Fix ggml_vec_mad_q4_1 too * Fix non-vectorised q4_1 vec mul
UltraRabbit
pushed a commit
to UltraRabbit/llama.cpp
that referenced
this pull request
Sep 23, 2026
The MTP graph in qwen35 does its own ggml_get_rows() on the token embedding table but never restores the primal basis, while llm_graph_context::build_inp_embd() does exactly that for the trunk (llama-graph.cpp, "a Hadamard-latent embedding table stores rotated rows; restore the primal basis right after the lookup"). On a Hadamard-folded model the draft head therefore consumes embeddings in the rotated basis, and llama_verify_hadamard_graph (correctly) refuses to build the graph, so in-file MTP cannot start at all: W llama_verify_hadamard_graph: latent lookup 'mtp_tok_embd-64' consumed by op=RMS_NORM name='norm-64' E llama_init_from_model: failed to initialize the context: Hadamard-latent table 'token_embd.weight' is read without the inverse transform E common_speculative_init_result: failed to create MTP context Reproduced with prism-b10683-d8f26ee (newest release with published assets, see ggml-org#193) and still present on b10709-9a9394a, on ProCreations/Ternary-Bonsai-2-27B-MTP (the PQ2_0 target with one MTP layer in the same file) with: llama-server -m Ternary-Bonsai-2-27B-PQ2_0-MTP-Q8_0.gguf --spec-type draft-mtp Fix: apply the same rot + signs inverse used by the trunk path, keyed on the table that was actually looked up (layer.nextn.embed_tokens when present, otherwise model.tok_embd). The same change ships as runtime/bonsai-mtp-embedding.patch in ProCreations/Ternary-Bonsai-2-27B-MTP (and their patched source archive); this is the equivalent for current prism. Credit for finding it belongs there.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 21, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 21, 2026
…r, QSA/indexer)
old-14's qwen4exp support plus old-15's qwen4exp/QSA/HC content, kept as ONE
block because QSA is intrinsic to the qwen4exp architecture (the old-15 QSA
memory wins, HC16 and qsa3 are load-bearing for the qwen4exp graph, and the
QSA/HC op dispatch is old-15):
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the
MTP draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm
policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, lightning-indexer.cu, with the CUDA dispatch
(GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE / INDEXER_FILL) and
their CPU references; the indexer cache in llama-memory-hybrid-idx.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (+ dispatch, supports_op, the
hc_combine_mix fusion) and the HC16 planar layout.
- the managed lazy reader (llama-lazy-reader) and the loader's managed_budget.
- the GDN beta-sigmoid fusion, the DSV4_HC split states in the Meta backend,
and the QSA/HC op definitions in ggml.h / ggml.c.
Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16,
GGML_CUDA_QSA_SLICES, GGML_CUDA_SPLICE_GATHER and the GDN_DBG_SKIP_* diagnostics once into values
rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call
overwrites, so a cached pointer is unsafe on Windows.
Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not
linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper.
Folded in from four contributor PRs on v12-097cf33d9-r80 (bsp2020 / briansp2020; block 10 also owns
the rope + FWHT fusion that ggml-org#172 and ggml-org#174 put here in r78):
- fused RMS_NORM + MUL + ROPE (+ FWHT) also for M-ROPE / IMROPE (PR ggml-org#175, 0001): the matcher admits
GGML_ROPE_TYPE_MROPE / IMROPE besides NORMAL / NEOX, and the kernel computes theta with rope_multi's
own section selection and float expressions (NEOX pairing, iw == i0, VISION still rejected). This is
the TODO item recorded against ggml-org#172: every Qwen3.x model runs rope_multi, so the fusion was inert on
the dense 27B and the 35B-A3B. Kill switch GGML_CUDA_DISABLE_ROPE_MROPE_FUSE=1.
- BF16 caches off the 5-op RMS_NORM + MUL + ROPE + VIEW + SET_ROWS fusion (PR ggml-org#175, 0002): that fusion
runs ggml_cuda_op_rms_norm_mul_rope_fused, which writes F32 / F16 rows only and otherwise aborts, but
ggml_cuda_should_fuse_rope_set_rows admits a BF16 SET_ROWS. Latent on r80 for a NEOX qk-norm model
with -ctk bf16, and reachable for the Qwen3.x models once 0001 lands. The 5-op match now declines a
BF16 SET_ROWS; the RMS_NORM + MUL + ROPE fusion and the plain SET_ROWS take it instead.
- the q8_0 K and V cache writes folded into the rope + FWHT kernel (PR ggml-org#173): with a q8_0 cache the K
path ends MUL_MAT(Hadamard), RESHAPE, VIEW, SET_ROWS and the V write (VIEW, SET_ROWS) follows it, so
the fused kernel now writes both cache rows through ggml_cuda_quantize_q8_0_group (k_set_rows_q8_0_group's
body, moved into set-rows.cuh). Only the K write is folded inside a concurrent-stream region
(GGML_CUDA_GRAPH_OPT=1). Kill switch GGML_CUDA_DISABLE_ROPE_FWHT_SET_ROWS=1.
- the GDN q and k l2 norms in one launch (PR ggml-org#179): build_gdn_l2_norm normalizes both with the identical
RMS_NORM + SCALE over views of the same conv output, so rms_norm_scale_pair_f32<256> takes blocks
[0, nrows) for q and [nrows, 2*nrows) for k with the same per-row body. Matcher additions: only
no-op nodes between the two pairs, both inputs F32 views of the same view_src with equal ne / nb,
equal eps and scale params, and the second input must not read the first pair's output.
Kill switch GGML_CUDA_DISABLE_NORM_PAIR=1.
Rejected in the same review: PR ggml-org#177 (GDN direct conv also writes the recurrent snapshot tails). It is
not run-to-run deterministic on the 35B-A3B prefill under HIP graph capture and does not reproduce the
stock logits there; record in archive/work/gdn-conv-snapshot-fold/.
Prefill cache staging hook (r83): ggml_backend_cuda_stage_from_cache is live again for the expert-cache
arena. The scheduler's whole-table staging copy of a host-resident expert table (stage_upload for
-sm layer, stage_gather for the -sm tensor meta split) is filled by one kernel from the arena (resident
experts, D2D) + the pinned host master (the rest, H2D) instead of a full H2D re-upload; the destination
is the ordinary staged table in expert order, so every consumer is unchanged. It declines to the plain
upload for an unusable/stood-down table, an unpinned host master or a geometry mismatch. Kill switch
GGML_MOE_CACHE_STAGE=0.
Slab narrow band (r85): the always-resident narrow band was an estimate taken before the first ring carve
(a `split_only` reserve of the `n_rs_batch` shape, 1792 MiB on the field config), and `set_narrow_floor`
cannot correct it afterwards because the ring's VA is frozen once it holds blocks. The real MTP
decode/verify graph needs 2304 MiB, so `work_alloc` classified every decode view as wide, placed it above
the H2D staging ring and evicted 3.65 GiB/device of arena. `work_alloc` now learns the band from the real
graph: a view that is not a requested wide (prefill) pass is narrow even above the floor while no wide view
is live (the ring region is only carved while one is), and the floor + ring are re-carved from it (the ring
is idle there). With a wide view live the above-the-ring placement is kept: the ring's VA must not move.
A refused raise is reported instead of dropped silently.
Measured (2x R9700, Flash-Next IQ4_NL -ncmoe 46, 131k ctx, -b/-ub 6144, MTP n6): 18 -> 1 re-arm, decode
23.6 -> 45.4 t/s (-sm tensor) / 35.8 (-sm layer), prefill unchanged; cost: 16k prefill 1443 -> 1411 t/s
(-2.2 %) because the correctly sized band moves the ring up 512 MiB.
The "may the band move" test is `ring_armed` alone: a live WIDE view is the only thing that makes the ring
region part of the reserved layout (`ring_armed` is set exactly when the boundary covers the carved ring).
`ring_used` is a bump cursor that stays at the end after a prefill, and `boundary > narrow_floor` is true
for an unrelated narrow view one chunk above the floor; treating either as "the ring may not move" left
the band unlearned on a single device -- measured gfx1100 `-ncmoe 40`: floor 1280 MiB against a 1536 MiB
decode view, one arena eviction per prefill -> decode transition. With this test the band is learned
there too (floor 1280 -> 1536, view placed at base 0, no eviction) and the two gfx1100 greedy hashes are
unchanged.
Fold (r86): beta 12, slab-transient-absorption (CUDA/slab half). The wide-phase
auxiliary region hosts the FA staging arena, the QSA3 workspace and the chunked-GDN
scratch; the workspace pool serves its large prefill temporaries from it; a graph
capture is never started while the wide band is armed; per-device free=/free_min/
outside_max/outside_failed diagnostics; default headroom 640 + 128 x n_gpu MiB.
Fold (r91): the chunked GDN prefill writes its rollback snapshots (fixed-D sequential tail).
The chunked kernel computes no per-token states, so a partial rollback into a batch it served
restored a state it never wrote (probe: logit NMSE 0.65, maxabs 12.5; see
archive/work/rs-rollback-boundary/). A single-sequence batch above the chunked threshold now
runs the chunked kernel on its first n_tokens - D tokens and the SEQUENTIAL kernel on the last
D = max(K>16?K:16, n_rs_batch) tokens in the same launch, so snapshot slots 0..D-1 exist for
every batch. No extra memory, no extra graph, ubatch unchanged (splitting the ubatch instead
measured -13.7 % pp512 and was reverted).
The bound is structural, not observational: cparams.n_rs_batch = the widest verify batch of any
enabled speculator (common_speculative_n_max() + 1, all draft and ngram types) and is carried
into this op as param 1, so a verify batch is never chunked; the tail covers the non-speculator
partial rewind (server cancel / --cache-reuse). D is fixed, not K-dependent, so plain decode and
MTP chunk a prefill identically.
Gates (gfx1201, 35B-A3B): probe P=2050 exact 0.0 and P=2048 rounding-level (1e-3) vs 0.65
without the tail; GATED_DELTA_NET 2/2 (48/48); prefill-logit vs the arm it replaces 0.0041 (fp32)
/ 0.0054 (bf16 chunked) and vs the exact sequential reference 0.004040 / 97.550 % (fp32) /
0.004408 / 97.055 % (bf16) -- no worse than the unpatched 0.004416 / 97.318 %; plain == MTP
byte-identical (64 tok and -n 3000, all four axes); recurrent depth 130 == 130, rollback 0 FAIL;
llama-bench pp512 -0.09 %, pp2048 -0.58 %, tg128 flat. gfx1151: 0.002389 / 98.069 % vs the
replaced arm, text unchanged. Kill switch GGML_CUDA_GDN_TAIL=0.
Fold (r95): rdna3-verify-fusions (PR ggml-org#192 + PR ggml-org#193, briansp2020): the verify-band Q8_1
pre-quantize fusions (residual ADD + rms_norm -> Q8_1, rms_norm -> Q8_1, gated unary -> Q8_1,
GLU -> Q8_1) and the direct GDN conv for 2..255-token verify batches are admitted on RDNA3_0
as well. Both gates become (RDNA4 || RDNA3_0), so RDNA4 and every other backend keep the
path they had and RDNA3_5 stays out. ggml_cuda_should_fuse_mul_mat_vec_q(..., verify_band)
is the single predicate the three Q8_1 fusions share, and its kernels are row-generic and
write the values quantize_q8_1 writes, so W = 1..8 stay bit-identical. The GDN conv keeps
its pre-existing single-device restriction. GGML_CUDA_FUSE_Q8_1_VERIFY=0 /
GGML_CUDA_FUSE_GDN_CONV_VERIFY=0 still turn them off.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 21, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 21, 2026
…r, QSA/indexer)
old-14's qwen4exp support plus old-15's qwen4exp/QSA/HC content, kept as ONE
block because QSA is intrinsic to the qwen4exp architecture (the old-15 QSA
memory wins, HC16 and qsa3 are load-bearing for the qwen4exp graph, and the
QSA/HC op dispatch is old-15):
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the
MTP draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm
policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, lightning-indexer.cu, with the CUDA dispatch
(GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE / INDEXER_FILL) and
their CPU references; the indexer cache in llama-memory-hybrid-idx.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (+ dispatch, supports_op, the
hc_combine_mix fusion) and the HC16 planar layout.
- the managed lazy reader (llama-lazy-reader) and the loader's managed_budget.
- the GDN beta-sigmoid fusion, the DSV4_HC split states in the Meta backend,
and the QSA/HC op definitions in ggml.h / ggml.c.
Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16,
GGML_CUDA_QSA_SLICES, GGML_CUDA_SPLICE_GATHER and the GDN_DBG_SKIP_* diagnostics once into values
rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call
overwrites, so a cached pointer is unsafe on Windows.
Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not
linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper.
Folded in from four contributor PRs on v12-097cf33d9-r80 (bsp2020 / briansp2020; block 10 also owns
the rope + FWHT fusion that ggml-org#172 and ggml-org#174 put here in r78):
- fused RMS_NORM + MUL + ROPE (+ FWHT) also for M-ROPE / IMROPE (PR ggml-org#175, 0001): the matcher admits
GGML_ROPE_TYPE_MROPE / IMROPE besides NORMAL / NEOX, and the kernel computes theta with rope_multi's
own section selection and float expressions (NEOX pairing, iw == i0, VISION still rejected). This is
the TODO item recorded against ggml-org#172: every Qwen3.x model runs rope_multi, so the fusion was inert on
the dense 27B and the 35B-A3B. Kill switch GGML_CUDA_DISABLE_ROPE_MROPE_FUSE=1.
- BF16 caches off the 5-op RMS_NORM + MUL + ROPE + VIEW + SET_ROWS fusion (PR ggml-org#175, 0002): that fusion
runs ggml_cuda_op_rms_norm_mul_rope_fused, which writes F32 / F16 rows only and otherwise aborts, but
ggml_cuda_should_fuse_rope_set_rows admits a BF16 SET_ROWS. Latent on r80 for a NEOX qk-norm model
with -ctk bf16, and reachable for the Qwen3.x models once 0001 lands. The 5-op match now declines a
BF16 SET_ROWS; the RMS_NORM + MUL + ROPE fusion and the plain SET_ROWS take it instead.
- the q8_0 K and V cache writes folded into the rope + FWHT kernel (PR ggml-org#173): with a q8_0 cache the K
path ends MUL_MAT(Hadamard), RESHAPE, VIEW, SET_ROWS and the V write (VIEW, SET_ROWS) follows it, so
the fused kernel now writes both cache rows through ggml_cuda_quantize_q8_0_group (k_set_rows_q8_0_group's
body, moved into set-rows.cuh). Only the K write is folded inside a concurrent-stream region
(GGML_CUDA_GRAPH_OPT=1). Kill switch GGML_CUDA_DISABLE_ROPE_FWHT_SET_ROWS=1.
- the GDN q and k l2 norms in one launch (PR ggml-org#179): build_gdn_l2_norm normalizes both with the identical
RMS_NORM + SCALE over views of the same conv output, so rms_norm_scale_pair_f32<256> takes blocks
[0, nrows) for q and [nrows, 2*nrows) for k with the same per-row body. Matcher additions: only
no-op nodes between the two pairs, both inputs F32 views of the same view_src with equal ne / nb,
equal eps and scale params, and the second input must not read the first pair's output.
Kill switch GGML_CUDA_DISABLE_NORM_PAIR=1.
Rejected in the same review: PR ggml-org#177 (GDN direct conv also writes the recurrent snapshot tails). It is
not run-to-run deterministic on the 35B-A3B prefill under HIP graph capture and does not reproduce the
stock logits there; record in archive/work/gdn-conv-snapshot-fold/.
Prefill cache staging hook (r83): ggml_backend_cuda_stage_from_cache is live again for the expert-cache
arena. The scheduler's whole-table staging copy of a host-resident expert table (stage_upload for
-sm layer, stage_gather for the -sm tensor meta split) is filled by one kernel from the arena (resident
experts, D2D) + the pinned host master (the rest, H2D) instead of a full H2D re-upload; the destination
is the ordinary staged table in expert order, so every consumer is unchanged. It declines to the plain
upload for an unusable/stood-down table, an unpinned host master or a geometry mismatch. Kill switch
GGML_MOE_CACHE_STAGE=0.
Slab narrow band (r85): the always-resident narrow band was an estimate taken before the first ring carve
(a `split_only` reserve of the `n_rs_batch` shape, 1792 MiB on the field config), and `set_narrow_floor`
cannot correct it afterwards because the ring's VA is frozen once it holds blocks. The real MTP
decode/verify graph needs 2304 MiB, so `work_alloc` classified every decode view as wide, placed it above
the H2D staging ring and evicted 3.65 GiB/device of arena. `work_alloc` now learns the band from the real
graph: a view that is not a requested wide (prefill) pass is narrow even above the floor while no wide view
is live (the ring region is only carved while one is), and the floor + ring are re-carved from it (the ring
is idle there). With a wide view live the above-the-ring placement is kept: the ring's VA must not move.
A refused raise is reported instead of dropped silently.
Measured (2x R9700, Flash-Next IQ4_NL -ncmoe 46, 131k ctx, -b/-ub 6144, MTP n6): 18 -> 1 re-arm, decode
23.6 -> 45.4 t/s (-sm tensor) / 35.8 (-sm layer), prefill unchanged; cost: 16k prefill 1443 -> 1411 t/s
(-2.2 %) because the correctly sized band moves the ring up 512 MiB.
The "may the band move" test is `ring_armed` alone: a live WIDE view is the only thing that makes the ring
region part of the reserved layout (`ring_armed` is set exactly when the boundary covers the carved ring).
`ring_used` is a bump cursor that stays at the end after a prefill, and `boundary > narrow_floor` is true
for an unrelated narrow view one chunk above the floor; treating either as "the ring may not move" left
the band unlearned on a single device -- measured gfx1100 `-ncmoe 40`: floor 1280 MiB against a 1536 MiB
decode view, one arena eviction per prefill -> decode transition. With this test the band is learned
there too (floor 1280 -> 1536, view placed at base 0, no eviction) and the two gfx1100 greedy hashes are
unchanged.
Fold (r86): beta 12, slab-transient-absorption (CUDA/slab half). The wide-phase
auxiliary region hosts the FA staging arena, the QSA3 workspace and the chunked-GDN
scratch; the workspace pool serves its large prefill temporaries from it; a graph
capture is never started while the wide band is armed; per-device free=/free_min/
outside_max/outside_failed diagnostics; default headroom 640 + 128 x n_gpu MiB.
Fold (r91): the chunked GDN prefill writes its rollback snapshots (fixed-D sequential tail).
The chunked kernel computes no per-token states, so a partial rollback into a batch it served
restored a state it never wrote (probe: logit NMSE 0.65, maxabs 12.5; see
archive/work/rs-rollback-boundary/). A single-sequence batch above the chunked threshold now
runs the chunked kernel on its first n_tokens - D tokens and the SEQUENTIAL kernel on the last
D = max(K>16?K:16, n_rs_batch) tokens in the same launch, so snapshot slots 0..D-1 exist for
every batch. No extra memory, no extra graph, ubatch unchanged (splitting the ubatch instead
measured -13.7 % pp512 and was reverted).
The bound is structural, not observational: cparams.n_rs_batch = the widest verify batch of any
enabled speculator (common_speculative_n_max() + 1, all draft and ngram types) and is carried
into this op as param 1, so a verify batch is never chunked; the tail covers the non-speculator
partial rewind (server cancel / --cache-reuse). D is fixed, not K-dependent, so plain decode and
MTP chunk a prefill identically.
Gates (gfx1201, 35B-A3B): probe P=2050 exact 0.0 and P=2048 rounding-level (1e-3) vs 0.65
without the tail; GATED_DELTA_NET 2/2 (48/48); prefill-logit vs the arm it replaces 0.0041 (fp32)
/ 0.0054 (bf16 chunked) and vs the exact sequential reference 0.004040 / 97.550 % (fp32) /
0.004408 / 97.055 % (bf16) -- no worse than the unpatched 0.004416 / 97.318 %; plain == MTP
byte-identical (64 tok and -n 3000, all four axes); recurrent depth 130 == 130, rollback 0 FAIL;
llama-bench pp512 -0.09 %, pp2048 -0.58 %, tg128 flat. gfx1151: 0.002389 / 98.069 % vs the
replaced arm, text unchanged. Kill switch GGML_CUDA_GDN_TAIL=0.
Fold (r95): rdna3-verify-fusions (PR ggml-org#192 + PR ggml-org#193, briansp2020): the verify-band Q8_1
pre-quantize fusions (residual ADD + rms_norm -> Q8_1, rms_norm -> Q8_1, gated unary -> Q8_1,
GLU -> Q8_1) and the direct GDN conv for 2..255-token verify batches are admitted on RDNA3_0
as well. Both gates become (RDNA4 || RDNA3_0), so RDNA4 and every other backend keep the
path they had and RDNA3_5 stays out. ggml_cuda_should_fuse_mul_mat_vec_q(..., verify_band)
is the single predicate the three Q8_1 fusions share, and its kernels are row-generic and
write the values quantize_q8_1 writes, so W = 1..8 stay bit-identical. The GDN conv keeps
its pre-existing single-device restriction. GGML_CUDA_FUSE_Q8_1_VERIFY=0 /
GGML_CUDA_FUSE_GDN_CONV_VERIFY=0 still turn them off.
GDN / attention-gate decode fusions, reported by @briansp2020 (PR ggml-org#200; R9700
gfx1201). Four bit-identical fusions, each with its own kill switch:
- GGML_CUDA_FUSE_GATE_BETA_BF16 (mmvf): qwen4exp's alpha/beta BF16 projections
and softplus(alpha + dt) * a in one mul_mat_vec_f launch (alpha rows, then
beta rows; each row runs the instantiation the unfused MUL_MAT takes, so both
outputs are bit-identical). The Q8_0 pair keeps ssm_gate_beta.
- GGML_CUDA_FUSE_GDN_CONV_NORM (gdn-conv): the q/k l2 norm pair folded into the
direct conv kernel, same reduction values and lanes as
rms_norm_scale_pair_f32<256>. Taken only when both SCALE outputs are disjoint
from the conv's x/state/weight/output: the allocator may place them over the
state or x (dead after the CONCAT in graph order) which the direct kernel
still reads, and without that check 35B-A3B gave run-to-run varying output.
- GGML_CUDA_FUSE_CONT_GATE_Q8_1 (unary): the attention gate view is read in
place by the gate -> Q8_1 kernel, so the CONT launches nothing.
- GGML_CUDA_FUSE_GDN_NORM_GATE (unary): qwen4exp's GDN output RMS_NORM * w is
computed inside the gate kernel, one row per 256-thread block exactly as
rms_norm_f32<256, true> runs it.
The norm's node lives several nodes before the gate node that computes it, so
the skip and the emit are two decisions joined by a context-pending pointer.
The submitted form guarded that with two GGML_ABORTs (norm never computed / gate
node never reached). This copy adds the structural guard instead: while a norm
is pending, no other fusion may start before its gate node, and the pending
norm's own MUL (dead: the gate kernel reads xn and w directly) is skipped. The
window is measured as span 4 with a fixed signature on the only graph this fires
on (qwen4exp), so the window nodes run unfused at the cost of one plain matmul
per GDN layer, and the aborts are unreachable rather than merely unlikely.
Gates (r101 base, gfx1201, ROCBLAS_USE_HIPBLASLT=0): greedy text identical in
five arms (27B plain and MTP n3, 35B-A3B 1 GPU and 2 GPU tensor, Flash-Next
IQ3_XXS 2 GPU tensor + shared MTP); test-logits-width-probe hashes identical to
r101; prefill-logit 0.000707 / 98.755 % (the r101 numbers exactly); SSM_CONV /
RMS_NORM / SCALE / MUL_MAT oracles 0 failures; rule-5 batched decode within
noise at B=1/4/8. Decode: 27B MTP +0.2-0.3 %, Flash-Next 2 GPU tensor +0.7 %,
35B-A3B 1 GPU tg128 -0.2 % (i.e. flat; the submitter's +0.5 % did not
reproduce). No GPU faults.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 21, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 21, 2026
…r, QSA/indexer)
old-14's qwen4exp support plus old-15's qwen4exp/QSA/HC content, kept as ONE
block because QSA is intrinsic to the qwen4exp architecture (the old-15 QSA
memory wins, HC16 and qsa3 are load-bearing for the qwen4exp graph, and the
QSA/HC op dispatch is old-15):
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the
MTP draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm
policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, lightning-indexer.cu, with the CUDA dispatch
(GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE / INDEXER_FILL) and
their CPU references; the indexer cache in llama-memory-hybrid-idx.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (+ dispatch, supports_op, the
hc_combine_mix fusion) and the HC16 planar layout.
- the managed lazy reader (llama-lazy-reader) and the loader's managed_budget.
- the GDN beta-sigmoid fusion, the DSV4_HC split states in the Meta backend,
and the QSA/HC op definitions in ggml.h / ggml.c.
Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16,
GGML_CUDA_QSA_SLICES, GGML_CUDA_SPLICE_GATHER and the GDN_DBG_SKIP_* diagnostics once into values
rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call
overwrites, so a cached pointer is unsafe on Windows.
Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not
linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper.
Folded in from four contributor PRs on v12-097cf33d9-r80 (bsp2020 / briansp2020; block 10 also owns
the rope + FWHT fusion that ggml-org#172 and ggml-org#174 put here in r78):
- fused RMS_NORM + MUL + ROPE (+ FWHT) also for M-ROPE / IMROPE (PR ggml-org#175, 0001): the matcher admits
GGML_ROPE_TYPE_MROPE / IMROPE besides NORMAL / NEOX, and the kernel computes theta with rope_multi's
own section selection and float expressions (NEOX pairing, iw == i0, VISION still rejected). This is
the TODO item recorded against ggml-org#172: every Qwen3.x model runs rope_multi, so the fusion was inert on
the dense 27B and the 35B-A3B. Kill switch GGML_CUDA_DISABLE_ROPE_MROPE_FUSE=1.
- BF16 caches off the 5-op RMS_NORM + MUL + ROPE + VIEW + SET_ROWS fusion (PR ggml-org#175, 0002): that fusion
runs ggml_cuda_op_rms_norm_mul_rope_fused, which writes F32 / F16 rows only and otherwise aborts, but
ggml_cuda_should_fuse_rope_set_rows admits a BF16 SET_ROWS. Latent on r80 for a NEOX qk-norm model
with -ctk bf16, and reachable for the Qwen3.x models once 0001 lands. The 5-op match now declines a
BF16 SET_ROWS; the RMS_NORM + MUL + ROPE fusion and the plain SET_ROWS take it instead.
- the q8_0 K and V cache writes folded into the rope + FWHT kernel (PR ggml-org#173): with a q8_0 cache the K
path ends MUL_MAT(Hadamard), RESHAPE, VIEW, SET_ROWS and the V write (VIEW, SET_ROWS) follows it, so
the fused kernel now writes both cache rows through ggml_cuda_quantize_q8_0_group (k_set_rows_q8_0_group's
body, moved into set-rows.cuh). Only the K write is folded inside a concurrent-stream region
(GGML_CUDA_GRAPH_OPT=1). Kill switch GGML_CUDA_DISABLE_ROPE_FWHT_SET_ROWS=1.
- the GDN q and k l2 norms in one launch (PR ggml-org#179): build_gdn_l2_norm normalizes both with the identical
RMS_NORM + SCALE over views of the same conv output, so rms_norm_scale_pair_f32<256> takes blocks
[0, nrows) for q and [nrows, 2*nrows) for k with the same per-row body. Matcher additions: only
no-op nodes between the two pairs, both inputs F32 views of the same view_src with equal ne / nb,
equal eps and scale params, and the second input must not read the first pair's output.
Kill switch GGML_CUDA_DISABLE_NORM_PAIR=1.
Rejected in the same review: PR ggml-org#177 (GDN direct conv also writes the recurrent snapshot tails). It is
not run-to-run deterministic on the 35B-A3B prefill under HIP graph capture and does not reproduce the
stock logits there; record in archive/work/gdn-conv-snapshot-fold/.
Prefill cache staging hook (r83): ggml_backend_cuda_stage_from_cache is live again for the expert-cache
arena. The scheduler's whole-table staging copy of a host-resident expert table (stage_upload for
-sm layer, stage_gather for the -sm tensor meta split) is filled by one kernel from the arena (resident
experts, D2D) + the pinned host master (the rest, H2D) instead of a full H2D re-upload; the destination
is the ordinary staged table in expert order, so every consumer is unchanged. It declines to the plain
upload for an unusable/stood-down table, an unpinned host master or a geometry mismatch. Kill switch
GGML_MOE_CACHE_STAGE=0.
Slab narrow band (r85): the always-resident narrow band was an estimate taken before the first ring carve
(a `split_only` reserve of the `n_rs_batch` shape, 1792 MiB on the field config), and `set_narrow_floor`
cannot correct it afterwards because the ring's VA is frozen once it holds blocks. The real MTP
decode/verify graph needs 2304 MiB, so `work_alloc` classified every decode view as wide, placed it above
the H2D staging ring and evicted 3.65 GiB/device of arena. `work_alloc` now learns the band from the real
graph: a view that is not a requested wide (prefill) pass is narrow even above the floor while no wide view
is live (the ring region is only carved while one is), and the floor + ring are re-carved from it (the ring
is idle there). With a wide view live the above-the-ring placement is kept: the ring's VA must not move.
A refused raise is reported instead of dropped silently.
Measured (2x R9700, Flash-Next IQ4_NL -ncmoe 46, 131k ctx, -b/-ub 6144, MTP n6): 18 -> 1 re-arm, decode
23.6 -> 45.4 t/s (-sm tensor) / 35.8 (-sm layer), prefill unchanged; cost: 16k prefill 1443 -> 1411 t/s
(-2.2 %) because the correctly sized band moves the ring up 512 MiB.
The "may the band move" test is `ring_armed` alone: a live WIDE view is the only thing that makes the ring
region part of the reserved layout (`ring_armed` is set exactly when the boundary covers the carved ring).
`ring_used` is a bump cursor that stays at the end after a prefill, and `boundary > narrow_floor` is true
for an unrelated narrow view one chunk above the floor; treating either as "the ring may not move" left
the band unlearned on a single device -- measured gfx1100 `-ncmoe 40`: floor 1280 MiB against a 1536 MiB
decode view, one arena eviction per prefill -> decode transition. With this test the band is learned
there too (floor 1280 -> 1536, view placed at base 0, no eviction) and the two gfx1100 greedy hashes are
unchanged.
Fold (r86): beta 12, slab-transient-absorption (CUDA/slab half). The wide-phase
auxiliary region hosts the FA staging arena, the QSA3 workspace and the chunked-GDN
scratch; the workspace pool serves its large prefill temporaries from it; a graph
capture is never started while the wide band is armed; per-device free=/free_min/
outside_max/outside_failed diagnostics; default headroom 640 + 128 x n_gpu MiB.
Fold (r91): the chunked GDN prefill writes its rollback snapshots (fixed-D sequential tail).
The chunked kernel computes no per-token states, so a partial rollback into a batch it served
restored a state it never wrote (probe: logit NMSE 0.65, maxabs 12.5; see
archive/work/rs-rollback-boundary/). A single-sequence batch above the chunked threshold now
runs the chunked kernel on its first n_tokens - D tokens and the SEQUENTIAL kernel on the last
D = max(K>16?K:16, n_rs_batch) tokens in the same launch, so snapshot slots 0..D-1 exist for
every batch. No extra memory, no extra graph, ubatch unchanged (splitting the ubatch instead
measured -13.7 % pp512 and was reverted).
The bound is structural, not observational: cparams.n_rs_batch = the widest verify batch of any
enabled speculator (common_speculative_n_max() + 1, all draft and ngram types) and is carried
into this op as param 1, so a verify batch is never chunked; the tail covers the non-speculator
partial rewind (server cancel / --cache-reuse). D is fixed, not K-dependent, so plain decode and
MTP chunk a prefill identically.
Gates (gfx1201, 35B-A3B): probe P=2050 exact 0.0 and P=2048 rounding-level (1e-3) vs 0.65
without the tail; GATED_DELTA_NET 2/2 (48/48); prefill-logit vs the arm it replaces 0.0041 (fp32)
/ 0.0054 (bf16 chunked) and vs the exact sequential reference 0.004040 / 97.550 % (fp32) /
0.004408 / 97.055 % (bf16) -- no worse than the unpatched 0.004416 / 97.318 %; plain == MTP
byte-identical (64 tok and -n 3000, all four axes); recurrent depth 130 == 130, rollback 0 FAIL;
llama-bench pp512 -0.09 %, pp2048 -0.58 %, tg128 flat. gfx1151: 0.002389 / 98.069 % vs the
replaced arm, text unchanged. Kill switch GGML_CUDA_GDN_TAIL=0.
Fold (r95): rdna3-verify-fusions (PR ggml-org#192 + PR ggml-org#193, briansp2020): the verify-band Q8_1
pre-quantize fusions (residual ADD + rms_norm -> Q8_1, rms_norm -> Q8_1, gated unary -> Q8_1,
GLU -> Q8_1) and the direct GDN conv for 2..255-token verify batches are admitted on RDNA3_0
as well. Both gates become (RDNA4 || RDNA3_0), so RDNA4 and every other backend keep the
path they had and RDNA3_5 stays out. ggml_cuda_should_fuse_mul_mat_vec_q(..., verify_band)
is the single predicate the three Q8_1 fusions share, and its kernels are row-generic and
write the values quantize_q8_1 writes, so W = 1..8 stay bit-identical. The GDN conv keeps
its pre-existing single-device restriction. GGML_CUDA_FUSE_Q8_1_VERIFY=0 /
GGML_CUDA_FUSE_GDN_CONV_VERIFY=0 still turn them off.
GDN / attention-gate decode fusions, reported by @briansp2020 (PR ggml-org#200; R9700
gfx1201). Four bit-identical fusions, each with its own kill switch:
- GGML_CUDA_FUSE_GATE_BETA_BF16 (mmvf): qwen4exp's alpha/beta BF16 projections
and softplus(alpha + dt) * a in one mul_mat_vec_f launch (alpha rows, then
beta rows; each row runs the instantiation the unfused MUL_MAT takes, so both
outputs are bit-identical). The Q8_0 pair keeps ssm_gate_beta.
- GGML_CUDA_FUSE_GDN_CONV_NORM (gdn-conv): the q/k l2 norm pair folded into the
direct conv kernel, same reduction values and lanes as
rms_norm_scale_pair_f32<256>. Taken only when both SCALE outputs are disjoint
from the conv's x/state/weight/output: the allocator may place them over the
state or x (dead after the CONCAT in graph order) which the direct kernel
still reads, and without that check 35B-A3B gave run-to-run varying output.
- GGML_CUDA_FUSE_CONT_GATE_Q8_1 (unary): the attention gate view is read in
place by the gate -> Q8_1 kernel, so the CONT launches nothing.
- GGML_CUDA_FUSE_GDN_NORM_GATE (unary): qwen4exp's GDN output RMS_NORM * w is
computed inside the gate kernel, one row per 256-thread block exactly as
rms_norm_f32<256, true> runs it.
The norm's node lives several nodes before the gate node that computes it, so
the skip and the emit are two decisions joined by a context-pending pointer.
The submitted form guarded that with two GGML_ABORTs (norm never computed / gate
node never reached). This copy adds the structural guard instead: while a norm
is pending, no other fusion may start before its gate node, and the pending
norm's own MUL (dead: the gate kernel reads xn and w directly) is skipped. The
window is measured as span 4 with a fixed signature on the only graph this fires
on (qwen4exp), so the window nodes run unfused at the cost of one plain matmul
per GDN layer, and the aborts are unreachable rather than merely unlikely.
Gates (r101 base, gfx1201, ROCBLAS_USE_HIPBLASLT=0): greedy text identical in
five arms (27B plain and MTP n3, 35B-A3B 1 GPU and 2 GPU tensor, Flash-Next
IQ3_XXS 2 GPU tensor + shared MTP); test-logits-width-probe hashes identical to
r101; prefill-logit 0.000707 / 98.755 % (the r101 numbers exactly); SSM_CONV /
RMS_NORM / SCALE / MUL_MAT oracles 0 failures; rule-5 batched decode within
noise at B=1/4/8. Decode: 27B MTP +0.2-0.3 %, Flash-Next 2 GPU tensor +0.7 %,
35B-A3B 1 GPU tg128 -0.2 % (i.e. flat; the submitter's +0.5 % did not
reproduce). No GPU faults.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 21, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 21, 2026
…r, QSA/indexer)
old-14's qwen4exp support plus old-15's qwen4exp/QSA/HC content, kept as ONE
block because QSA is intrinsic to the qwen4exp architecture (the old-15 QSA
memory wins, HC16 and qsa3 are load-bearing for the qwen4exp graph, and the
QSA/HC op dispatch is old-15):
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the
MTP draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm
policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, lightning-indexer.cu, with the CUDA dispatch
(GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE / INDEXER_FILL) and
their CPU references; the indexer cache in llama-memory-hybrid-idx.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (+ dispatch, supports_op, the
hc_combine_mix fusion) and the HC16 planar layout.
- the managed lazy reader (llama-lazy-reader) and the loader's managed_budget.
- the GDN beta-sigmoid fusion, the DSV4_HC split states in the Meta backend,
and the QSA/HC op definitions in ggml.h / ggml.c.
Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16,
GGML_CUDA_QSA_SLICES, GGML_CUDA_SPLICE_GATHER and the GDN_DBG_SKIP_* diagnostics once into values
rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call
overwrites, so a cached pointer is unsafe on Windows.
Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not
linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper.
Folded in from four contributor PRs on v12-097cf33d9-r80 (bsp2020 / briansp2020; block 10 also owns
the rope + FWHT fusion that ggml-org#172 and ggml-org#174 put here in r78):
- fused RMS_NORM + MUL + ROPE (+ FWHT) also for M-ROPE / IMROPE (PR ggml-org#175, 0001): the matcher admits
GGML_ROPE_TYPE_MROPE / IMROPE besides NORMAL / NEOX, and the kernel computes theta with rope_multi's
own section selection and float expressions (NEOX pairing, iw == i0, VISION still rejected). This is
the TODO item recorded against ggml-org#172: every Qwen3.x model runs rope_multi, so the fusion was inert on
the dense 27B and the 35B-A3B. Kill switch GGML_CUDA_DISABLE_ROPE_MROPE_FUSE=1.
- BF16 caches off the 5-op RMS_NORM + MUL + ROPE + VIEW + SET_ROWS fusion (PR ggml-org#175, 0002): that fusion
runs ggml_cuda_op_rms_norm_mul_rope_fused, which writes F32 / F16 rows only and otherwise aborts, but
ggml_cuda_should_fuse_rope_set_rows admits a BF16 SET_ROWS. Latent on r80 for a NEOX qk-norm model
with -ctk bf16, and reachable for the Qwen3.x models once 0001 lands. The 5-op match now declines a
BF16 SET_ROWS; the RMS_NORM + MUL + ROPE fusion and the plain SET_ROWS take it instead.
- the q8_0 K and V cache writes folded into the rope + FWHT kernel (PR ggml-org#173): with a q8_0 cache the K
path ends MUL_MAT(Hadamard), RESHAPE, VIEW, SET_ROWS and the V write (VIEW, SET_ROWS) follows it, so
the fused kernel now writes both cache rows through ggml_cuda_quantize_q8_0_group (k_set_rows_q8_0_group's
body, moved into set-rows.cuh). Only the K write is folded inside a concurrent-stream region
(GGML_CUDA_GRAPH_OPT=1). Kill switch GGML_CUDA_DISABLE_ROPE_FWHT_SET_ROWS=1.
- the GDN q and k l2 norms in one launch (PR ggml-org#179): build_gdn_l2_norm normalizes both with the identical
RMS_NORM + SCALE over views of the same conv output, so rms_norm_scale_pair_f32<256> takes blocks
[0, nrows) for q and [nrows, 2*nrows) for k with the same per-row body. Matcher additions: only
no-op nodes between the two pairs, both inputs F32 views of the same view_src with equal ne / nb,
equal eps and scale params, and the second input must not read the first pair's output.
Kill switch GGML_CUDA_DISABLE_NORM_PAIR=1.
Rejected in the same review: PR ggml-org#177 (GDN direct conv also writes the recurrent snapshot tails). It is
not run-to-run deterministic on the 35B-A3B prefill under HIP graph capture and does not reproduce the
stock logits there; record in archive/work/gdn-conv-snapshot-fold/.
Prefill cache staging hook (r83): ggml_backend_cuda_stage_from_cache is live again for the expert-cache
arena. The scheduler's whole-table staging copy of a host-resident expert table (stage_upload for
-sm layer, stage_gather for the -sm tensor meta split) is filled by one kernel from the arena (resident
experts, D2D) + the pinned host master (the rest, H2D) instead of a full H2D re-upload; the destination
is the ordinary staged table in expert order, so every consumer is unchanged. It declines to the plain
upload for an unusable/stood-down table, an unpinned host master or a geometry mismatch. Kill switch
GGML_MOE_CACHE_STAGE=0.
Slab narrow band (r85): the always-resident narrow band was an estimate taken before the first ring carve
(a `split_only` reserve of the `n_rs_batch` shape, 1792 MiB on the field config), and `set_narrow_floor`
cannot correct it afterwards because the ring's VA is frozen once it holds blocks. The real MTP
decode/verify graph needs 2304 MiB, so `work_alloc` classified every decode view as wide, placed it above
the H2D staging ring and evicted 3.65 GiB/device of arena. `work_alloc` now learns the band from the real
graph: a view that is not a requested wide (prefill) pass is narrow even above the floor while no wide view
is live (the ring region is only carved while one is), and the floor + ring are re-carved from it (the ring
is idle there). With a wide view live the above-the-ring placement is kept: the ring's VA must not move.
A refused raise is reported instead of dropped silently.
Measured (2x R9700, Flash-Next IQ4_NL -ncmoe 46, 131k ctx, -b/-ub 6144, MTP n6): 18 -> 1 re-arm, decode
23.6 -> 45.4 t/s (-sm tensor) / 35.8 (-sm layer), prefill unchanged; cost: 16k prefill 1443 -> 1411 t/s
(-2.2 %) because the correctly sized band moves the ring up 512 MiB.
The "may the band move" test is `ring_armed` alone: a live WIDE view is the only thing that makes the ring
region part of the reserved layout (`ring_armed` is set exactly when the boundary covers the carved ring).
`ring_used` is a bump cursor that stays at the end after a prefill, and `boundary > narrow_floor` is true
for an unrelated narrow view one chunk above the floor; treating either as "the ring may not move" left
the band unlearned on a single device -- measured gfx1100 `-ncmoe 40`: floor 1280 MiB against a 1536 MiB
decode view, one arena eviction per prefill -> decode transition. With this test the band is learned
there too (floor 1280 -> 1536, view placed at base 0, no eviction) and the two gfx1100 greedy hashes are
unchanged.
Fold (r86): beta 12, slab-transient-absorption (CUDA/slab half). The wide-phase
auxiliary region hosts the FA staging arena, the QSA3 workspace and the chunked-GDN
scratch; the workspace pool serves its large prefill temporaries from it; a graph
capture is never started while the wide band is armed; per-device free=/free_min/
outside_max/outside_failed diagnostics; default headroom 640 + 128 x n_gpu MiB.
Fold (r91): the chunked GDN prefill writes its rollback snapshots (fixed-D sequential tail).
The chunked kernel computes no per-token states, so a partial rollback into a batch it served
restored a state it never wrote (probe: logit NMSE 0.65, maxabs 12.5; see
archive/work/rs-rollback-boundary/). A single-sequence batch above the chunked threshold now
runs the chunked kernel on its first n_tokens - D tokens and the SEQUENTIAL kernel on the last
D = max(K>16?K:16, n_rs_batch) tokens in the same launch, so snapshot slots 0..D-1 exist for
every batch. No extra memory, no extra graph, ubatch unchanged (splitting the ubatch instead
measured -13.7 % pp512 and was reverted).
The bound is structural, not observational: cparams.n_rs_batch = the widest verify batch of any
enabled speculator (common_speculative_n_max() + 1, all draft and ngram types) and is carried
into this op as param 1, so a verify batch is never chunked; the tail covers the non-speculator
partial rewind (server cancel / --cache-reuse). D is fixed, not K-dependent, so plain decode and
MTP chunk a prefill identically.
Gates (gfx1201, 35B-A3B): probe P=2050 exact 0.0 and P=2048 rounding-level (1e-3) vs 0.65
without the tail; GATED_DELTA_NET 2/2 (48/48); prefill-logit vs the arm it replaces 0.0041 (fp32)
/ 0.0054 (bf16 chunked) and vs the exact sequential reference 0.004040 / 97.550 % (fp32) /
0.004408 / 97.055 % (bf16) -- no worse than the unpatched 0.004416 / 97.318 %; plain == MTP
byte-identical (64 tok and -n 3000, all four axes); recurrent depth 130 == 130, rollback 0 FAIL;
llama-bench pp512 -0.09 %, pp2048 -0.58 %, tg128 flat. gfx1151: 0.002389 / 98.069 % vs the
replaced arm, text unchanged. Kill switch GGML_CUDA_GDN_TAIL=0.
Fold (r95): rdna3-verify-fusions (PR ggml-org#192 + PR ggml-org#193, briansp2020): the verify-band Q8_1
pre-quantize fusions (residual ADD + rms_norm -> Q8_1, rms_norm -> Q8_1, gated unary -> Q8_1,
GLU -> Q8_1) and the direct GDN conv for 2..255-token verify batches are admitted on RDNA3_0
as well. Both gates become (RDNA4 || RDNA3_0), so RDNA4 and every other backend keep the
path they had and RDNA3_5 stays out. ggml_cuda_should_fuse_mul_mat_vec_q(..., verify_band)
is the single predicate the three Q8_1 fusions share, and its kernels are row-generic and
write the values quantize_q8_1 writes, so W = 1..8 stay bit-identical. The GDN conv keeps
its pre-existing single-device restriction. GGML_CUDA_FUSE_Q8_1_VERIFY=0 /
GGML_CUDA_FUSE_GDN_CONV_VERIFY=0 still turn them off.
GDN / attention-gate decode fusions, reported by @briansp2020 (PR ggml-org#200; R9700
gfx1201). Four bit-identical fusions, each with its own kill switch:
- GGML_CUDA_FUSE_GATE_BETA_BF16 (mmvf): qwen4exp's alpha/beta BF16 projections
and softplus(alpha + dt) * a in one mul_mat_vec_f launch (alpha rows, then
beta rows; each row runs the instantiation the unfused MUL_MAT takes, so both
outputs are bit-identical). The Q8_0 pair keeps ssm_gate_beta.
- GGML_CUDA_FUSE_GDN_CONV_NORM (gdn-conv): the q/k l2 norm pair folded into the
direct conv kernel, same reduction values and lanes as
rms_norm_scale_pair_f32<256>. Taken only when both SCALE outputs are disjoint
from the conv's x/state/weight/output: the allocator may place them over the
state or x (dead after the CONCAT in graph order) which the direct kernel
still reads, and without that check 35B-A3B gave run-to-run varying output.
- GGML_CUDA_FUSE_CONT_GATE_Q8_1 (unary): the attention gate view is read in
place by the gate -> Q8_1 kernel, so the CONT launches nothing.
- GGML_CUDA_FUSE_GDN_NORM_GATE (unary): qwen4exp's GDN output RMS_NORM * w is
computed inside the gate kernel, one row per 256-thread block exactly as
rms_norm_f32<256, true> runs it.
The norm's node lives several nodes before the gate node that computes it, so
the skip and the emit are two decisions joined by a context-pending pointer.
The submitted form guarded that with two GGML_ABORTs (norm never computed / gate
node never reached). This copy adds the structural guard instead: while a norm
is pending, no other fusion may start before its gate node, and the pending
norm's own MUL (dead: the gate kernel reads xn and w directly) is skipped. The
window is measured as span 4 with a fixed signature on the only graph this fires
on (qwen4exp), so the window nodes run unfused at the cost of one plain matmul
per GDN layer, and the aborts are unreachable rather than merely unlikely.
Gates (r101 base, gfx1201, ROCBLAS_USE_HIPBLASLT=0): greedy text identical in
five arms (27B plain and MTP n3, 35B-A3B 1 GPU and 2 GPU tensor, Flash-Next
IQ3_XXS 2 GPU tensor + shared MTP); test-logits-width-probe hashes identical to
r101; prefill-logit 0.000707 / 98.755 % (the r101 numbers exactly); SSM_CONV /
RMS_NORM / SCALE / MUL_MAT oracles 0 failures; rule-5 batched decode within
noise at B=1/4/8. Decode: 27B MTP +0.2-0.3 %, Flash-Next 2 GPU tensor +0.7 %,
35B-A3B 1 GPU tg128 -0.2 % (i.e. flat; the submitter's +0.5 % did not
reproduce). No GPU faults.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 22, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 22, 2026
…r, QSA/indexer)
old-14's qwen4exp support plus old-15's qwen4exp/QSA/HC content, kept as ONE
block because QSA is intrinsic to the qwen4exp architecture (the old-15 QSA
memory wins, HC16 and qsa3 are load-bearing for the qwen4exp graph, and the
QSA/HC op dispatch is old-15):
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the
MTP draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm
policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, lightning-indexer.cu, with the CUDA dispatch
(GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE / INDEXER_FILL) and
their CPU references; the indexer cache in llama-memory-hybrid-idx.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (+ dispatch, supports_op, the
hc_combine_mix fusion) and the HC16 planar layout.
- the managed lazy reader (llama-lazy-reader) and the loader's managed_budget.
- the GDN beta-sigmoid fusion, the DSV4_HC split states in the Meta backend,
and the QSA/HC op definitions in ggml.h / ggml.c.
Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16,
GGML_CUDA_QSA_SLICES, GGML_CUDA_SPLICE_GATHER and the GDN_DBG_SKIP_* diagnostics once into values
rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call
overwrites, so a cached pointer is unsafe on Windows.
Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not
linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper.
Folded in from four contributor PRs on v12-097cf33d9-r80 (bsp2020 / briansp2020; block 10 also owns
the rope + FWHT fusion that ggml-org#172 and ggml-org#174 put here in r78):
- fused RMS_NORM + MUL + ROPE (+ FWHT) also for M-ROPE / IMROPE (PR ggml-org#175, 0001): the matcher admits
GGML_ROPE_TYPE_MROPE / IMROPE besides NORMAL / NEOX, and the kernel computes theta with rope_multi's
own section selection and float expressions (NEOX pairing, iw == i0, VISION still rejected). This is
the TODO item recorded against ggml-org#172: every Qwen3.x model runs rope_multi, so the fusion was inert on
the dense 27B and the 35B-A3B. Kill switch GGML_CUDA_DISABLE_ROPE_MROPE_FUSE=1.
- BF16 caches off the 5-op RMS_NORM + MUL + ROPE + VIEW + SET_ROWS fusion (PR ggml-org#175, 0002): that fusion
runs ggml_cuda_op_rms_norm_mul_rope_fused, which writes F32 / F16 rows only and otherwise aborts, but
ggml_cuda_should_fuse_rope_set_rows admits a BF16 SET_ROWS. Latent on r80 for a NEOX qk-norm model
with -ctk bf16, and reachable for the Qwen3.x models once 0001 lands. The 5-op match now declines a
BF16 SET_ROWS; the RMS_NORM + MUL + ROPE fusion and the plain SET_ROWS take it instead.
- the q8_0 K and V cache writes folded into the rope + FWHT kernel (PR ggml-org#173): with a q8_0 cache the K
path ends MUL_MAT(Hadamard), RESHAPE, VIEW, SET_ROWS and the V write (VIEW, SET_ROWS) follows it, so
the fused kernel now writes both cache rows through ggml_cuda_quantize_q8_0_group (k_set_rows_q8_0_group's
body, moved into set-rows.cuh). Only the K write is folded inside a concurrent-stream region
(GGML_CUDA_GRAPH_OPT=1). Kill switch GGML_CUDA_DISABLE_ROPE_FWHT_SET_ROWS=1.
- the GDN q and k l2 norms in one launch (PR ggml-org#179): build_gdn_l2_norm normalizes both with the identical
RMS_NORM + SCALE over views of the same conv output, so rms_norm_scale_pair_f32<256> takes blocks
[0, nrows) for q and [nrows, 2*nrows) for k with the same per-row body. Matcher additions: only
no-op nodes between the two pairs, both inputs F32 views of the same view_src with equal ne / nb,
equal eps and scale params, and the second input must not read the first pair's output.
Kill switch GGML_CUDA_DISABLE_NORM_PAIR=1.
Rejected in the same review: PR ggml-org#177 (GDN direct conv also writes the recurrent snapshot tails). It is
not run-to-run deterministic on the 35B-A3B prefill under HIP graph capture and does not reproduce the
stock logits there; record in archive/work/gdn-conv-snapshot-fold/.
Prefill cache staging hook (r83): ggml_backend_cuda_stage_from_cache is live again for the expert-cache
arena. The scheduler's whole-table staging copy of a host-resident expert table (stage_upload for
-sm layer, stage_gather for the -sm tensor meta split) is filled by one kernel from the arena (resident
experts, D2D) + the pinned host master (the rest, H2D) instead of a full H2D re-upload; the destination
is the ordinary staged table in expert order, so every consumer is unchanged. It declines to the plain
upload for an unusable/stood-down table, an unpinned host master or a geometry mismatch. Kill switch
GGML_MOE_CACHE_STAGE=0.
Slab narrow band (r85): the always-resident narrow band was an estimate taken before the first ring carve
(a `split_only` reserve of the `n_rs_batch` shape, 1792 MiB on the field config), and `set_narrow_floor`
cannot correct it afterwards because the ring's VA is frozen once it holds blocks. The real MTP
decode/verify graph needs 2304 MiB, so `work_alloc` classified every decode view as wide, placed it above
the H2D staging ring and evicted 3.65 GiB/device of arena. `work_alloc` now learns the band from the real
graph: a view that is not a requested wide (prefill) pass is narrow even above the floor while no wide view
is live (the ring region is only carved while one is), and the floor + ring are re-carved from it (the ring
is idle there). With a wide view live the above-the-ring placement is kept: the ring's VA must not move.
A refused raise is reported instead of dropped silently.
Measured (2x R9700, Flash-Next IQ4_NL -ncmoe 46, 131k ctx, -b/-ub 6144, MTP n6): 18 -> 1 re-arm, decode
23.6 -> 45.4 t/s (-sm tensor) / 35.8 (-sm layer), prefill unchanged; cost: 16k prefill 1443 -> 1411 t/s
(-2.2 %) because the correctly sized band moves the ring up 512 MiB.
The "may the band move" test is `ring_armed` alone: a live WIDE view is the only thing that makes the ring
region part of the reserved layout (`ring_armed` is set exactly when the boundary covers the carved ring).
`ring_used` is a bump cursor that stays at the end after a prefill, and `boundary > narrow_floor` is true
for an unrelated narrow view one chunk above the floor; treating either as "the ring may not move" left
the band unlearned on a single device -- measured gfx1100 `-ncmoe 40`: floor 1280 MiB against a 1536 MiB
decode view, one arena eviction per prefill -> decode transition. With this test the band is learned
there too (floor 1280 -> 1536, view placed at base 0, no eviction) and the two gfx1100 greedy hashes are
unchanged.
Fold (r86): beta 12, slab-transient-absorption (CUDA/slab half). The wide-phase
auxiliary region hosts the FA staging arena, the QSA3 workspace and the chunked-GDN
scratch; the workspace pool serves its large prefill temporaries from it; a graph
capture is never started while the wide band is armed; per-device free=/free_min/
outside_max/outside_failed diagnostics; default headroom 640 + 128 x n_gpu MiB.
Fold (r91): the chunked GDN prefill writes its rollback snapshots (fixed-D sequential tail).
The chunked kernel computes no per-token states, so a partial rollback into a batch it served
restored a state it never wrote (probe: logit NMSE 0.65, maxabs 12.5; see
archive/work/rs-rollback-boundary/). A single-sequence batch above the chunked threshold now
runs the chunked kernel on its first n_tokens - D tokens and the SEQUENTIAL kernel on the last
D = max(K>16?K:16, n_rs_batch) tokens in the same launch, so snapshot slots 0..D-1 exist for
every batch. No extra memory, no extra graph, ubatch unchanged (splitting the ubatch instead
measured -13.7 % pp512 and was reverted).
The bound is structural, not observational: cparams.n_rs_batch = the widest verify batch of any
enabled speculator (common_speculative_n_max() + 1, all draft and ngram types) and is carried
into this op as param 1, so a verify batch is never chunked; the tail covers the non-speculator
partial rewind (server cancel / --cache-reuse). D is fixed, not K-dependent, so plain decode and
MTP chunk a prefill identically.
Gates (gfx1201, 35B-A3B): probe P=2050 exact 0.0 and P=2048 rounding-level (1e-3) vs 0.65
without the tail; GATED_DELTA_NET 2/2 (48/48); prefill-logit vs the arm it replaces 0.0041 (fp32)
/ 0.0054 (bf16 chunked) and vs the exact sequential reference 0.004040 / 97.550 % (fp32) /
0.004408 / 97.055 % (bf16) -- no worse than the unpatched 0.004416 / 97.318 %; plain == MTP
byte-identical (64 tok and -n 3000, all four axes); recurrent depth 130 == 130, rollback 0 FAIL;
llama-bench pp512 -0.09 %, pp2048 -0.58 %, tg128 flat. gfx1151: 0.002389 / 98.069 % vs the
replaced arm, text unchanged. Kill switch GGML_CUDA_GDN_TAIL=0.
Fold (r95): rdna3-verify-fusions (PR ggml-org#192 + PR ggml-org#193, briansp2020): the verify-band Q8_1
pre-quantize fusions (residual ADD + rms_norm -> Q8_1, rms_norm -> Q8_1, gated unary -> Q8_1,
GLU -> Q8_1) and the direct GDN conv for 2..255-token verify batches are admitted on RDNA3_0
as well. Both gates become (RDNA4 || RDNA3_0), so RDNA4 and every other backend keep the
path they had and RDNA3_5 stays out. ggml_cuda_should_fuse_mul_mat_vec_q(..., verify_band)
is the single predicate the three Q8_1 fusions share, and its kernels are row-generic and
write the values quantize_q8_1 writes, so W = 1..8 stay bit-identical. The GDN conv keeps
its pre-existing single-device restriction. GGML_CUDA_FUSE_Q8_1_VERIFY=0 /
GGML_CUDA_FUSE_GDN_CONV_VERIFY=0 still turn them off.
GDN / attention-gate decode fusions, reported by @briansp2020 (PR ggml-org#200; R9700
gfx1201). Four bit-identical fusions, each with its own kill switch:
- GGML_CUDA_FUSE_GATE_BETA_BF16 (mmvf): qwen4exp's alpha/beta BF16 projections
and softplus(alpha + dt) * a in one mul_mat_vec_f launch (alpha rows, then
beta rows; each row runs the instantiation the unfused MUL_MAT takes, so both
outputs are bit-identical). The Q8_0 pair keeps ssm_gate_beta.
- GGML_CUDA_FUSE_GDN_CONV_NORM (gdn-conv): the q/k l2 norm pair folded into the
direct conv kernel, same reduction values and lanes as
rms_norm_scale_pair_f32<256>. Taken only when both SCALE outputs are disjoint
from the conv's x/state/weight/output: the allocator may place them over the
state or x (dead after the CONCAT in graph order) which the direct kernel
still reads, and without that check 35B-A3B gave run-to-run varying output.
- GGML_CUDA_FUSE_CONT_GATE_Q8_1 (unary): the attention gate view is read in
place by the gate -> Q8_1 kernel, so the CONT launches nothing.
- GGML_CUDA_FUSE_GDN_NORM_GATE (unary): qwen4exp's GDN output RMS_NORM * w is
computed inside the gate kernel, one row per 256-thread block exactly as
rms_norm_f32<256, true> runs it.
The norm's node lives several nodes before the gate node that computes it, so
the skip and the emit are two decisions joined by a context-pending pointer.
The submitted form guarded that with two GGML_ABORTs (norm never computed / gate
node never reached). This copy adds the structural guard instead: while a norm
is pending, no other fusion may start before its gate node, and the pending
norm's own MUL (dead: the gate kernel reads xn and w directly) is skipped. The
window is measured as span 4 with a fixed signature on the only graph this fires
on (qwen4exp), so the window nodes run unfused at the cost of one plain matmul
per GDN layer, and the aborts are unreachable rather than merely unlikely.
Gates (r101 base, gfx1201, ROCBLAS_USE_HIPBLASLT=0): greedy text identical in
five arms (27B plain and MTP n3, 35B-A3B 1 GPU and 2 GPU tensor, Flash-Next
IQ3_XXS 2 GPU tensor + shared MTP); test-logits-width-probe hashes identical to
r101; prefill-logit 0.000707 / 98.755 % (the r101 numbers exactly); SSM_CONV /
RMS_NORM / SCALE / MUL_MAT oracles 0 failures; rule-5 batched decode within
noise at B=1/4/8. Decode: 27B MTP +0.2-0.3 %, Flash-Next 2 GPU tensor +0.7 %,
35B-A3B 1 GPU tg128 -0.2 % (i.e. flat; the submitter's +0.5 % did not
reproduce). No GPU faults.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 22, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 22, 2026
…r, QSA/indexer)
old-14's qwen4exp support plus old-15's qwen4exp/QSA/HC content, kept as ONE
block because QSA is intrinsic to the qwen4exp architecture (the old-15 QSA
memory wins, HC16 and qsa3 are load-bearing for the qwen4exp graph, and the
QSA/HC op dispatch is old-15):
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the
MTP draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm
policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, lightning-indexer.cu, with the CUDA dispatch
(GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE / INDEXER_FILL) and
their CPU references; the indexer cache in llama-memory-hybrid-idx.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (+ dispatch, supports_op, the
hc_combine_mix fusion) and the HC16 planar layout.
- the managed lazy reader (llama-lazy-reader) and the loader's managed_budget.
- the GDN beta-sigmoid fusion, the DSV4_HC split states in the Meta backend,
and the QSA/HC op definitions in ggml.h / ggml.c.
Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16,
GGML_CUDA_QSA_SLICES, GGML_CUDA_SPLICE_GATHER and the GDN_DBG_SKIP_* diagnostics once into values
rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call
overwrites, so a cached pointer is unsafe on Windows.
Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not
linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper.
Folded in from four contributor PRs on v12-097cf33d9-r80 (bsp2020 / briansp2020; block 10 also owns
the rope + FWHT fusion that ggml-org#172 and ggml-org#174 put here in r78):
- fused RMS_NORM + MUL + ROPE (+ FWHT) also for M-ROPE / IMROPE (PR ggml-org#175, 0001): the matcher admits
GGML_ROPE_TYPE_MROPE / IMROPE besides NORMAL / NEOX, and the kernel computes theta with rope_multi's
own section selection and float expressions (NEOX pairing, iw == i0, VISION still rejected). This is
the TODO item recorded against ggml-org#172: every Qwen3.x model runs rope_multi, so the fusion was inert on
the dense 27B and the 35B-A3B. Kill switch GGML_CUDA_DISABLE_ROPE_MROPE_FUSE=1.
- BF16 caches off the 5-op RMS_NORM + MUL + ROPE + VIEW + SET_ROWS fusion (PR ggml-org#175, 0002): that fusion
runs ggml_cuda_op_rms_norm_mul_rope_fused, which writes F32 / F16 rows only and otherwise aborts, but
ggml_cuda_should_fuse_rope_set_rows admits a BF16 SET_ROWS. Latent on r80 for a NEOX qk-norm model
with -ctk bf16, and reachable for the Qwen3.x models once 0001 lands. The 5-op match now declines a
BF16 SET_ROWS; the RMS_NORM + MUL + ROPE fusion and the plain SET_ROWS take it instead.
- the q8_0 K and V cache writes folded into the rope + FWHT kernel (PR ggml-org#173): with a q8_0 cache the K
path ends MUL_MAT(Hadamard), RESHAPE, VIEW, SET_ROWS and the V write (VIEW, SET_ROWS) follows it, so
the fused kernel now writes both cache rows through ggml_cuda_quantize_q8_0_group (k_set_rows_q8_0_group's
body, moved into set-rows.cuh). Only the K write is folded inside a concurrent-stream region
(GGML_CUDA_GRAPH_OPT=1). Kill switch GGML_CUDA_DISABLE_ROPE_FWHT_SET_ROWS=1.
- the GDN q and k l2 norms in one launch (PR ggml-org#179): build_gdn_l2_norm normalizes both with the identical
RMS_NORM + SCALE over views of the same conv output, so rms_norm_scale_pair_f32<256> takes blocks
[0, nrows) for q and [nrows, 2*nrows) for k with the same per-row body. Matcher additions: only
no-op nodes between the two pairs, both inputs F32 views of the same view_src with equal ne / nb,
equal eps and scale params, and the second input must not read the first pair's output.
Kill switch GGML_CUDA_DISABLE_NORM_PAIR=1.
Rejected in the same review: PR ggml-org#177 (GDN direct conv also writes the recurrent snapshot tails). It is
not run-to-run deterministic on the 35B-A3B prefill under HIP graph capture and does not reproduce the
stock logits there; record in archive/work/gdn-conv-snapshot-fold/.
Prefill cache staging hook (r83): ggml_backend_cuda_stage_from_cache is live again for the expert-cache
arena. The scheduler's whole-table staging copy of a host-resident expert table (stage_upload for
-sm layer, stage_gather for the -sm tensor meta split) is filled by one kernel from the arena (resident
experts, D2D) + the pinned host master (the rest, H2D) instead of a full H2D re-upload; the destination
is the ordinary staged table in expert order, so every consumer is unchanged. It declines to the plain
upload for an unusable/stood-down table, an unpinned host master or a geometry mismatch. Kill switch
GGML_MOE_CACHE_STAGE=0.
Slab narrow band (r85): the always-resident narrow band was an estimate taken before the first ring carve
(a `split_only` reserve of the `n_rs_batch` shape, 1792 MiB on the field config), and `set_narrow_floor`
cannot correct it afterwards because the ring's VA is frozen once it holds blocks. The real MTP
decode/verify graph needs 2304 MiB, so `work_alloc` classified every decode view as wide, placed it above
the H2D staging ring and evicted 3.65 GiB/device of arena. `work_alloc` now learns the band from the real
graph: a view that is not a requested wide (prefill) pass is narrow even above the floor while no wide view
is live (the ring region is only carved while one is), and the floor + ring are re-carved from it (the ring
is idle there). With a wide view live the above-the-ring placement is kept: the ring's VA must not move.
A refused raise is reported instead of dropped silently.
Measured (2x R9700, Flash-Next IQ4_NL -ncmoe 46, 131k ctx, -b/-ub 6144, MTP n6): 18 -> 1 re-arm, decode
23.6 -> 45.4 t/s (-sm tensor) / 35.8 (-sm layer), prefill unchanged; cost: 16k prefill 1443 -> 1411 t/s
(-2.2 %) because the correctly sized band moves the ring up 512 MiB.
The "may the band move" test is `ring_armed` alone: a live WIDE view is the only thing that makes the ring
region part of the reserved layout (`ring_armed` is set exactly when the boundary covers the carved ring).
`ring_used` is a bump cursor that stays at the end after a prefill, and `boundary > narrow_floor` is true
for an unrelated narrow view one chunk above the floor; treating either as "the ring may not move" left
the band unlearned on a single device -- measured gfx1100 `-ncmoe 40`: floor 1280 MiB against a 1536 MiB
decode view, one arena eviction per prefill -> decode transition. With this test the band is learned
there too (floor 1280 -> 1536, view placed at base 0, no eviction) and the two gfx1100 greedy hashes are
unchanged.
Fold (r86): beta 12, slab-transient-absorption (CUDA/slab half). The wide-phase
auxiliary region hosts the FA staging arena, the QSA3 workspace and the chunked-GDN
scratch; the workspace pool serves its large prefill temporaries from it; a graph
capture is never started while the wide band is armed; per-device free=/free_min/
outside_max/outside_failed diagnostics; default headroom 640 + 128 x n_gpu MiB.
Fold (r91): the chunked GDN prefill writes its rollback snapshots (fixed-D sequential tail).
The chunked kernel computes no per-token states, so a partial rollback into a batch it served
restored a state it never wrote (probe: logit NMSE 0.65, maxabs 12.5; see
archive/work/rs-rollback-boundary/). A single-sequence batch above the chunked threshold now
runs the chunked kernel on its first n_tokens - D tokens and the SEQUENTIAL kernel on the last
D = max(K>16?K:16, n_rs_batch) tokens in the same launch, so snapshot slots 0..D-1 exist for
every batch. No extra memory, no extra graph, ubatch unchanged (splitting the ubatch instead
measured -13.7 % pp512 and was reverted).
The bound is structural, not observational: cparams.n_rs_batch = the widest verify batch of any
enabled speculator (common_speculative_n_max() + 1, all draft and ngram types) and is carried
into this op as param 1, so a verify batch is never chunked; the tail covers the non-speculator
partial rewind (server cancel / --cache-reuse). D is fixed, not K-dependent, so plain decode and
MTP chunk a prefill identically.
Gates (gfx1201, 35B-A3B): probe P=2050 exact 0.0 and P=2048 rounding-level (1e-3) vs 0.65
without the tail; GATED_DELTA_NET 2/2 (48/48); prefill-logit vs the arm it replaces 0.0041 (fp32)
/ 0.0054 (bf16 chunked) and vs the exact sequential reference 0.004040 / 97.550 % (fp32) /
0.004408 / 97.055 % (bf16) -- no worse than the unpatched 0.004416 / 97.318 %; plain == MTP
byte-identical (64 tok and -n 3000, all four axes); recurrent depth 130 == 130, rollback 0 FAIL;
llama-bench pp512 -0.09 %, pp2048 -0.58 %, tg128 flat. gfx1151: 0.002389 / 98.069 % vs the
replaced arm, text unchanged. Kill switch GGML_CUDA_GDN_TAIL=0.
Fold (r95): rdna3-verify-fusions (PR ggml-org#192 + PR ggml-org#193, briansp2020): the verify-band Q8_1
pre-quantize fusions (residual ADD + rms_norm -> Q8_1, rms_norm -> Q8_1, gated unary -> Q8_1,
GLU -> Q8_1) and the direct GDN conv for 2..255-token verify batches are admitted on RDNA3_0
as well. Both gates become (RDNA4 || RDNA3_0), so RDNA4 and every other backend keep the
path they had and RDNA3_5 stays out. ggml_cuda_should_fuse_mul_mat_vec_q(..., verify_band)
is the single predicate the three Q8_1 fusions share, and its kernels are row-generic and
write the values quantize_q8_1 writes, so W = 1..8 stay bit-identical. The GDN conv keeps
its pre-existing single-device restriction. GGML_CUDA_FUSE_Q8_1_VERIFY=0 /
GGML_CUDA_FUSE_GDN_CONV_VERIFY=0 still turn them off.
GDN / attention-gate decode fusions, reported by @briansp2020 (PR ggml-org#200; R9700
gfx1201). Four bit-identical fusions, each with its own kill switch:
- GGML_CUDA_FUSE_GATE_BETA_BF16 (mmvf): qwen4exp's alpha/beta BF16 projections
and softplus(alpha + dt) * a in one mul_mat_vec_f launch (alpha rows, then
beta rows; each row runs the instantiation the unfused MUL_MAT takes, so both
outputs are bit-identical). The Q8_0 pair keeps ssm_gate_beta.
- GGML_CUDA_FUSE_GDN_CONV_NORM (gdn-conv): the q/k l2 norm pair folded into the
direct conv kernel, same reduction values and lanes as
rms_norm_scale_pair_f32<256>. Taken only when both SCALE outputs are disjoint
from the conv's x/state/weight/output: the allocator may place them over the
state or x (dead after the CONCAT in graph order) which the direct kernel
still reads, and without that check 35B-A3B gave run-to-run varying output.
- GGML_CUDA_FUSE_CONT_GATE_Q8_1 (unary): the attention gate view is read in
place by the gate -> Q8_1 kernel, so the CONT launches nothing.
- GGML_CUDA_FUSE_GDN_NORM_GATE (unary): qwen4exp's GDN output RMS_NORM * w is
computed inside the gate kernel, one row per 256-thread block exactly as
rms_norm_f32<256, true> runs it.
The norm's node lives several nodes before the gate node that computes it, so
the skip and the emit are two decisions joined by a context-pending pointer.
The submitted form guarded that with two GGML_ABORTs (norm never computed / gate
node never reached). This copy adds the structural guard instead: while a norm
is pending, no other fusion may start before its gate node, and the pending
norm's own MUL (dead: the gate kernel reads xn and w directly) is skipped. The
window is measured as span 4 with a fixed signature on the only graph this fires
on (qwen4exp), so the window nodes run unfused at the cost of one plain matmul
per GDN layer, and the aborts are unreachable rather than merely unlikely.
Gates (r101 base, gfx1201, ROCBLAS_USE_HIPBLASLT=0): greedy text identical in
five arms (27B plain and MTP n3, 35B-A3B 1 GPU and 2 GPU tensor, Flash-Next
IQ3_XXS 2 GPU tensor + shared MTP); test-logits-width-probe hashes identical to
r101; prefill-logit 0.000707 / 98.755 % (the r101 numbers exactly); SSM_CONV /
RMS_NORM / SCALE / MUL_MAT oracles 0 failures; rule-5 batched decode within
noise at B=1/4/8. Decode: 27B MTP +0.2-0.3 %, Flash-Next 2 GPU tensor +0.7 %,
35B-A3B 1 GPU tg128 -0.2 % (i.e. flat; the submitter's +0.5 % did not
reproduce). No GPU faults.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 22, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 22, 2026
…r, QSA/indexer)
old-14's qwen4exp support plus old-15's qwen4exp/QSA/HC content, kept as ONE
block because QSA is intrinsic to the qwen4exp architecture (the old-15 QSA
memory wins, HC16 and qsa3 are load-bearing for the qwen4exp graph, and the
QSA/HC op dispatch is old-15):
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the
MTP draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm
policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, lightning-indexer.cu, with the CUDA dispatch
(GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE / INDEXER_FILL) and
their CPU references; the indexer cache in llama-memory-hybrid-idx.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (+ dispatch, supports_op, the
hc_combine_mix fusion) and the HC16 planar layout.
- the managed lazy reader (llama-lazy-reader) and the loader's managed_budget.
- the GDN beta-sigmoid fusion, the DSV4_HC split states in the Meta backend,
and the QSA/HC op definitions in ggml.h / ggml.c.
Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16,
GGML_CUDA_QSA_SLICES, GGML_CUDA_SPLICE_GATHER and the GDN_DBG_SKIP_* diagnostics once into values
rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call
overwrites, so a cached pointer is unsafe on Windows.
Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not
linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper.
Folded in from four contributor PRs on v12-097cf33d9-r80 (bsp2020 / briansp2020; block 10 also owns
the rope + FWHT fusion that ggml-org#172 and ggml-org#174 put here in r78):
- fused RMS_NORM + MUL + ROPE (+ FWHT) also for M-ROPE / IMROPE (PR ggml-org#175, 0001): the matcher admits
GGML_ROPE_TYPE_MROPE / IMROPE besides NORMAL / NEOX, and the kernel computes theta with rope_multi's
own section selection and float expressions (NEOX pairing, iw == i0, VISION still rejected). This is
the TODO item recorded against ggml-org#172: every Qwen3.x model runs rope_multi, so the fusion was inert on
the dense 27B and the 35B-A3B. Kill switch GGML_CUDA_DISABLE_ROPE_MROPE_FUSE=1.
- BF16 caches off the 5-op RMS_NORM + MUL + ROPE + VIEW + SET_ROWS fusion (PR ggml-org#175, 0002): that fusion
runs ggml_cuda_op_rms_norm_mul_rope_fused, which writes F32 / F16 rows only and otherwise aborts, but
ggml_cuda_should_fuse_rope_set_rows admits a BF16 SET_ROWS. Latent on r80 for a NEOX qk-norm model
with -ctk bf16, and reachable for the Qwen3.x models once 0001 lands. The 5-op match now declines a
BF16 SET_ROWS; the RMS_NORM + MUL + ROPE fusion and the plain SET_ROWS take it instead.
- the q8_0 K and V cache writes folded into the rope + FWHT kernel (PR ggml-org#173): with a q8_0 cache the K
path ends MUL_MAT(Hadamard), RESHAPE, VIEW, SET_ROWS and the V write (VIEW, SET_ROWS) follows it, so
the fused kernel now writes both cache rows through ggml_cuda_quantize_q8_0_group (k_set_rows_q8_0_group's
body, moved into set-rows.cuh). Only the K write is folded inside a concurrent-stream region
(GGML_CUDA_GRAPH_OPT=1). Kill switch GGML_CUDA_DISABLE_ROPE_FWHT_SET_ROWS=1.
- the GDN q and k l2 norms in one launch (PR ggml-org#179): build_gdn_l2_norm normalizes both with the identical
RMS_NORM + SCALE over views of the same conv output, so rms_norm_scale_pair_f32<256> takes blocks
[0, nrows) for q and [nrows, 2*nrows) for k with the same per-row body. Matcher additions: only
no-op nodes between the two pairs, both inputs F32 views of the same view_src with equal ne / nb,
equal eps and scale params, and the second input must not read the first pair's output.
Kill switch GGML_CUDA_DISABLE_NORM_PAIR=1.
Rejected in the same review: PR ggml-org#177 (GDN direct conv also writes the recurrent snapshot tails). It is
not run-to-run deterministic on the 35B-A3B prefill under HIP graph capture and does not reproduce the
stock logits there; record in archive/work/gdn-conv-snapshot-fold/.
Prefill cache staging hook (r83): ggml_backend_cuda_stage_from_cache is live again for the expert-cache
arena. The scheduler's whole-table staging copy of a host-resident expert table (stage_upload for
-sm layer, stage_gather for the -sm tensor meta split) is filled by one kernel from the arena (resident
experts, D2D) + the pinned host master (the rest, H2D) instead of a full H2D re-upload; the destination
is the ordinary staged table in expert order, so every consumer is unchanged. It declines to the plain
upload for an unusable/stood-down table, an unpinned host master or a geometry mismatch. Kill switch
GGML_MOE_CACHE_STAGE=0.
Slab narrow band (r85): the always-resident narrow band was an estimate taken before the first ring carve
(a `split_only` reserve of the `n_rs_batch` shape, 1792 MiB on the field config), and `set_narrow_floor`
cannot correct it afterwards because the ring's VA is frozen once it holds blocks. The real MTP
decode/verify graph needs 2304 MiB, so `work_alloc` classified every decode view as wide, placed it above
the H2D staging ring and evicted 3.65 GiB/device of arena. `work_alloc` now learns the band from the real
graph: a view that is not a requested wide (prefill) pass is narrow even above the floor while no wide view
is live (the ring region is only carved while one is), and the floor + ring are re-carved from it (the ring
is idle there). With a wide view live the above-the-ring placement is kept: the ring's VA must not move.
A refused raise is reported instead of dropped silently.
Measured (2x R9700, Flash-Next IQ4_NL -ncmoe 46, 131k ctx, -b/-ub 6144, MTP n6): 18 -> 1 re-arm, decode
23.6 -> 45.4 t/s (-sm tensor) / 35.8 (-sm layer), prefill unchanged; cost: 16k prefill 1443 -> 1411 t/s
(-2.2 %) because the correctly sized band moves the ring up 512 MiB.
The "may the band move" test is `ring_armed` alone: a live WIDE view is the only thing that makes the ring
region part of the reserved layout (`ring_armed` is set exactly when the boundary covers the carved ring).
`ring_used` is a bump cursor that stays at the end after a prefill, and `boundary > narrow_floor` is true
for an unrelated narrow view one chunk above the floor; treating either as "the ring may not move" left
the band unlearned on a single device -- measured gfx1100 `-ncmoe 40`: floor 1280 MiB against a 1536 MiB
decode view, one arena eviction per prefill -> decode transition. With this test the band is learned
there too (floor 1280 -> 1536, view placed at base 0, no eviction) and the two gfx1100 greedy hashes are
unchanged.
Fold (r86): beta 12, slab-transient-absorption (CUDA/slab half). The wide-phase
auxiliary region hosts the FA staging arena, the QSA3 workspace and the chunked-GDN
scratch; the workspace pool serves its large prefill temporaries from it; a graph
capture is never started while the wide band is armed; per-device free=/free_min/
outside_max/outside_failed diagnostics; default headroom 640 + 128 x n_gpu MiB.
Fold (r91): the chunked GDN prefill writes its rollback snapshots (fixed-D sequential tail).
The chunked kernel computes no per-token states, so a partial rollback into a batch it served
restored a state it never wrote (probe: logit NMSE 0.65, maxabs 12.5; see
archive/work/rs-rollback-boundary/). A single-sequence batch above the chunked threshold now
runs the chunked kernel on its first n_tokens - D tokens and the SEQUENTIAL kernel on the last
D = max(K>16?K:16, n_rs_batch) tokens in the same launch, so snapshot slots 0..D-1 exist for
every batch. No extra memory, no extra graph, ubatch unchanged (splitting the ubatch instead
measured -13.7 % pp512 and was reverted).
The bound is structural, not observational: cparams.n_rs_batch = the widest verify batch of any
enabled speculator (common_speculative_n_max() + 1, all draft and ngram types) and is carried
into this op as param 1, so a verify batch is never chunked; the tail covers the non-speculator
partial rewind (server cancel / --cache-reuse). D is fixed, not K-dependent, so plain decode and
MTP chunk a prefill identically.
Gates (gfx1201, 35B-A3B): probe P=2050 exact 0.0 and P=2048 rounding-level (1e-3) vs 0.65
without the tail; GATED_DELTA_NET 2/2 (48/48); prefill-logit vs the arm it replaces 0.0041 (fp32)
/ 0.0054 (bf16 chunked) and vs the exact sequential reference 0.004040 / 97.550 % (fp32) /
0.004408 / 97.055 % (bf16) -- no worse than the unpatched 0.004416 / 97.318 %; plain == MTP
byte-identical (64 tok and -n 3000, all four axes); recurrent depth 130 == 130, rollback 0 FAIL;
llama-bench pp512 -0.09 %, pp2048 -0.58 %, tg128 flat. gfx1151: 0.002389 / 98.069 % vs the
replaced arm, text unchanged. Kill switch GGML_CUDA_GDN_TAIL=0.
Fold (r95): rdna3-verify-fusions (PR ggml-org#192 + PR ggml-org#193, briansp2020): the verify-band Q8_1
pre-quantize fusions (residual ADD + rms_norm -> Q8_1, rms_norm -> Q8_1, gated unary -> Q8_1,
GLU -> Q8_1) and the direct GDN conv for 2..255-token verify batches are admitted on RDNA3_0
as well. Both gates become (RDNA4 || RDNA3_0), so RDNA4 and every other backend keep the
path they had and RDNA3_5 stays out. ggml_cuda_should_fuse_mul_mat_vec_q(..., verify_band)
is the single predicate the three Q8_1 fusions share, and its kernels are row-generic and
write the values quantize_q8_1 writes, so W = 1..8 stay bit-identical. The GDN conv keeps
its pre-existing single-device restriction. GGML_CUDA_FUSE_Q8_1_VERIFY=0 /
GGML_CUDA_FUSE_GDN_CONV_VERIFY=0 still turn them off.
GDN / attention-gate decode fusions, reported by @briansp2020 (PR ggml-org#200; R9700
gfx1201). Four bit-identical fusions, each with its own kill switch:
- GGML_CUDA_FUSE_GATE_BETA_BF16 (mmvf): qwen4exp's alpha/beta BF16 projections
and softplus(alpha + dt) * a in one mul_mat_vec_f launch (alpha rows, then
beta rows; each row runs the instantiation the unfused MUL_MAT takes, so both
outputs are bit-identical). The Q8_0 pair keeps ssm_gate_beta.
- GGML_CUDA_FUSE_GDN_CONV_NORM (gdn-conv): the q/k l2 norm pair folded into the
direct conv kernel, same reduction values and lanes as
rms_norm_scale_pair_f32<256>. Taken only when both SCALE outputs are disjoint
from the conv's x/state/weight/output: the allocator may place them over the
state or x (dead after the CONCAT in graph order) which the direct kernel
still reads, and without that check 35B-A3B gave run-to-run varying output.
- GGML_CUDA_FUSE_CONT_GATE_Q8_1 (unary): the attention gate view is read in
place by the gate -> Q8_1 kernel, so the CONT launches nothing.
- GGML_CUDA_FUSE_GDN_NORM_GATE (unary): qwen4exp's GDN output RMS_NORM * w is
computed inside the gate kernel, one row per 256-thread block exactly as
rms_norm_f32<256, true> runs it.
The norm's node lives several nodes before the gate node that computes it, so
the skip and the emit are two decisions joined by a context-pending pointer.
The submitted form guarded that with two GGML_ABORTs (norm never computed / gate
node never reached). This copy adds the structural guard instead: while a norm
is pending, no other fusion may start before its gate node, and the pending
norm's own MUL (dead: the gate kernel reads xn and w directly) is skipped. The
window is measured as span 4 with a fixed signature on the only graph this fires
on (qwen4exp), so the window nodes run unfused at the cost of one plain matmul
per GDN layer, and the aborts are unreachable rather than merely unlikely.
Gates (r101 base, gfx1201, ROCBLAS_USE_HIPBLASLT=0): greedy text identical in
five arms (27B plain and MTP n3, 35B-A3B 1 GPU and 2 GPU tensor, Flash-Next
IQ3_XXS 2 GPU tensor + shared MTP); test-logits-width-probe hashes identical to
r101; prefill-logit 0.000707 / 98.755 % (the r101 numbers exactly); SSM_CONV /
RMS_NORM / SCALE / MUL_MAT oracles 0 failures; rule-5 batched decode within
noise at B=1/4/8. Decode: 27B MTP +0.2-0.3 %, Flash-Next 2 GPU tensor +0.7 %,
35B-A3B 1 GPU tg128 -0.2 % (i.e. flat; the submitter's +0.5 % did not
reproduce). No GPU faults.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 22, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 22, 2026
…r, QSA/indexer)
old-14's qwen4exp support plus old-15's qwen4exp/QSA/HC content, kept as ONE
block because QSA is intrinsic to the qwen4exp architecture (the old-15 QSA
memory wins, HC16 and qsa3 are load-bearing for the qwen4exp graph, and the
QSA/HC op dispatch is old-15):
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the
MTP draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm
policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, lightning-indexer.cu, with the CUDA dispatch
(GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE / INDEXER_FILL) and
their CPU references; the indexer cache in llama-memory-hybrid-idx.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (+ dispatch, supports_op, the
hc_combine_mix fusion) and the HC16 planar layout.
- the managed lazy reader (llama-lazy-reader) and the loader's managed_budget.
- the GDN beta-sigmoid fusion, the DSV4_HC split states in the Meta backend,
and the QSA/HC op definitions in ggml.h / ggml.c.
Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16,
GGML_CUDA_QSA_SLICES, GGML_CUDA_SPLICE_GATHER and the GDN_DBG_SKIP_* diagnostics once into values
rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call
overwrites, so a cached pointer is unsafe on Windows.
Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not
linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper.
Folded in from four contributor PRs on v12-097cf33d9-r80 (bsp2020 / briansp2020; block 10 also owns
the rope + FWHT fusion that ggml-org#172 and ggml-org#174 put here in r78):
- fused RMS_NORM + MUL + ROPE (+ FWHT) also for M-ROPE / IMROPE (PR ggml-org#175, 0001): the matcher admits
GGML_ROPE_TYPE_MROPE / IMROPE besides NORMAL / NEOX, and the kernel computes theta with rope_multi's
own section selection and float expressions (NEOX pairing, iw == i0, VISION still rejected). This is
the TODO item recorded against ggml-org#172: every Qwen3.x model runs rope_multi, so the fusion was inert on
the dense 27B and the 35B-A3B. Kill switch GGML_CUDA_DISABLE_ROPE_MROPE_FUSE=1.
- BF16 caches off the 5-op RMS_NORM + MUL + ROPE + VIEW + SET_ROWS fusion (PR ggml-org#175, 0002): that fusion
runs ggml_cuda_op_rms_norm_mul_rope_fused, which writes F32 / F16 rows only and otherwise aborts, but
ggml_cuda_should_fuse_rope_set_rows admits a BF16 SET_ROWS. Latent on r80 for a NEOX qk-norm model
with -ctk bf16, and reachable for the Qwen3.x models once 0001 lands. The 5-op match now declines a
BF16 SET_ROWS; the RMS_NORM + MUL + ROPE fusion and the plain SET_ROWS take it instead.
- the q8_0 K and V cache writes folded into the rope + FWHT kernel (PR ggml-org#173): with a q8_0 cache the K
path ends MUL_MAT(Hadamard), RESHAPE, VIEW, SET_ROWS and the V write (VIEW, SET_ROWS) follows it, so
the fused kernel now writes both cache rows through ggml_cuda_quantize_q8_0_group (k_set_rows_q8_0_group's
body, moved into set-rows.cuh). Only the K write is folded inside a concurrent-stream region
(GGML_CUDA_GRAPH_OPT=1). Kill switch GGML_CUDA_DISABLE_ROPE_FWHT_SET_ROWS=1.
- the GDN q and k l2 norms in one launch (PR ggml-org#179): build_gdn_l2_norm normalizes both with the identical
RMS_NORM + SCALE over views of the same conv output, so rms_norm_scale_pair_f32<256> takes blocks
[0, nrows) for q and [nrows, 2*nrows) for k with the same per-row body. Matcher additions: only
no-op nodes between the two pairs, both inputs F32 views of the same view_src with equal ne / nb,
equal eps and scale params, and the second input must not read the first pair's output.
Kill switch GGML_CUDA_DISABLE_NORM_PAIR=1.
Rejected in the same review: PR ggml-org#177 (GDN direct conv also writes the recurrent snapshot tails). It is
not run-to-run deterministic on the 35B-A3B prefill under HIP graph capture and does not reproduce the
stock logits there; record in archive/work/gdn-conv-snapshot-fold/.
Prefill cache staging hook (r83): ggml_backend_cuda_stage_from_cache is live again for the expert-cache
arena. The scheduler's whole-table staging copy of a host-resident expert table (stage_upload for
-sm layer, stage_gather for the -sm tensor meta split) is filled by one kernel from the arena (resident
experts, D2D) + the pinned host master (the rest, H2D) instead of a full H2D re-upload; the destination
is the ordinary staged table in expert order, so every consumer is unchanged. It declines to the plain
upload for an unusable/stood-down table, an unpinned host master or a geometry mismatch. Kill switch
GGML_MOE_CACHE_STAGE=0.
Slab narrow band (r85): the always-resident narrow band was an estimate taken before the first ring carve
(a `split_only` reserve of the `n_rs_batch` shape, 1792 MiB on the field config), and `set_narrow_floor`
cannot correct it afterwards because the ring's VA is frozen once it holds blocks. The real MTP
decode/verify graph needs 2304 MiB, so `work_alloc` classified every decode view as wide, placed it above
the H2D staging ring and evicted 3.65 GiB/device of arena. `work_alloc` now learns the band from the real
graph: a view that is not a requested wide (prefill) pass is narrow even above the floor while no wide view
is live (the ring region is only carved while one is), and the floor + ring are re-carved from it (the ring
is idle there). With a wide view live the above-the-ring placement is kept: the ring's VA must not move.
A refused raise is reported instead of dropped silently.
Measured (2x R9700, Flash-Next IQ4_NL -ncmoe 46, 131k ctx, -b/-ub 6144, MTP n6): 18 -> 1 re-arm, decode
23.6 -> 45.4 t/s (-sm tensor) / 35.8 (-sm layer), prefill unchanged; cost: 16k prefill 1443 -> 1411 t/s
(-2.2 %) because the correctly sized band moves the ring up 512 MiB.
The "may the band move" test is `ring_armed` alone: a live WIDE view is the only thing that makes the ring
region part of the reserved layout (`ring_armed` is set exactly when the boundary covers the carved ring).
`ring_used` is a bump cursor that stays at the end after a prefill, and `boundary > narrow_floor` is true
for an unrelated narrow view one chunk above the floor; treating either as "the ring may not move" left
the band unlearned on a single device -- measured gfx1100 `-ncmoe 40`: floor 1280 MiB against a 1536 MiB
decode view, one arena eviction per prefill -> decode transition. With this test the band is learned
there too (floor 1280 -> 1536, view placed at base 0, no eviction) and the two gfx1100 greedy hashes are
unchanged.
Fold (r86): beta 12, slab-transient-absorption (CUDA/slab half). The wide-phase
auxiliary region hosts the FA staging arena, the QSA3 workspace and the chunked-GDN
scratch; the workspace pool serves its large prefill temporaries from it; a graph
capture is never started while the wide band is armed; per-device free=/free_min/
outside_max/outside_failed diagnostics; default headroom 640 + 128 x n_gpu MiB.
Fold (r91): the chunked GDN prefill writes its rollback snapshots (fixed-D sequential tail).
The chunked kernel computes no per-token states, so a partial rollback into a batch it served
restored a state it never wrote (probe: logit NMSE 0.65, maxabs 12.5; see
archive/work/rs-rollback-boundary/). A single-sequence batch above the chunked threshold now
runs the chunked kernel on its first n_tokens - D tokens and the SEQUENTIAL kernel on the last
D = max(K>16?K:16, n_rs_batch) tokens in the same launch, so snapshot slots 0..D-1 exist for
every batch. No extra memory, no extra graph, ubatch unchanged (splitting the ubatch instead
measured -13.7 % pp512 and was reverted).
The bound is structural, not observational: cparams.n_rs_batch = the widest verify batch of any
enabled speculator (common_speculative_n_max() + 1, all draft and ngram types) and is carried
into this op as param 1, so a verify batch is never chunked; the tail covers the non-speculator
partial rewind (server cancel / --cache-reuse). D is fixed, not K-dependent, so plain decode and
MTP chunk a prefill identically.
Gates (gfx1201, 35B-A3B): probe P=2050 exact 0.0 and P=2048 rounding-level (1e-3) vs 0.65
without the tail; GATED_DELTA_NET 2/2 (48/48); prefill-logit vs the arm it replaces 0.0041 (fp32)
/ 0.0054 (bf16 chunked) and vs the exact sequential reference 0.004040 / 97.550 % (fp32) /
0.004408 / 97.055 % (bf16) -- no worse than the unpatched 0.004416 / 97.318 %; plain == MTP
byte-identical (64 tok and -n 3000, all four axes); recurrent depth 130 == 130, rollback 0 FAIL;
llama-bench pp512 -0.09 %, pp2048 -0.58 %, tg128 flat. gfx1151: 0.002389 / 98.069 % vs the
replaced arm, text unchanged. Kill switch GGML_CUDA_GDN_TAIL=0.
Fold (r95): rdna3-verify-fusions (PR ggml-org#192 + PR ggml-org#193, briansp2020): the verify-band Q8_1
pre-quantize fusions (residual ADD + rms_norm -> Q8_1, rms_norm -> Q8_1, gated unary -> Q8_1,
GLU -> Q8_1) and the direct GDN conv for 2..255-token verify batches are admitted on RDNA3_0
as well. Both gates become (RDNA4 || RDNA3_0), so RDNA4 and every other backend keep the
path they had and RDNA3_5 stays out. ggml_cuda_should_fuse_mul_mat_vec_q(..., verify_band)
is the single predicate the three Q8_1 fusions share, and its kernels are row-generic and
write the values quantize_q8_1 writes, so W = 1..8 stay bit-identical. The GDN conv keeps
its pre-existing single-device restriction. GGML_CUDA_FUSE_Q8_1_VERIFY=0 /
GGML_CUDA_FUSE_GDN_CONV_VERIFY=0 still turn them off.
GDN / attention-gate decode fusions, reported by @briansp2020 (PR ggml-org#200; R9700
gfx1201). Four bit-identical fusions, each with its own kill switch:
- GGML_CUDA_FUSE_GATE_BETA_BF16 (mmvf): qwen4exp's alpha/beta BF16 projections
and softplus(alpha + dt) * a in one mul_mat_vec_f launch (alpha rows, then
beta rows; each row runs the instantiation the unfused MUL_MAT takes, so both
outputs are bit-identical). The Q8_0 pair keeps ssm_gate_beta.
- GGML_CUDA_FUSE_GDN_CONV_NORM (gdn-conv): the q/k l2 norm pair folded into the
direct conv kernel, same reduction values and lanes as
rms_norm_scale_pair_f32<256>. Taken only when both SCALE outputs are disjoint
from the conv's x/state/weight/output: the allocator may place them over the
state or x (dead after the CONCAT in graph order) which the direct kernel
still reads, and without that check 35B-A3B gave run-to-run varying output.
- GGML_CUDA_FUSE_CONT_GATE_Q8_1 (unary): the attention gate view is read in
place by the gate -> Q8_1 kernel, so the CONT launches nothing.
- GGML_CUDA_FUSE_GDN_NORM_GATE (unary): qwen4exp's GDN output RMS_NORM * w is
computed inside the gate kernel, one row per 256-thread block exactly as
rms_norm_f32<256, true> runs it.
The norm's node lives several nodes before the gate node that computes it, so
the skip and the emit are two decisions joined by a context-pending pointer.
The submitted form guarded that with two GGML_ABORTs (norm never computed / gate
node never reached). This copy adds the structural guard instead: while a norm
is pending, no other fusion may start before its gate node, and the pending
norm's own MUL (dead: the gate kernel reads xn and w directly) is skipped. The
window is measured as span 4 with a fixed signature on the only graph this fires
on (qwen4exp), so the window nodes run unfused at the cost of one plain matmul
per GDN layer, and the aborts are unreachable rather than merely unlikely.
Gates (r101 base, gfx1201, ROCBLAS_USE_HIPBLASLT=0): greedy text identical in
five arms (27B plain and MTP n3, 35B-A3B 1 GPU and 2 GPU tensor, Flash-Next
IQ3_XXS 2 GPU tensor + shared MTP); test-logits-width-probe hashes identical to
r101; prefill-logit 0.000707 / 98.755 % (the r101 numbers exactly); SSM_CONV /
RMS_NORM / SCALE / MUL_MAT oracles 0 failures; rule-5 batched decode within
noise at B=1/4/8. Decode: 27B MTP +0.2-0.3 %, Flash-Next 2 GPU tensor +0.7 %,
35B-A3B 1 GPU tg128 -0.2 % (i.e. flat; the submitter's +0.5 % did not
reproduce). No GPU faults.
GDN gate/beta BF16 fusion: multi-sequence fix + regression test (issue ggml-org#204, r108).
ggml_cuda_gate_beta_bf16_try_fuse passed the gate chain's MUL as the mmvf dst. The
launch takes its column count from dst->ne[1] and its channel count from dst->ne[2];
with more than one sequence the gate MUL is [nr, n_seq_tokens, n_seqs] while the
projections and y are flat [nr, n_tokens] / [ne10, n_tokens] with one channel, so
every channel past the first re-ran the beta rows and wrote the alpha/beta values
computed from y's first n_seq_tokens columns: every sequence but the first silently
took sequence 0's gates. The text stays coherent, which is why the same-seed and
MTP gates never saw it - only multi-sequence serving did (reported 4-stream draft
acceptance 0.85 -> 0.82 at n_seq_tokens 2).
The launch now uses the projection's own shape: both tensors are contiguous with
the same element count (checked by the matcher), so it is the same memory, written
flat. A single-sequence launch is unchanged (identical ne/nb/data), so single-stream
numerics are bit-identical; multi-sequence decode/verify now matches the unfused
chain exactly.
tests/test-fusion-gate-beta-multiseq.cpp is the regression test. It needs no model:
it builds the chain for a matrix of (n_seq_tokens, n_seqs) shapes on the GPU
backend, compares the fused launch against the same graph with the fusion made
ineligible (one extra consumer of the ADD output makes ggml_can_fuse_subgraph
reject the window), requires bit-identical gate and beta values, and proves the
fusion actually fired (the skipped SOFTPLUS output is left at a NaN sentinel), so
a change that silently stops matching fails the test instead of passing it
vacuously.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 22, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 22, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 22, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 22, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 23, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 23, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 23, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 23, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 23, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 23, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 24, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 24, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 24, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 24, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Includes vectorised inference code, quantisation and a counterpart to the Q4_0 multipart fix we introduced a while ago. Tested working up to 13B, though I can't confidently say anything about the impact on quality (especially since the RMS norm patch also just landed). Speed overheads relative to Q4_0 seem to be about 50%. This should give us a viable framework to evaluate Q4_1 quantization on x86 machines.
What's missing is accelerated inference code for for ARM NEON - I have no access to any machine that has it, so I'm going to have to delegate there.