Q8 0 support by gopinath87607 · Pull Request #255 · Niko1221/Strata

gopinath87607 · 2026-09-30T15:33:52Z

Q8_0: the checkpoint quant the engine did not know

Stacked on #216 (the session carve) - merge that first. Until it merges, this diff shows both changes;
afterwards GitHub shrinks it to the Q8_0 commit alone.

Qwen3.8-Flash-Next-Q8_0 is the same qwen4exp topology as the IQ3_S build, but every expert, embedding and
PLE tensor arrives Q8_0 - a type three hard gates refused. All of this is code: no data ships with the PR
(.gitignore keeps /packs/ and /mtp/ out), so other users pack their own GGUF locally, the same flow
IQ3_S/OrcaRouter users already follow (docs/ORCA.md).

GPU EXPERT KERNELS. Fmt<8> in iq_kernels.cu (qk 32, 4 ints per block, vec_dot_q8_0_q8_1), a dq_q8_0
dequantizer, and case 8 in the grouped gate/up/down switches: native_expert_grouped runs Q8_0 experts,
the embedding table is Q8_0, and parity against ggml-cpu's own vec_dot on identical weights holds to 1e-7.
THE PLE READER LOSES ITS 90-BYTE ROW. ROW_BYTES becomes a runtime row_bytes (90 B at IQ4_NL, 170 B at
Q8_0) everywhere it was compile-time - row cache, page bounds, issue() - so the per-layer token table
reads either quant.
CONV1D, THE ONE REAL BUG. The Q8_0 checkpoint ships blk.1.ple_conv1d.weight as F32 (163,840 B) but the
engine's PLE kernel reads F16, so half of every f32 word was decoded as a weight: mean |w| 0.43 against the
true 0.0062, layer-1 routing flipped, and the output was PLAUSIBLE GARBAGE, not a crash. iq_pack.py now
writes that tensor as pack index kind 3 (the loader narrows F32 to F16 round-to-nearest-even; proven
bit-exact against the IQ3_S pack's own tensor), narrowing happens whether or not --compat-bf16 is given,
and generate.cpp refuses to boot a pack whose conv1d is still wide instead of decoding it as garbage.
SHARDS BY ROLE. In this split the gate/up/down of layers 17/35 sit in different shards than their
neighbours; the pack index carries a per-role shard column and the engine resolves each role's file, so
--native names any shard of the family and the head is auto-located by output.weight.
THE ARENA ASKS THP FOR 2 MB PAGES. reserve() falls back from MAP_HUGETLB to plain 4 KB pages and stayed
there; MADV_HUGEPAGE costs nothing and needs no hugetlb pool or privilege, and 116.3 of the arena's
119.5 GiB lands on transparent 2 MB pages. Measured neutral on the reference rig - the expert streams are
multi-MB sequential reads, so TLB reach was never the wall - kept for the documented design intent
(include/strata/core/pinned.hpp), A/B with STRATA_NO_THP=1 like the Windows large-pages switch.
SERVER. The start-up "reading N GB" narration now takes the true arena size from the pack's
native_experts.txt header (128 GB; the named shard holds 49) with the shard-family sum as fallback.

HOW OTHERS GET IT - nothing is uploaded, every byte of model data is produced on their machine:

git fetch && git checkout <branch>          # the code IS the Q8_0 support
python3 tools/iq_pack.py --gguf Qwen3.8-Flash-Next-Q8_0-00001-of-00006.gguf \
       --out packs/q8_0 --compat-bf16       # local conversion from their own GGUF
python3 tools/mtp_fetch.py fetch --out mtp-src       # the MTP drafter, once per model family
python3 tools/mtp_pack.py --src mtp-src --out mtp/mtp-q2_0.gguf
python3 tools/mtp_rt.py  --gguf mtp/mtp-q2_0.gguf --out mtp/rt

Three things worth saying out loud: --compat-bf16 is mandatory for the plain Q8_0 checkpoint (the routers
and projections ship Q8_0 but the engine reads BF16; the packer refuses without the flag, by design); the
conv1d fix lives in the packer, so every rebuilt pack is correct and every OLD broken pack is refused at
boot; and the MTP drafter is per model family, not per quant - build it once, share it between IQ3_S and
Q8_0.

VERIFIED AGAINST AN ORACLE, not against another quant. llama.cpp's CPU build (which knows qwen4exp) fed
the exact token ids the engine processed, greedy: the fixed build's first tokens match it exactly and the
two then split on a genuine near-tie (Strata's GPU experts quantize activations to Q8_1; llama.cpp CPU uses
Q8_0 for Q8_0 weights) - the "divergence from IQ3_S" that started this was IQ3_S being a different
quantization, not a bug.
COST, STATED PLAINLY. Experts stream 2.51 GB per token against IQ3_S's 0.61, so decode runs ~21-23 tok/s
against IQ3_S's 38.7 on the reference 4-card rig, with ~51% expert-cache hits (3,637 of 24,576 pairs
resident at 5.22 MB an expert). Prefill is chunk-bound: 115 tok/s at the 2048-token chunk the first config
shipped, 202 tok/s at 4096 (+75%) - the engine clamps an explicit chunk to what every cache participant can
lend. The 119.5 GiB expert arena needs numactl --interleave=0,1 on dual-socket boards. Quality is the
point of Q8_0; the trade is deliberate.

🤖 Generated with Claude Code

Two things a layer split was paying for and not using. * THE SESSION CARVE. A stage carved state for ALL 48 layers whatever slice it ran - every QSA KV pool at max_cells, every GDN recurrence - the whole-model-sized-buffer disease the chunked-QSA-prefill PR cured upstream. `session_bytes`/`session_init` take [layer_lo, layer_hi), and a session now owns only that range: `qsa_states`/`gdn_state` keep GLOBAL ordinal indexing and simply hold fewer rows (`qsa_ord0`/`qsa_alloc`, `gdn_ord0`/`gdn_alloc`), per-layer consumers subtract `gdn_ord0`, and the model-level uses - the shared RoPE table, the prefill staging identity, the MTP rope borrow - read `qsa_states[qsa_primary()]`. A range with no QSA layer still carves ONE primary state so those uses stay valid, and the default full range is byte-identical to the old carve. On the 2x3060 + 2x5060 rig, CUDA1's [30, 48) session costs 0.93 GiB: 7.83 GiB free before the carve, 6.90 GiB after (`session [30, 48) and the head`). * PER-STAGE PROMPT LOANS. `no_prefill_borrow = true` for split stages said "each stage's prompt path has its own buffers" - true, but that is a reason to give each stage its own LOAN, not to make every stage withhold a chunk-sized reserve from its cache for the whole session, and forced on it also collapsed `--prefill auto` to 2048. Each prompt path now borrows the tail of ITS OWN stage's cache (`PfPart`), so outside the prompt the whole cache is expert cache: `--prefill auto` is the largest chunk every participant can lend (8192 tokens here), a participant refills its loan into its own slots and marks only its own layers (a slot refilled into another cache, or another stage's rows marked resident, is silent and produces plausible tokens), and the chunk is the largest one EVERY participant can lend rather than one card's answer imposed on the others. A participant that cannot lend the buffers falls back to the prompt path's own, as a lone one always could; with a single participant the plan reduces to `plan_lend` exactly, so the single-GPU loan is unchanged from main - the percentage cap stays an AUTO-chunk rule (an explicit `--prefill` is the operator's number and a loan of it only has to leave the 128-slot floor). The flat 1 GiB withheld from every stage after the first becomes what is actually held back after the weights, session and drafter exist: 96 MiB of verify windows, plus 1000 MiB of drafter and head on the ONE stage that carries them. CUDA0 lends 2162 slots (3.98 GiB), CUDA1 1993 of its 2653 (3.97 GiB). * REPORTING. `expert_slots`/`expert_cache_mib` sum every tier: a 4-GPU 256K run reported CUDA0's cache alone (3327 experts) while the four cards held 13310, so the Monitor tab was wrong by 4x for every multi-GPU config. `expert_slots_primary`/`expert_cache_primary_mib` keep the old figure. The helper ranks' ms_begin/ms_wait are cumulative since boot and were printed next to per-request deltas; they take deltas now. KV-streaming stats iterate the session's OWNED ordinals. * REFUSALS. `--gpu-only-full` and `--gpu-stages` replay every layer through CUDA0's session, which a split carves to CUDA0's range, so they are refused with a layer split rather than answering about another stage's state; the token-graph hit path is disabled under a split (such a graph cannot span stages); and the sessions are allocated AFTER the split search, because the search prices each placement by what that range's session will cost on that device (`session_bytes` is pure arithmetic). Co-Authored-By: Claude Code <noreply@anthropic.com>

Qwen3.8-Flash-Next-Q8_0 is the same qwen4exp topology as the IQ3_S build, but every expert, embedding and PLE tensor arrives Q8_0 - a type three hard gates refused. All of this is code: no data ships in the repo (.gitignore keeps /packs/ and /mtp/ out), so other users pack their own GGUF locally, the same flow IQ3_S/OrcaRouter users already follow (docs/ORCA.md). * GPU EXPERT KERNELS. Fmt<8> in iq_kernels.cu (qk 32, 4 ints per block, vec_dot_q8_0_q8_1), a dq_q8_0 dequantizer, and case 8 in the grouped gate/up/down switches: native_expert_grouped runs Q8_0 experts, the embedding table is Q8_0, and parity against ggml-cpu's own vec_dot on identical weights holds to 1e-7. * THE PLE READER LOSES ITS 90-BYTE ROW. ROW_BYTES becomes a runtime row_bytes (90 B at IQ4_NL, 170 B at Q8_0) everywhere it was compile-time - row cache, page bounds, issue() - so the per-layer token table reads either quant. * CONV1D, THE ONE REAL BUG. The Q8_0 checkpoint ships blk.1.ple_conv1d.weight as F32 (163,840 B) but the engine's PLE kernel reads F16, so half of every f32 word was decoded as a weight: mean |w| 0.43 against the true 0.0062, layer-1 routing flipped, and the output was PLAUSIBLE GARBAGE, not a crash. iq_pack.py writes that tensor as pack index kind 3 (the loader narrows F32 to F16 round-to-nearest-even; proven bit-exact against the IQ3_S pack's own tensor), narrowing happens whether or not --compat-bf16 is given, and generate.cpp refuses to boot a pack whose conv1d is still wide instead of decoding it as garbage. * SHARDS BY ROLE. In this split the gate/up/down of layers 17/35 sit in different shards than their neighbours; the pack index carries a per-role shard column and the engine resolves each role's file, so --native names any shard of the family and the head is auto-located by output.weight. * THE ARENA ASKS THP FOR 2 MB PAGES. reserve() falls back from MAP_HUGETLB to plain 4 KB pages and stays there; MADV_HUGEPAGE costs nothing and needs no hugetlb pool or privilege, and 116.3 of the arena's 119.5 GiB lands on transparent 2 MB pages. Measured neutral on the reference rig - the expert streams are multi-MB sequential reads, so TLB reach was never the wall - kept for the documented design intent (include/strata/core/pinned.hpp), A/B with STRATA_NO_THP=1 like the Windows large-pages switch. * SERVER. The start-up "reading N GB" narration takes the true arena size from the pack's native_experts.txt header (128 GB; the named shard holds 49) with the shard-family sum as fallback. Verified against an oracle, not against another quant: llama.cpp's CPU build (which knows qwen4exp) fed the exact token ids the engine processed, greedy - the fixed build's first tokens match it exactly and the two then split on a genuine near-tie (Strata's GPU experts quantize activations to Q8_1; llama.cpp CPU uses Q8_0 for Q8_0 weights). Others get it the same way IQ3_S users do: pack their own GGUF with tools/iq_pack.py --compat-bf16 (mandatory for this checkpoint; the packer refuses without it, by design), share the family MTP drafter, launch. Cost, stated plainly: experts stream 2.51 GB per token against IQ3_S's 0.61, so decode runs ~21-23 tok/s against 38.7 on the reference 4-card rig (~51% expert-cache hits, 3,637 of 24,576 pairs resident); prefill is chunk-bound - 115 tok/s at a 2048-token chunk, 202 at 4096, and the engine clamps an explicit chunk to what every cache participant can lend. The arena needs numactl --interleave=0,1 on dual-socket boards. Quality is the point of Q8_0; the trade is deliberate. Co-Authored-By: Claude Code <noreply@anthropic.com>

Niko1221 · 2026-09-30T18:43:34Z

Thanks! This overlaps the Unsloth UD-Q4_K_XL port we're doing now (see #245). We're taking your pieces into it with credit: the Q8_0 expert kernels (cross-checked against the fork's), the per-role shard column in your comma form (an old engine fails loudly on it), the split-key validation, and the ple_conv1d F32->F16 fix with the refusal of wide conv1d packs. Not taken: THP on by default (measured neutral; it would have to be opt-in) and the carried #216 commits (in 0.1.30). Closing in favour of the port.

…tensors from their owning shard, per-role expert files (native_experts.txt v4) Groundwork for Unsloth's UD-Q4_K_XL (4 shards: shard 1 metadata only, output/token_embd/PLE table in shard 2, layer 11's down in shard 2 and its gate/up in shard 3). Existing packs load unchanged. - include/strata/artifact/gguf_split.hpp: gguf_split_paths (from eddoursul/Strata 8029fa9) finds the shards from any shard's name; a missing shard is an error naming it. It replaces generate.cpp's model_shards(), which skipped a shard it could not open. - gguf_reader.hpp: GgufModel (8029fa9) looks a tensor up across the shards and refuses a name in two shards; the split keys are validated as #255 (gopinath87607) does for the head: shard 1 carries the architecture, later shards declare split.count / split.no / split.tensors.count consistently, and the directories add up to split.tensors.count. tensor_payload_bytes / in_bounds for payload checks. - NativeHead::load / NativeEmbed::load take the shard list: the architecture from the metadata shard, the tensor from the shard that holds it; the embedding also checks its payload is in the file. - generate: --native resolves all shards once (native_shards), the head from --native-head-gguf's family, the model keys from the metadata shard; --ple-gguf defaults to the shard that holds per_layer_token_embd.weight. - native_experts.txt: the shard column may be per role, `gate,up,down` (#255's comma form; empty = the --native shard). The reader accepts v2-v4 and refuses a higher version; an engine older than this one fails to open a "gate,up,down" file loudly. Fixes the parse's own aliasing hazard (parts.assign(3, parts[0])). - load_experts_gguf reads each role from its file, after check_experts_gguf proved every (file, offset) is that tensor, of the pack's type and dimensions, and inside its file. - Tests (synthetic, no model, no GPU): gguf_split_test (4 shards, missing shard, duplicate, split keys, count, truncation); expert_layout_test (copies of the Q2_0/IQ2_XS/IQ3_XXS/IQ3_S/Coder/Swift packs' native_experts.txt parse field for field as before; v4 column; v5 / 4 / 2 names refused; the arena from split synthetic shards at 2560/640 with Q4_K/Q5_1 and Q5_K/Q8_0, every byte compared, and wrong offset / file / type / shape / truncated / missing shard refused). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…split keys checked, experts.bin reused only with its source sidecar, atomic publication - Each expert role is resolved in the shard that holds it and its offset is absolute in THAT shard (the first role's data_start was used for all three; #255, gopinath87607). A layer whose roles are in different shards gets the per-role column `gate,up,down` and the file becomes v4; every other pack keeps the v3 header and bytes (checked on Q2_0, IQ3_XXS, IQ3_S, Coder IQ1_M, Swift IQ2_XS: index.txt, native_experts.txt and dense.bin identical to main's packer). - The split keys are checked as the engine does (shard 1 has the metadata; split.count / split.no / split.tensors.count consistent). The expert table is computed and checked before any file of the pack changes. - experts.bin is written under a temporary name and gets experts.bin.src.json (the shards' names and sizes, the sha256 of native_experts.txt); it is reused only when that matches, never by size. An experts.bin of another source is refused without --experts-bin (the engine would read it). - dense.bin and index.txt are published together after the old completion marker (native_experts.txt) is removed, so a stop part-way never mixes files of two runs. - .absolute() stays (not .resolve()): symlinked HF snapshot names keep the split naming. - Tests: a synthetic 4-shard GGUF (shard 1 metadata only, layer 1 split per role; offsets per shard, experts.bin byte for byte), v3 kept when nothing is split, missing shard, duplicate tensor, a shard of another split, truncated shard, wrong type (pack untouched), interrupted pack (experts and index), stale same-size experts.bin. Idea of per-role names also in #247 (jagsan-cyber). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…BF16 by default, PLE conv F32 -> F16 by the loader, conversions.json; the engine refuses a wide conv1d - FORM (eddoursul/Strata's list of which tensor the engine reads as BF16, F16 or F32) decides the pack form; the conversion itself is upstream's bf16_bytes (gguf-py, finite check, round-to-nearest- even). By default only lossless conversions happen: F32 whose values are exactly BF16 (low 16 bits zero; before, only the routers) and F16/BF16 widened to F32. Rounding (inexact F32, Q8_0 hyper- connection / PLE value / output_hc, F16 -> BF16) still needs --compat-bf16. NaN/Inf are refused even where the low bits are zero. - A Q8_0 PLE key stays native (served from the GGUF, like Q2_0); other quantized keys take the BF16 path as before. (The engine's native PLE key learns Q8_0 in the formats change.) - blk.1.ple_conv1d stored F32 (Q8_0 / UD-Q4_K_XL files) goes into dense.bin as index kind 3 and the loader narrows it to F16 (#255, gopinath87607), with or without --compat-bf16; a value beyond F16's range is refused. generate refuses a pack whose conv1d is not F16 (#255's check, which also accepts the canonical Q2_0 pack's verbatim-F16 kind 0 - #255 as written refuses that pack). - conversions.json (schema 1): the source shards' names and sizes, and per converted tensor its shard, source/destination type, method, exact flag, largest absolute error and source sha256. compat-bf16.json is written as before. Refusals happen before anything is published. - Byte identity: index.txt, native_experts.txt and dense.bin of Q2_0, IQ3_XXS, IQ3_S, Coder IQ1_M and Swift IQ2_XS are identical to main's packer (Swift's 96 exact F32 routers are now listed in conversions.json). - Tests: exact F32->BF16 by default, inexact only with --compat-bf16 (error bound), NaN/+-Inf refused with and without the flag (previous pack untouched), conv1d kind 3 (bytes, record, F16 kept, overflow refused), Q8_0 projections need the flag and are recorded, Q8_0 key native. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…_XL), per-role format lists, native_expert_supported at startup, Q8_0 embedding and PLE key - Dots: llama.cpp's vec_dot_q4_K / q5_K / q8_0_q8_1 transcribed (VDR 2), via eddoursul/Strata 8029fa9 and #255 (gopinath87607, Q8_0), which agree. Q5_1's min term multiplies the sum of the QUANTIZED activations (ggml-cpu's convention, as the fork does), not llama.cpp CUDA's ds.y. - Dequantizers dq_q4_k / dq_q5_k / dq_q5_1 / dq_q8_0 (llama.cpp's formulas) for the prompt path's FP16 fallback, f32 dumps and the Q8_0 embedding; iq_row_bytes / is_iq know the four types; iq_dequant_gu_f16 now refuses an unknown type instead of leaving its output unwritten. - The grouped gate/up, down and iq_mmvq switches are generated from STRATA_GU_FMTS / STRATA_D_FMTS / STRATA_MMVQ_FMTS; the existing entries launch the same kernels as before. - native_expert_supported(gu, d, n_embd, n_ff): generate checks every layer of a native pack right after expert_layout_load and refuses an unsupported pair by name before anything is allocated (it used to exit from inside the first verify window). - Native PLE key: Q8_0 accepted (NativeDense, generate, ple.cu); native_mmvq already runs it. - native_expert_parity: --synthetic GU/DOWN (random ggml-quantized blobs at 2560/640; port of the fork's mode), GPU dequant f32/f16 and embedding rows vs ggml to_float bit for bit, split models via GgufModel, --q5_1-min. CTests: q4_K/q5_1, q4_K/q8_0, q5_K/q8_0, q8_0/q8_0, the Q5_1 min term, and q6_K refused. RTX 5070: GPU and CPU within 1.25-1.37e-2 of the float reference for all three real pairs; every dequantized value identical; Q5_1 GPU vs ggml-cpu 7.5e-5 (vs 1.95e-2 for the ds.y variant); IQ3_XXS/IQ4_NL and IQ2_XS/Q2_0 unchanged. - Not yet: #242's decode-once traits for the new types (they take the per-entry kernel; see phaseA-tests/s5_cs3_design.md), prompt MMQ for Q4_K/Q5_K/Q5_1, identity gates. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

… the mapped mode without experts.bin, --resident-budget-gib, the tier counters - docs/UNSLOTH_Q4.md: what UD-Q4_K_XL is (Hugging Face revision 38bb39e, the four shards with sizes and SHA-256), disk/RAM/GPU needs (111.3 GB download, 1.4 GB pack, no experts.bin), the iq_pack.py command and why --compat-bf16, the numerical policy (native experts/embedding/head/projections/PLE; the converted small tensors listed in conversions.json), the MTP draft layer, a server config (--native shard 1, --ple-gguf found by name, --resident-budget-gib N with N = RAM minus 20-24 GB), measured speed on an RTX 5070 / 64 GB PC (7-8.5 tok/s at 40 GiB; ~60 s to the first answer), the switches (STRATA_LOOKAHEAD=0, STRATA_KQ256=1, STRATA_PARTIAL_PIN=1, STRATA_FETCH_THREADS), scope and validation, credits (eddoursul #245, gopinath87607 #255, jagsan-cyber #247). Experimental, not in setup. - docs/DETAILS.md: the mapped low-RAM mode reads a native pack's experts from the GGUF when it has no experts.bin (checked, identical answers on the Coder); --resident-budget-gib and the routing-aware prefetch; where the tier counters are (--stats, the server log, /metrics' ram_blobs / file_blobs / file_mb); a pointer from "Which model?". Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…tensors from their owning shard, per-role expert files (native_experts.txt v4) Groundwork for Unsloth's UD-Q4_K_XL (4 shards: shard 1 metadata only, output/token_embd/PLE table in shard 2, layer 11's down in shard 2 and its gate/up in shard 3). Existing packs load unchanged. - include/strata/artifact/gguf_split.hpp: gguf_split_paths (from eddoursul/Strata 8029fa9) finds the shards from any shard's name; a missing shard is an error naming it. It replaces generate.cpp's model_shards(), which skipped a shard it could not open. - gguf_reader.hpp: GgufModel (8029fa9) looks a tensor up across the shards and refuses a name in two shards; the split keys are validated as #255 (gopinath87607) does for the head: shard 1 carries the architecture, later shards declare split.count / split.no / split.tensors.count consistently, and the directories add up to split.tensors.count. tensor_payload_bytes / in_bounds for payload checks. - NativeHead::load / NativeEmbed::load take the shard list: the architecture from the metadata shard, the tensor from the shard that holds it; the embedding also checks its payload is in the file. - generate: --native resolves all shards once (native_shards), the head from --native-head-gguf's family, the model keys from the metadata shard; --ple-gguf defaults to the shard that holds per_layer_token_embd.weight. - native_experts.txt: the shard column may be per role, `gate,up,down` (#255's comma form; empty = the --native shard). The reader accepts v2-v4 and refuses a higher version; an engine older than this one fails to open a "gate,up,down" file loudly. Fixes the parse's own aliasing hazard (parts.assign(3, parts[0])). - load_experts_gguf reads each role from its file, after check_experts_gguf proved every (file, offset) is that tensor, of the pack's type and dimensions, and inside its file. - Tests (synthetic, no model, no GPU): gguf_split_test (4 shards, missing shard, duplicate, split keys, count, truncation); expert_layout_test (copies of the Q2_0/IQ2_XS/IQ3_XXS/IQ3_S/Coder/Swift packs' native_experts.txt parse field for field as before; v4 column; v5 / 4 / 2 names refused; the arena from split synthetic shards at 2560/640 with Q4_K/Q5_1 and Q5_K/Q8_0, every byte compared, and wrong offset / file / type / shape / truncated / missing shard refused).

…split keys checked, experts.bin reused only with its source sidecar, atomic publication - Each expert role is resolved in the shard that holds it and its offset is absolute in THAT shard (the first role's data_start was used for all three; #255, gopinath87607). A layer whose roles are in different shards gets the per-role column `gate,up,down` and the file becomes v4; every other pack keeps the v3 header and bytes (checked on Q2_0, IQ3_XXS, IQ3_S, Coder IQ1_M, Swift IQ2_XS: index.txt, native_experts.txt and dense.bin identical to main's packer). - The split keys are checked as the engine does (shard 1 has the metadata; split.count / split.no / split.tensors.count consistent). The expert table is computed and checked before any file of the pack changes. - experts.bin is written under a temporary name and gets experts.bin.src.json (the shards' names and sizes, the sha256 of native_experts.txt); it is reused only when that matches, never by size. An experts.bin of another source is refused without --experts-bin (the engine would read it). - dense.bin and index.txt are published together after the old completion marker (native_experts.txt) is removed, so a stop part-way never mixes files of two runs. - .absolute() stays (not .resolve()): symlinked HF snapshot names keep the split naming. - Tests: a synthetic 4-shard GGUF (shard 1 metadata only, layer 1 split per role; offsets per shard, experts.bin byte for byte), v3 kept when nothing is split, missing shard, duplicate tensor, a shard of another split, truncated shard, wrong type (pack untouched), interrupted pack (experts and index), stale same-size experts.bin. Idea of per-role names also in #247 (jagsan-cyber).

…BF16 by default, PLE conv F32 -> F16 by the loader, conversions.json; the engine refuses a wide conv1d - FORM (eddoursul/Strata's list of which tensor the engine reads as BF16, F16 or F32) decides the pack form; the conversion itself is upstream's bf16_bytes (gguf-py, finite check, round-to-nearest- even). By default only lossless conversions happen: F32 whose values are exactly BF16 (low 16 bits zero; before, only the routers) and F16/BF16 widened to F32. Rounding (inexact F32, Q8_0 hyper- connection / PLE value / output_hc, F16 -> BF16) still needs --compat-bf16. NaN/Inf are refused even where the low bits are zero. - A Q8_0 PLE key stays native (served from the GGUF, like Q2_0); other quantized keys take the BF16 path as before. (The engine's native PLE key learns Q8_0 in the formats change.) - blk.1.ple_conv1d stored F32 (Q8_0 / UD-Q4_K_XL files) goes into dense.bin as index kind 3 and the loader narrows it to F16 (#255, gopinath87607), with or without --compat-bf16; a value beyond F16's range is refused. generate refuses a pack whose conv1d is not F16 (#255's check, which also accepts the canonical Q2_0 pack's verbatim-F16 kind 0 - #255 as written refuses that pack). - conversions.json (schema 1): the source shards' names and sizes, and per converted tensor its shard, source/destination type, method, exact flag, largest absolute error and source sha256. compat-bf16.json is written as before. Refusals happen before anything is published. - Byte identity: index.txt, native_experts.txt and dense.bin of Q2_0, IQ3_XXS, IQ3_S, Coder IQ1_M and Swift IQ2_XS are identical to main's packer (Swift's 96 exact F32 routers are now listed in conversions.json). - Tests: exact F32->BF16 by default, inexact only with --compat-bf16 (error bound), NaN/+-Inf refused with and without the flag (previous pack untouched), conv1d kind 3 (bytes, record, F16 kept, overflow refused), Q8_0 projections need the flag and are recorded, Q8_0 key native.

…_XL), per-role format lists, native_expert_supported at startup, Q8_0 embedding and PLE key - Dots: llama.cpp's vec_dot_q4_K / q5_K / q8_0_q8_1 transcribed (VDR 2), via eddoursul/Strata 8029fa9 and #255 (gopinath87607, Q8_0), which agree. Q5_1's min term multiplies the sum of the QUANTIZED activations (ggml-cpu's convention, as the fork does), not llama.cpp CUDA's ds.y. - Dequantizers dq_q4_k / dq_q5_k / dq_q5_1 / dq_q8_0 (llama.cpp's formulas) for the prompt path's FP16 fallback, f32 dumps and the Q8_0 embedding; iq_row_bytes / is_iq know the four types; iq_dequant_gu_f16 now refuses an unknown type instead of leaving its output unwritten. - The grouped gate/up, down and iq_mmvq switches are generated from STRATA_GU_FMTS / STRATA_D_FMTS / STRATA_MMVQ_FMTS; the existing entries launch the same kernels as before. - native_expert_supported(gu, d, n_embd, n_ff): generate checks every layer of a native pack right after expert_layout_load and refuses an unsupported pair by name before anything is allocated (it used to exit from inside the first verify window). - Native PLE key: Q8_0 accepted (NativeDense, generate, ple.cu); native_mmvq already runs it. - native_expert_parity: --synthetic GU/DOWN (random ggml-quantized blobs at 2560/640; port of the fork's mode), GPU dequant f32/f16 and embedding rows vs ggml to_float bit for bit, split models via GgufModel, --q5_1-min. CTests: q4_K/q5_1, q4_K/q8_0, q5_K/q8_0, q8_0/q8_0, the Q5_1 min term, and q6_K refused. RTX 5070: GPU and CPU within 1.25-1.37e-2 of the float reference for all three real pairs; every dequantized value identical; Q5_1 GPU vs ggml-cpu 7.5e-5 (vs 1.95e-2 for the ds.y variant); IQ3_XXS/IQ4_NL and IQ2_XS/Q2_0 unchanged. - Not yet: #242's decode-once traits for the new types (they take the per-entry kernel; see phaseA-tests/s5_cs3_design.md), prompt MMQ for Q4_K/Q5_K/Q5_1, identity gates.

… the mapped mode without experts.bin, --resident-budget-gib, the tier counters - docs/UNSLOTH_Q4.md: what UD-Q4_K_XL is (Hugging Face revision 38bb39e, the four shards with sizes and SHA-256), disk/RAM/GPU needs (111.3 GB download, 1.4 GB pack, no experts.bin), the iq_pack.py command and why --compat-bf16, the numerical policy (native experts/embedding/head/projections/PLE; the converted small tensors listed in conversions.json), the MTP draft layer, a server config (--native shard 1, --ple-gguf found by name, --resident-budget-gib N with N = RAM minus 20-24 GB), measured speed on an RTX 5070 / 64 GB PC (7-8.5 tok/s at 40 GiB; ~60 s to the first answer), the switches (STRATA_LOOKAHEAD=0, STRATA_KQ256=1, STRATA_PARTIAL_PIN=1, STRATA_FETCH_THREADS), scope and validation, credits (eddoursul #245, gopinath87607 #255, jagsan-cyber #247). Experimental, not in setup. - docs/DETAILS.md: the mapped low-RAM mode reads a native pack's experts from the GGUF when it has no experts.bin (checked, identical answers on the Coder); --resident-budget-gib and the routing-aware prefetch; where the tier counters are (--stats, the server log, /metrics' ram_blobs / file_blobs / file_mb); a pointer from "Which model?".

…tensors from their owning shard, per-role expert files (native_experts.txt v4) Groundwork for Unsloth's UD-Q4_K_XL (4 shards: shard 1 metadata only, output/token_embd/PLE table in shard 2, layer 11's down in shard 2 and its gate/up in shard 3). Existing packs load unchanged. - include/strata/artifact/gguf_split.hpp: gguf_split_paths (from eddoursul/Strata 8029fa9) finds the shards from any shard's name; a missing shard is an error naming it. It replaces generate.cpp's model_shards(), which skipped a shard it could not open. - gguf_reader.hpp: GgufModel (8029fa9) looks a tensor up across the shards and refuses a name in two shards; the split keys are validated as Niko1221#255 (gopinath87607) does for the head: shard 1 carries the architecture, later shards declare split.count / split.no / split.tensors.count consistently, and the directories add up to split.tensors.count. tensor_payload_bytes / in_bounds for payload checks. - NativeHead::load / NativeEmbed::load take the shard list: the architecture from the metadata shard, the tensor from the shard that holds it; the embedding also checks its payload is in the file. - generate: --native resolves all shards once (native_shards), the head from --native-head-gguf's family, the model keys from the metadata shard; --ple-gguf defaults to the shard that holds per_layer_token_embd.weight. - native_experts.txt: the shard column may be per role, `gate,up,down` (Niko1221#255's comma form; empty = the --native shard). The reader accepts v2-v4 and refuses a higher version; an engine older than this one fails to open a "gate,up,down" file loudly. Fixes the parse's own aliasing hazard (parts.assign(3, parts[0])). - load_experts_gguf reads each role from its file, after check_experts_gguf proved every (file, offset) is that tensor, of the pack's type and dimensions, and inside its file. - Tests (synthetic, no model, no GPU): gguf_split_test (4 shards, missing shard, duplicate, split keys, count, truncation); expert_layout_test (copies of the Q2_0/IQ2_XS/IQ3_XXS/IQ3_S/Coder/Swift packs' native_experts.txt parse field for field as before; v4 column; v5 / 4 / 2 names refused; the arena from split synthetic shards at 2560/640 with Q4_K/Q5_1 and Q5_K/Q8_0, every byte compared, and wrong offset / file / type / shape / truncated / missing shard refused).

…split keys checked, experts.bin reused only with its source sidecar, atomic publication - Each expert role is resolved in the shard that holds it and its offset is absolute in THAT shard (the first role's data_start was used for all three; Niko1221#255, gopinath87607). A layer whose roles are in different shards gets the per-role column `gate,up,down` and the file becomes v4; every other pack keeps the v3 header and bytes (checked on Q2_0, IQ3_XXS, IQ3_S, Coder IQ1_M, Swift IQ2_XS: index.txt, native_experts.txt and dense.bin identical to main's packer). - The split keys are checked as the engine does (shard 1 has the metadata; split.count / split.no / split.tensors.count consistent). The expert table is computed and checked before any file of the pack changes. - experts.bin is written under a temporary name and gets experts.bin.src.json (the shards' names and sizes, the sha256 of native_experts.txt); it is reused only when that matches, never by size. An experts.bin of another source is refused without --experts-bin (the engine would read it). - dense.bin and index.txt are published together after the old completion marker (native_experts.txt) is removed, so a stop part-way never mixes files of two runs. - .absolute() stays (not .resolve()): symlinked HF snapshot names keep the split naming. - Tests: a synthetic 4-shard GGUF (shard 1 metadata only, layer 1 split per role; offsets per shard, experts.bin byte for byte), v3 kept when nothing is split, missing shard, duplicate tensor, a shard of another split, truncated shard, wrong type (pack untouched), interrupted pack (experts and index), stale same-size experts.bin. Idea of per-role names also in Niko1221#247 (jagsan-cyber).

…BF16 by default, PLE conv F32 -> F16 by the loader, conversions.json; the engine refuses a wide conv1d - FORM (eddoursul/Strata's list of which tensor the engine reads as BF16, F16 or F32) decides the pack form; the conversion itself is upstream's bf16_bytes (gguf-py, finite check, round-to-nearest- even). By default only lossless conversions happen: F32 whose values are exactly BF16 (low 16 bits zero; before, only the routers) and F16/BF16 widened to F32. Rounding (inexact F32, Q8_0 hyper- connection / PLE value / output_hc, F16 -> BF16) still needs --compat-bf16. NaN/Inf are refused even where the low bits are zero. - A Q8_0 PLE key stays native (served from the GGUF, like Q2_0); other quantized keys take the BF16 path as before. (The engine's native PLE key learns Q8_0 in the formats change.) - blk.1.ple_conv1d stored F32 (Q8_0 / UD-Q4_K_XL files) goes into dense.bin as index kind 3 and the loader narrows it to F16 (Niko1221#255, gopinath87607), with or without --compat-bf16; a value beyond F16's range is refused. generate refuses a pack whose conv1d is not F16 (Niko1221#255's check, which also accepts the canonical Q2_0 pack's verbatim-F16 kind 0 - Niko1221#255 as written refuses that pack). - conversions.json (schema 1): the source shards' names and sizes, and per converted tensor its shard, source/destination type, method, exact flag, largest absolute error and source sha256. compat-bf16.json is written as before. Refusals happen before anything is published. - Byte identity: index.txt, native_experts.txt and dense.bin of Q2_0, IQ3_XXS, IQ3_S, Coder IQ1_M and Swift IQ2_XS are identical to main's packer (Swift's 96 exact F32 routers are now listed in conversions.json). - Tests: exact F32->BF16 by default, inexact only with --compat-bf16 (error bound), NaN/+-Inf refused with and without the flag (previous pack untouched), conv1d kind 3 (bytes, record, F16 kept, overflow refused), Q8_0 projections need the flag and are recorded, Q8_0 key native.

…_XL), per-role format lists, native_expert_supported at startup, Q8_0 embedding and PLE key - Dots: llama.cpp's vec_dot_q4_K / q5_K / q8_0_q8_1 transcribed (VDR 2), via eddoursul/Strata 8029fa9 and Niko1221#255 (gopinath87607, Q8_0), which agree. Q5_1's min term multiplies the sum of the QUANTIZED activations (ggml-cpu's convention, as the fork does), not llama.cpp CUDA's ds.y. - Dequantizers dq_q4_k / dq_q5_k / dq_q5_1 / dq_q8_0 (llama.cpp's formulas) for the prompt path's FP16 fallback, f32 dumps and the Q8_0 embedding; iq_row_bytes / is_iq know the four types; iq_dequant_gu_f16 now refuses an unknown type instead of leaving its output unwritten. - The grouped gate/up, down and iq_mmvq switches are generated from STRATA_GU_FMTS / STRATA_D_FMTS / STRATA_MMVQ_FMTS; the existing entries launch the same kernels as before. - native_expert_supported(gu, d, n_embd, n_ff): generate checks every layer of a native pack right after expert_layout_load and refuses an unsupported pair by name before anything is allocated (it used to exit from inside the first verify window). - Native PLE key: Q8_0 accepted (NativeDense, generate, ple.cu); native_mmvq already runs it. - native_expert_parity: --synthetic GU/DOWN (random ggml-quantized blobs at 2560/640; port of the fork's mode), GPU dequant f32/f16 and embedding rows vs ggml to_float bit for bit, split models via GgufModel, --q5_1-min. CTests: q4_K/q5_1, q4_K/q8_0, q5_K/q8_0, q8_0/q8_0, the Q5_1 min term, and q6_K refused. RTX 5070: GPU and CPU within 1.25-1.37e-2 of the float reference for all three real pairs; every dequantized value identical; Q5_1 GPU vs ggml-cpu 7.5e-5 (vs 1.95e-2 for the ds.y variant); IQ3_XXS/IQ4_NL and IQ2_XS/Q2_0 unchanged. - Not yet: Niko1221#242's decode-once traits for the new types (they take the per-entry kernel; see phaseA-tests/s5_cs3_design.md), prompt MMQ for Q4_K/Q5_K/Q5_1, identity gates.

… the mapped mode without experts.bin, --resident-budget-gib, the tier counters - docs/UNSLOTH_Q4.md: what UD-Q4_K_XL is (Hugging Face revision 38bb39e, the four shards with sizes and SHA-256), disk/RAM/GPU needs (111.3 GB download, 1.4 GB pack, no experts.bin), the iq_pack.py command and why --compat-bf16, the numerical policy (native experts/embedding/head/projections/PLE; the converted small tensors listed in conversions.json), the MTP draft layer, a server config (--native shard 1, --ple-gguf found by name, --resident-budget-gib N with N = RAM minus 20-24 GB), measured speed on an RTX 5070 / 64 GB PC (7-8.5 tok/s at 40 GiB; ~60 s to the first answer), the switches (STRATA_LOOKAHEAD=0, STRATA_KQ256=1, STRATA_PARTIAL_PIN=1, STRATA_FETCH_THREADS), scope and validation, credits (eddoursul Niko1221#245, gopinath87607 Niko1221#255, jagsan-cyber Niko1221#247). Experimental, not in setup. - docs/DETAILS.md: the mapped low-RAM mode reads a native pack's experts from the GGUF when it has no experts.bin (checked, identical answers on the Coder); --resident-budget-gib and the routing-aware prefetch; where the tier counters are (--stats, the server log, /metrics' ram_blobs / file_blobs / file_mb); a pointer from "Which model?".

…tensors from their owning shard, per-role expert files (native_experts.txt v4) Groundwork for Unsloth's UD-Q4_K_XL (4 shards: shard 1 metadata only, output/token_embd/PLE table in shard 2, layer 11's down in shard 2 and its gate/up in shard 3). Existing packs load unchanged. - include/strata/artifact/gguf_split.hpp: gguf_split_paths (from eddoursul/Strata 8029fa9) finds the shards from any shard's name; a missing shard is an error naming it. It replaces generate.cpp's model_shards(), which skipped a shard it could not open. - gguf_reader.hpp: GgufModel (8029fa9) looks a tensor up across the shards and refuses a name in two shards; the split keys are validated as Niko1221#255 (gopinath87607) does for the head: shard 1 carries the architecture, later shards declare split.count / split.no / split.tensors.count consistently, and the directories add up to split.tensors.count. tensor_payload_bytes / in_bounds for payload checks. - NativeHead::load / NativeEmbed::load take the shard list: the architecture from the metadata shard, the tensor from the shard that holds it; the embedding also checks its payload is in the file. - generate: --native resolves all shards once (native_shards), the head from --native-head-gguf's family, the model keys from the metadata shard; --ple-gguf defaults to the shard that holds per_layer_token_embd.weight. - native_experts.txt: the shard column may be per role, `gate,up,down` (Niko1221#255's comma form; empty = the --native shard). The reader accepts v2-v4 and refuses a higher version; an engine older than this one fails to open a "gate,up,down" file loudly. Fixes the parse's own aliasing hazard (parts.assign(3, parts[0])). - load_experts_gguf reads each role from its file, after check_experts_gguf proved every (file, offset) is that tensor, of the pack's type and dimensions, and inside its file. - Tests (synthetic, no model, no GPU): gguf_split_test (4 shards, missing shard, duplicate, split keys, count, truncation); expert_layout_test (copies of the Q2_0/IQ2_XS/IQ3_XXS/IQ3_S/Coder/Swift packs' native_experts.txt parse field for field as before; v4 column; v5 / 4 / 2 names refused; the arena from split synthetic shards at 2560/640 with Q4_K/Q5_1 and Q5_K/Q8_0, every byte compared, and wrong offset / file / type / shape / truncated / missing shard refused). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…split keys checked, experts.bin reused only with its source sidecar, atomic publication - Each expert role is resolved in the shard that holds it and its offset is absolute in THAT shard (the first role's data_start was used for all three; Niko1221#255, gopinath87607). A layer whose roles are in different shards gets the per-role column `gate,up,down` and the file becomes v4; every other pack keeps the v3 header and bytes (checked on Q2_0, IQ3_XXS, IQ3_S, Coder IQ1_M, Swift IQ2_XS: index.txt, native_experts.txt and dense.bin identical to main's packer). - The split keys are checked as the engine does (shard 1 has the metadata; split.count / split.no / split.tensors.count consistent). The expert table is computed and checked before any file of the pack changes. - experts.bin is written under a temporary name and gets experts.bin.src.json (the shards' names and sizes, the sha256 of native_experts.txt); it is reused only when that matches, never by size. An experts.bin of another source is refused without --experts-bin (the engine would read it). - dense.bin and index.txt are published together after the old completion marker (native_experts.txt) is removed, so a stop part-way never mixes files of two runs. - .absolute() stays (not .resolve()): symlinked HF snapshot names keep the split naming. - Tests: a synthetic 4-shard GGUF (shard 1 metadata only, layer 1 split per role; offsets per shard, experts.bin byte for byte), v3 kept when nothing is split, missing shard, duplicate tensor, a shard of another split, truncated shard, wrong type (pack untouched), interrupted pack (experts and index), stale same-size experts.bin. Idea of per-role names also in Niko1221#247 (jagsan-cyber). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…BF16 by default, PLE conv F32 -> F16 by the loader, conversions.json; the engine refuses a wide conv1d - FORM (eddoursul/Strata's list of which tensor the engine reads as BF16, F16 or F32) decides the pack form; the conversion itself is upstream's bf16_bytes (gguf-py, finite check, round-to-nearest- even). By default only lossless conversions happen: F32 whose values are exactly BF16 (low 16 bits zero; before, only the routers) and F16/BF16 widened to F32. Rounding (inexact F32, Q8_0 hyper- connection / PLE value / output_hc, F16 -> BF16) still needs --compat-bf16. NaN/Inf are refused even where the low bits are zero. - A Q8_0 PLE key stays native (served from the GGUF, like Q2_0); other quantized keys take the BF16 path as before. (The engine's native PLE key learns Q8_0 in the formats change.) - blk.1.ple_conv1d stored F32 (Q8_0 / UD-Q4_K_XL files) goes into dense.bin as index kind 3 and the loader narrows it to F16 (Niko1221#255, gopinath87607), with or without --compat-bf16; a value beyond F16's range is refused. generate refuses a pack whose conv1d is not F16 (Niko1221#255's check, which also accepts the canonical Q2_0 pack's verbatim-F16 kind 0 - Niko1221#255 as written refuses that pack). - conversions.json (schema 1): the source shards' names and sizes, and per converted tensor its shard, source/destination type, method, exact flag, largest absolute error and source sha256. compat-bf16.json is written as before. Refusals happen before anything is published. - Byte identity: index.txt, native_experts.txt and dense.bin of Q2_0, IQ3_XXS, IQ3_S, Coder IQ1_M and Swift IQ2_XS are identical to main's packer (Swift's 96 exact F32 routers are now listed in conversions.json). - Tests: exact F32->BF16 by default, inexact only with --compat-bf16 (error bound), NaN/+-Inf refused with and without the flag (previous pack untouched), conv1d kind 3 (bytes, record, F16 kept, overflow refused), Q8_0 projections need the flag and are recorded, Q8_0 key native. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…_XL), per-role format lists, native_expert_supported at startup, Q8_0 embedding and PLE key - Dots: llama.cpp's vec_dot_q4_K / q5_K / q8_0_q8_1 transcribed (VDR 2), via eddoursul/Strata 8029fa9 and Niko1221#255 (gopinath87607, Q8_0), which agree. Q5_1's min term multiplies the sum of the QUANTIZED activations (ggml-cpu's convention, as the fork does), not llama.cpp CUDA's ds.y. - Dequantizers dq_q4_k / dq_q5_k / dq_q5_1 / dq_q8_0 (llama.cpp's formulas) for the prompt path's FP16 fallback, f32 dumps and the Q8_0 embedding; iq_row_bytes / is_iq know the four types; iq_dequant_gu_f16 now refuses an unknown type instead of leaving its output unwritten. - The grouped gate/up, down and iq_mmvq switches are generated from STRATA_GU_FMTS / STRATA_D_FMTS / STRATA_MMVQ_FMTS; the existing entries launch the same kernels as before. - native_expert_supported(gu, d, n_embd, n_ff): generate checks every layer of a native pack right after expert_layout_load and refuses an unsupported pair by name before anything is allocated (it used to exit from inside the first verify window). - Native PLE key: Q8_0 accepted (NativeDense, generate, ple.cu); native_mmvq already runs it. - native_expert_parity: --synthetic GU/DOWN (random ggml-quantized blobs at 2560/640; port of the fork's mode), GPU dequant f32/f16 and embedding rows vs ggml to_float bit for bit, split models via GgufModel, --q5_1-min. CTests: q4_K/q5_1, q4_K/q8_0, q5_K/q8_0, q8_0/q8_0, the Q5_1 min term, and q6_K refused. RTX 5070: GPU and CPU within 1.25-1.37e-2 of the float reference for all three real pairs; every dequantized value identical; Q5_1 GPU vs ggml-cpu 7.5e-5 (vs 1.95e-2 for the ds.y variant); IQ3_XXS/IQ4_NL and IQ2_XS/Q2_0 unchanged. - Not yet: Niko1221#242's decode-once traits for the new types (they take the per-entry kernel; see phaseA-tests/s5_cs3_design.md), prompt MMQ for Q4_K/Q5_K/Q5_1, identity gates. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

… the mapped mode without experts.bin, --resident-budget-gib, the tier counters - docs/UNSLOTH_Q4.md: what UD-Q4_K_XL is (Hugging Face revision 38bb39e, the four shards with sizes and SHA-256), disk/RAM/GPU needs (111.3 GB download, 1.4 GB pack, no experts.bin), the iq_pack.py command and why --compat-bf16, the numerical policy (native experts/embedding/head/projections/PLE; the converted small tensors listed in conversions.json), the MTP draft layer, a server config (--native shard 1, --ple-gguf found by name, --resident-budget-gib N with N = RAM minus 20-24 GB), measured speed on an RTX 5070 / 64 GB PC (7-8.5 tok/s at 40 GiB; ~60 s to the first answer), the switches (STRATA_LOOKAHEAD=0, STRATA_KQ256=1, STRATA_PARTIAL_PIN=1, STRATA_FETCH_THREADS), scope and validation, credits (eddoursul Niko1221#245, gopinath87607 Niko1221#255, jagsan-cyber Niko1221#247). Experimental, not in setup. - docs/DETAILS.md: the mapped low-RAM mode reads a native pack's experts from the GGUF when it has no experts.bin (checked, identical answers on the Coder); --resident-budget-gib and the routing-aware prefetch; where the tier counters are (--stats, the server log, /metrics' ram_blobs / file_blobs / file_mb); a pointer from "Which model?". Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…ership (Niko1221#255) STRATA_PIPED_CONCURRENT is accepted again: the prompt is read on a thread of its own while the slot groups keep decoding through the same pump, instead of stopping for the read. - The reader calls Prefill::run only. The host thread stays the single ResidentSlots submitter: it pumps windows, consumes the ordered control prefix (a BSTOP/BPRIORITY past a queued admission still waits) and stops the read at its next chunk on BYIELD. A part read gives way through the existing ResidentSlots::park, extracted as park_partial and shared with the sequential controls; no mutable BSlot write and no copy_to_slot. - The slots' doorbells are served on a thread of their own under Verifier::set_external_serve(true) (the half that had no caller), so neither the reader nor the host thread makes the CUDA call a window may be waiting on (Niko1221#31). pump wakes it after each launch; RAII clears the external-serve flag and joins on every exit, including failures. - Prefill::stop_at_chunk/stopped_at, reserve_uploads, MtpDrafter::reserve_prefill and ConcurrentReadSetup are the seams; the mode is off beside the foreground budget, batched MTP, packed/retained prefill, resident multiplexing, the judge lane, --adapt-async/--spec-split, and every source the existing ConcurrentReadSetup refuses, with one printed reason. - A non-pipelined --batch reserves one PGroup so the same pump can drive it; the pipelined summary line stays piped-only. Scope: the engine builds, and the extracted read_part harness (which compiles the real concurrent code on the CPU) passes. NOT device-exercised: the only model here is the 78 GB Flash-Next pack, which needs --windowed-experts, and the gate deliberately excludes staged/windowed sources; without it the three-GPU admission refuses the 121 GB demand, and no smaller qwen4exp model exists on this box. The ticket's one-GPU token-identity/cancel/drain acceptance is therefore still open.

`--retire-resident-source` is how the daily profile serves a fully resident model: after the startup fill the windowed ring is retired and can hand out no more buffers, so the concurrent reader cannot share its recycling with the decoding windows. Expose that state as `ExpertSource::io_retired()` (true only for a retired `WindowedExpertSource`) and let the concurrent gate accept it: every other non-file/non-arena source still refuses. This removes the I/O-safety objection for the fully-resident case. Two daily flags still keep the mode off there - `--foreground-budget-ms` (the reader loop ignores budget quanta) and `--batch-mtp*` (the drafter is shared with the decode side) - and both remain explicit refusals with one printed reason. Built; the extracted read_part harness still passes. Not device-exercised: a retired ring requires --host-native, which requires the MPS ceiling the operator's supervisor declares, so this session cannot start that configuration.

…arger profile (Niko1221#255 follow-up) A layer-split stage sizes its expert cache from the profile and, when the profile ranks more pairs than fit, used to end the start with "complete local residency does not fit". That is true of the two-V100 pair for this model: 2x32 GB admits ~18,557 of the 24,576 pairs at 256k context. The sizing loop already kept the hottest pairs that fit; the fill now keeps exactly those and streams the rest, with one line naming both counts. The cache still opens fully populated, which the windowed source requires, and the phase-3 report reads "N of its M profiled pairs". A profile that fits (the 3-GPU daily) is unchanged. This makes the pair configuration possible without hand-truncating a profile per split; it does not make all 24,576 pairs resident, which 64 GB cannot hold.

The Niko1221#255 follow-up truncated each stage's fill order to `sized` before the loop that fills `sized`, so every split stage resized its order to the empty vector it had not built yet: "the profile ranks 9728 pairs, this stage admits 0 slots", then "expert cache: no room", and the 3-GPU daily profile could no longer start. The two-V100 case the change was written for hit the same empty `sized`. Run the sizing loop first and truncate afterwards. The fill order stays the sized prefix, which open_sized requires: slot i carries the capacity of the pair at order[i]. Found while installing the MTP frontier fix: a binary built from 6de7e0f refused startup on the daily profile before any request was served.

Three gates refused --prefill-pipe compositions by returning a process exit: the setup check against retained prefill and the budgeted 1024 bisection, the request path that resumes a partial handoff, and the park of a partial handoff. The repin record (evidence/upstream-tracking/repin-0.1.45-merge-20261018.json) names why: upstream retains ConvCheckpoint::PromptPlan through the slot handoff, which the fork's ResidentSlot does not carry yet (Niko1221#255). Until that port lands, each gate now degrades the request: a piped read keeps its slot and runs whole rather than parking a partial handoff, and a request that would resume a partial or compose with retained/bisected prefill reads unpiped. The engine keeps serving; only the pipe's benefit is lost for that request.

A piped read could not park a partial handoff: its ConvCheckpoint::PromptPlan lived only in the read's own scope, so a BYIELD had nothing to resume onto the same chunk grid and the frontend's yield path either exited or held the slot for the whole read (measured earlier: 0 yields and concurrent requests waiting 59-63 s against 8.6-26.1 s unpiped). The plan now travels with the partial handoff, in the same shape the pending 1024 split already uses: ResidentAdmission carries it on a park, ResidentSlot keeps it beside prefill_split_until, restored() hands it to the working record only for a full-prefix continuation, and take_prefill_plan() is consumed where the resumed read starts. The read then validates and adopts it (read_part already had that branch, and its alignment, end and pipe-key checks re-plan safely if anything differs). The two refusals that died this path - the park itself and the resume of a partial handoff - are removed: they were placeholders for exactly this port.

gopinath87607 mentioned this pull request Sep 30, 2026

Multi-GPU: the session carve and the per-stage prompt loans #216

Closed

Gopi and others added 2 commits September 30, 2026 21:10

gopinath87607 force-pushed the q8-0-support branch from 11c3e9d to ee878b2 Compare September 30, 2026 15:43

Niko1221 closed this Sep 30, 2026

Navigation Menu

Repository navigation

Q8 0 support - #255

Q8 0 support#255
gopinath87607 wants to merge 2 commits into
Niko1221:mainfrom
gopinath87607:q8-0-support

gopinath87607 commented Sep 30, 2026

Uh oh!

Niko1221 commented Sep 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

gopinath87607 commented Sep 30, 2026

Uh oh!

Niko1221 commented Sep 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants