Repository navigation
native dense: run the architecture guard only on the first shard of a split - #648
Closed
jollyroger1480 wants to merge 629 commits into
Closed
jollyroger1480 wants to merge 629 commits into
jollyroger1480 wants to merge 629 commits into
Conversation
…060 XT, Niko1221#256 by Efeisot) as community-validated cards Cut down from Niko1221#254 and Niko1221#256 onto 0.1.30's AMD tables: - setup.py: AMD_ARCHS / AMD_NAMES / ROCM_INDEXES / AMD_CARDS list gfx1101 (TheRock gfx110X-dgpu) and gfx1200 (gfx120X-all); amd_gpus() takes the sysfs root so the detection can be tested. - cmake/hip_backend.cmake: gfx1101 and gfx1200 build without the "not validated" warning (a status line names them community-validated); gfx1102 (Niko1221#192: ctest only) keeps the warning. - docs/AMD_HIP.md (Community-validated cards: the owners' setups, ctest and numbers) and README. - tools/test_setup_amd.py: detection on a mocked KFD topology (every family, the CPU node, an iGPU and gfx1102 listed as not supported, product-name fallbacks, the ROCm index per family). Left out: the PRs' own cmake/device.cu arch checks (0.1.30's compiled_for covers them), the dp4a change (main has it), math_constants.h, the bench JSONs, the launcher, and both hipBLASLt tables (neither PR measured the prompt speed without its table). Co-Authored-By: Joseph Hohertz <jhohertz@gmail.com> Co-Authored-By: efeisot <efeproofkmr@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…D arch; the engine compiled for each)
setup refused AMD GPU lists at install and at start ("NVIDIA only for now"), although the engine's layer split runs
on two AMD cards (0.1.30, from a hand-written config).
- amd_parse_gpus(): --gpus "1,0" (setup's HIP numbering, the first card the main one) or "all" (every supported
card, the most VRAM first); every card must be in AMD_ARCHS, else a plain message with what can be used.
- install: --backend hip --gpus writes "gpu": [..] + "layer_split" like the NVIDIA path; build_engine_hip compiles
for the set of the chosen cards' archs (CMAKE_HIP_ARCHITECTURES "gfx1100;gfx1201" already worked) and records
it in BUILD.json; an engine whose archs cover the cards is not rebuilt; update_installed_engine keeps every arch.
- rocm_root(archs): cards of two families need a system ROCm (TheRock's wheels hold one family) - said plainly.
- start: --gpus on an AMD install is saved like on NVIDIA (refused when the engine has no code for a card);
--gpus all finds the AMD cards on a PC without NVIDIA cards that can share.
- docs/AMD_HIP.md, README; tools/test_setup_amd.py: lists, "all", refusals, the one-family wheel rule, the arch set
in the build and BUILD.json.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…sliced pin is opt-in (STRATA_ARENA_PIN_GIB=auto) - by default the arena pins as in 0.1.30; on this 64 GB PC the cap pinned 26 instead of 30 GiB and cost ~20% prompt speed and 2-4% IQ3_S decode Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…file reads, a partial pin of the RAM tier, bit-exact AVX2 multi-token kernels for Q4_K / Q5_1 / Q8_0 experts, decode-only tier stats Where the 288 ms/token at a 40 GiB budget went (UD-Q4_K_XL, --stats): of the 945 ms CPU pool call per verify round (3.46 tokens), 871 ms were the dispatch thread assembling missed experts from the mapped GGUF one after another (~391 blobs, 1.3 GB per round, ~1.5 GB/s: page faults from the SSD); the kernels were 42 ms gate/up and 30 ms down. At 24 GiB, 771 ms/round more went into the PCIe planning, which asked blob() - a file read with the GGUF in place - of every miss just to learn it was not pinned. - ExpertSource::prefetch: the dispatch hands the source the layer's missed experts before its jobs loop; FileExpertSource with the GGUF in place assembles them on STRATA_FETCH_THREADS threads (default 8) so the SSD sees a queue instead of one fault at a time. A no-op for every other source. Staging buffers being filled are never reused, and a blob asked while its fill runs waits for it. - The PCIe planning asks pinned() before blob(): the same decisions and bytes for every source. - --resident-budget-gib: when the driver refuses to page-lock the whole budget, the locked arena's hottest prefix (profile order) is registered with cudaHostRegister, as much as it takes (from the whole down in 2 GiB steps), so the GPU can read those misses over PCIe and the prompt path can DMA them; pinned() answers for that prefix only. STRATA_PARTIAL_PIN=0 is the A/B arm. - src/kernels/cpu/kq_avx2.cpp: ggml-cpu's AVX2 vec_dot for Q4_K x Q8_K, Q5_1 x Q8_1 and Q8_0 x Q8_0 with the token loop inside the block loop - the weight side decoded once per verify window, each token's integer chain and float operations exactly ggml's, so the results are ggml's to the bit for any group size (no Niko1221#152 rule). native_gu_rows / native_down_rows use them for those types (only UD-Q4_K_XL has them); STRATA_NO_KQ256 is the A/B arm. native_expert_parity checks them bit for bit against vec_dot for 1..8 tokens and times 4 tokens. - Stats: the decode's file tier alone (blobs, MB, MB/round, reading ms/round and GB/s per thread) and the prompt path's copies separately. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
# Conflicts: # serve/server.py # src/core/expert_source.cpp
…_K/Q5_1/Q8_0 multi-token kernels from two tokens on - Registering 30 GiB of a 40 GiB RAM tier left the driver unable to page-lock the prompt path's token id buffer afterwards (RTX 5070, WDDM: "prefill: the token id buffer"); the prefix is now capped at 24 GiB, the size a whole 24 GiB budget page-locks with, and STRATA_PARTIAL_PIN_GIB sets it. - A verify window's expert groups hold ~1.4 tokens on average; one token goes to ggml's own dot (the same bits, without the multi-token kernel's per-token arrays). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…ssed slices at once (PrefetchVirtualMemory) The copies' page faults read a few clusters each; one PrefetchVirtualMemory call for every role slice about to be copied lets the memory manager queue large reads at once, and the parallel copies then find the pages resident or in flight. Only the GGUF-in-place source's prefetch uses it; STRATA_FETCH_PVM=0 is the A/B arm. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…IN_GIB; the split's prompt chunk steps down - The expert arena was pinned at most 8 GiB whenever a layer split (or remote experts) ran, on every OS - a Windows/WDDM workaround that cost a Linux 4090 + 3060 split two thirds of its prompt speed (Niko1221#253, bettercallcaleb). The cap now applies only under WDDM (_WIN32, or WSL2 detected by /dev/dxg), for multi_gpu and expert_cache_remote alike. STRATA_ARENA_PIN_GIB overrides it on any OS: "0" pins the whole arena, N > 0 at most N GiB (arena_pin_cap_gib(), -1 when unset: the same name and meaning as the S2 branch's helper, file-local here). - The split's prompt chunk no longer stops the start when its buffers do not fit: a prompt path without a loan first gets the largest chunk whose buffers (Prefill::bytes_needed) leave 512 MiB free on its device, and an init that still fails steps down one size (8192 -> 6144 -> ... -> 2048 -> 1536 -> 1024 -> 512): Prefill::reset() frees what the failed init made, the loans are re-priced, and the failed allocation's sticky error is consumed (else the next launch check reported "out of memory"). - pinned.cu: the capped note no longer says "for CUDA1". - docs: MULTI_GPU.md, SECOND_GPU.md. Measured on amdbox (Linux, R9700 + RX 9070 XT, gpu [1, 0], Coder IQ1_M, 128 greedy tokens; tok/s): | cache | arm | 4K cold | 4K warm | 16K cold | 16K warm | decode | | auto (holds ~all) | 8 GiB cap | 678 | 1,355 | 1,709 | 1,880 | 44.5-51.5 | | | whole arena | 703 | 1,234 | 1,729 | 1,864 | 44.6-51.4 | | fixed 1500 on CUDA0 | 8 GiB cap | 608 | 1,101 | 1,287 | | 34.5-36.1 | | | whole arena | 716 | 1,160 | 1,803 | | 34.8-36.0 | (2 starts per arm for auto, 5 for fixed.) With most experts streamed the whole arena reads a 16K prompt 40% faster; with every expert already on the cards the two are within noise except 4K warm (-9%: the pinned share raises the auto lend to 90% and the streamed ring to 384 slots). Identity: the fixed-cache arms gave the same tokens in all 5 + 5 starts (4K, 16K, 4K again), and the auto arms 2 + 2. The step-down was exercised on the 9070 XT with --expert-cache 3000 --no-prefill-borrow --prefill 8192 ("need 4528 MiB, 2910 MiB free: 2048-token chunks"; with the init retry alone it went 8192 -> ... -> 2048, and 1536 ran all four requests). Known, also on main: that own-buffer configuration at 2048 makes hipBLAS pick hipBLASLt kernels whose code objects this ROCm 7.14 lacks (TensileLibrary_..._gfx1201-xnack+.co not found, "launchKernel failed") and the prompt hangs. The WDDM side (Windows, WSL2) is unchanged and not re-measured here. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…one when it holds every profiled expert + KV
Before any later stage loads, split-auto checks whether CUDA0 alone holds every profiled expert pair plus the whole
session (KV of every layer), the drafter + head, the verify windows and the reserve; if so it clears the split (one
GPU) and logs why, else it logs how much of the profile CUDA0 would hold and keeps the split. Default behaviour is
unchanged: the engine flag is opt-in, and the server passes it only for a config with "split_skip_if_fits": true
(and a GPU list). Not with remote expert caches.
Measured on amdbox (Coder IQ1_M, --expert-cache auto --prefill auto, 128 greedy tokens, gpu [1, 0] = R9700 first,
2 starts per arm, warm = the second request of each size):
| arm | 4K warm | 16K warm | decode |
| split (flag off) | 1,244 | 1,897 | 50.5-53.0 |
| --split-skip-if-fits (on) | 1,776 | 1,791 | 59.7-61.8 |
| R9700 alone | 1,787 | 1,801 | 59.9-61.3 |
The flag's run is the R9700-alone run (same tokens, all four requests); with the 9070 XT first it keeps the split
("would hold only 9.84 of the profile's 23.42 GiB"). On this pair the split still reads 16K prompts ~5% faster.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
# Conflicts: # src/core/pinned.cu
…aster on an AVX2-only CPU (Ryzen 9 3900X: gate/up -2.6..-4.7% at 1 token, down -3..-5% at 2 tokens) This reverts commit 9e9dce4. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…t an expert boundary; the Q4_K/Q5_1/Q8_0 multi-token kernels opt-in (STRATA_KQ256=1) - Locking the whole 40 GiB arena in the working set and then registering its prefix made the next device allocation fail (the prompt path's token id buffer); the registered prefix is page-locked by the driver already, so only the rest is locked now. 24 GiB registered out of a 40 GiB budget then starts and runs. - The registered prefix ends at an expert boundary: a blob that began inside it and ran past it was taken as page-locked by cudaMemcpyAsync and refused (an adaptive refill failed, with 16 GiB registered). - The prompt path's token id buffer error names the CUDA error. - kq_avx2 multi-token kernels: bit-exact, but no faster in the engine (the rows phases stayed at 72 ms per verify round at a 40 GiB budget; a window's expert groups hold ~1.4 tokens and ggml's per-token calls find the weights in L1), so they are opt-in with STRATA_KQ256=1; native_expert_parity still checks them bit for bit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…m-split down kernel (164 blocks), rs/lo/inject in up's prologue (STRATA_GR_V3=0 = old)
…); its shared memory is sized from the card's opt-in limit - one or two column halves, else the default kernels (Turing/HIP 64 KB fit with two; capability-based split as proposed by @Adamyno); the neutral knobs (DOWN_R2, UP_PF, KSPLIT) dropped (review)
…ure was generated, as its generator does; a partly missing set still fails; the open handle of a half-present fixture is closed Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…er on this layer's MoE input warms its predicted experts' pages - RouterLookahead: when the CPU pool starts layer l, a worker thread applies layer l+1's router (a BF16 host copy of ffn_gate_inp, 126 MB for 48 layers) to layer l's MoE input - the router's own input - for each token of the window, takes the top STRATA_LOOKAHEAD_K (default 10), drops the experts the GPU cache (host_res) or the RAM copy holds, and asks the source to warm the rest. It never waits: a prediction still running makes the next one skip. STRATA_LOOKAHEAD=0 is the A/B arm. - ExpertSource::warm / warms(): FileExpertSource with the GGUF in place issues one PrefetchVirtualMemory for the predicted slices (madvise(WILLNEED) elsewhere). Pages only: blob(), the experts computed and their bytes do not change (the same answers in every A/B run). Every other source: no-op, and the predictor is not started. - bf16_rows_dot_multi (kq_avx2.cpp, AVX2): the router read once for all tokens of the window. - Stats: experts warmed, how many of the decode's file-tier reads had been warmed, the predictor's time and skips. UD-Q4_K_XL at a 40 GiB budget (single-token-router version): ~48-54% of the file tier's reads had been warmed; reading time 850-968 -> 630-730 ms per verify round; decode 5.31/6.60 tok/s off vs 7.08/6.79/6.91 on. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
Add preliminary native v0.1.26 results, raw run JSON, scrubbed guest hardware facts, and the local API harness. Prompt and memory topology limitations are documented.
On the RTX 5070 / 64 GB PC the GPU computing a share of the misses over PCIe measured no faster than the CPU: 7.30 and 7.44 tok/s with 24 and 16 GiB of a 40 GiB budget registered, against 7.05-7.74 unpinned, and registering costs startup time and driver memory. The default budget mode is now locked-only; the routing-aware prefetch (multi-run A/B: 6.93 -> 7.91 tok/s mean, identical answers) stays on with the GGUF in place. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
… the mapped mode without experts.bin, --resident-budget-gib, the tier counters - docs/UNSLOTH_Q4.md: what UD-Q4_K_XL is (Hugging Face revision 38bb39e, the four shards with sizes and SHA-256), disk/RAM/GPU needs (111.3 GB download, 1.4 GB pack, no experts.bin), the iq_pack.py command and why --compat-bf16, the numerical policy (native experts/embedding/head/projections/PLE; the converted small tensors listed in conversions.json), the MTP draft layer, a server config (--native shard 1, --ple-gguf found by name, --resident-budget-gib N with N = RAM minus 20-24 GB), measured speed on an RTX 5070 / 64 GB PC (7-8.5 tok/s at 40 GiB; ~60 s to the first answer), the switches (STRATA_LOOKAHEAD=0, STRATA_KQ256=1, STRATA_PARTIAL_PIN=1, STRATA_FETCH_THREADS), scope and validation, credits (eddoursul Niko1221#245, gopinath87607 Niko1221#255, jagsan-cyber Niko1221#247). Experimental, not in setup. - docs/DETAILS.md: the mapped low-RAM mode reads a native pack's experts from the GGUF when it has no experts.bin (checked, identical answers on the Coder); --resident-budget-gib and the routing-aware prefetch; where the tier counters are (--stats, the server log, /metrics' ram_blobs / file_blobs / file_mb); a pointer from "Which model?". Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…plit GGUF, Q4_K/Q5_K/Q5_1/Q8_0 experts, RAM budget + SSD tier with routing prefetch), the low-RAM mode reads GGUF experts in place, Windows GGUF load 2x (Niko1221#230), server status race (Niko1221#266) and tokenizer (Niko1221#268) fixes, truncated tool calls (Niko1221#211/Niko1221#231), AMD intrinsics (Niko1221#262); setup requires it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…ATA_HIP_WMMA=1, gfx12 only) The v2 tensor-core design with gfx12's v_wmma_f32_16x16x16_f16: one block per (query, KV head), wave w owns dims [64w, 64w+64) (int8 scale group w), q and p split into FP16 hi + lo parts, int8 codes exact as FP16, scales in FP32, the four groups' q.k partials added in a fixed order, online softmax over 32-cell chunks. Masked (non-resident) cells get no weight, as in the decode kernel. FP32-level accuracy, not bitwise equal to the default kernel, hence opt-in. hip_prompt_attn_wmma = qsa_prompt_attn_parity built for HIP (skips off gfx12). NVIDIA builds unchanged (all new code is under __HIPCC__ / the HIP test section). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…ame (default on HIP; STRATA_HIP_ROUTER_OLD=1: old) On RDNA4 the router took 39 us per call in decode (16% of the decode GPU time): 512 dependent FP64 adds, ten selection passes with 20 block barriers, a renormalisation re-reading its weights from global memory. The sum's only use is inv = (float)(1.0 / sum): all terms are positive with the largest exactly 1, so the serial sum and a tree sum are within 2^-43 of each other; when the bracket tree*(1 -+ 2^-40) maps to one float, that float is the serial sum's (1/x and the cast are monotone), otherwise thread 0 runs the serial sum as before. The selection runs on wave 0 from registers (the argmax with the lowest index on ties is unique), the renormalisation from registers in rank order. hip_router_fast checks ids and weights bitwise against the portable kernel (65,536 rows with ties, near-ties, NaN rows, the forced serial path; and 256 experts / k 8). CUDA builds unchanged (HIP-only code). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
… unterminated Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…n's logits STRATA_DUMP_FIRST_LOGITS=<file>: the logits of the first window's first position (the token the prompt predicts), n_vocab float32, written once per run. Two engine variants run on the same prompt then give a first-token KL divergence: how far a prompt-path change moves the model's distribution, not only whether its top token changed. On a layer split the last stage's verifier writes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio
…t; STRATA_EMULATE_CC for tests - generate.cpp: before the expert arena loads, the engine names the card and its compute capability and checks that the build has code for it (device_code_error: cudaFuncGetAttributes on a kernel). A binary built for other GPUs used to fail at the first kernel, after the whole arena had loaded; now it stops in ~0.3 s and says why. - include/strata/core/emulate.hpp: STRATA_EMULATE_CC=75|80|86|89 makes the host's device queries (compute capability, shared memory per block) answer as that generation, for a PTX build of it JIT-compiled on a newer card - ggml's MMQ config, the QSA kernels' choice, fused_gr's shared memory. Tests only. - STRATA_QSA_WARP=1|select|attn: the pre-sm_80 QSA kernels on any card (A/B). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio
The verify window waits for the adaptive tier's copies since Niko1221#463; STRATA_ADAPT_NOWAIT=1 starts it as 0.1.37 did (the non-blocking query), for timing that wait. Default unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…of KeyError: 'args' (Niko1221#549) installed_configs() and update_install() keep only JSON objects with "exe" and "args"; any other strata-*.json in the folder gets a one-line warning naming it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…1#545) It names the largest max_tokens that fits and the existing "fit_max_tokens": true config switch. The check itself is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…mmended VRAM reserve (3072 MiB) With the default 700 MiB reserve the expert cache fills an AMD card that also drives a KDE/Wayland desktop. When the desktop then needs VRAM (the browser --open starts, a new app), amdgpu moves GPU memory to system RAM (GTT), which the experts already fill: the OOM killer ends kwin_wayland / plasmashell (Niko1221#560, R9700), or the compositor fails to pin its framebuffer (Niko1221#516, 7900 XTX). --vram-reserve-mib 3072 / 4000 fixed it in both reports. A recommendation, nothing is changed: setup prints the tip on a Linux desktop session (WAYLAND_DISPLAY or DISPLAY) with the HIP backend and no reserve given, and the server prints it after the start when the engine reports less than 2048 MiB of VRAM free (INFO vram_free_mib) and the reserve is under 3072 - before --open starts the browser. INSTALL.md and AMD_HIP.md describe the symptom and the fix. No engine code. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…elps (Niko1221#530) A reply that ends with finish "length" inside the reasoning has no answer; the server log now says so and names reasoning_budget_tokens (per request, or in the config for every request). Nothing the client receives changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
… and timings use it (Niko1221#548, as Niko1221#536) Dev::put uploads with a pageable cudaMemcpy, which can return before its DMA lands, and the graph replays and the timing cases run on a cudaStreamNonBlocking stream that does not wait for the legacy stream: the graph could read the previous replay's logits (one argmax token off; 5 of 6 replays on an RTX 5060 Ti, now and then on a 5090). Dev::put now synchronizes the device after its copy, which covers run_graph_case and bench(). Test only; no engine code. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…en libcudart is older than the headers A CUDA 13 build that linked the system's CUDA 12 libcudart.so (a kit with a dangling libcudart.so symlink) read cudaDeviceProp shifted: sharedMemPerBlockOptin came back as 1, Niko1221#420's fits() then refused MMQ for every i-quant and the prompts ran ~4x slower, with a log line printing "1 bytes of shared memory per block". - ggml_cuda_info() (CUDA) reads cc, smpbo, smpb, nsm, warp size, integrated and cooperative launch with cudaDeviceGetAttribute, which is ABI-stable. A matched build reads the same values: the same MMQ choices, the same output. - The engine warns at start when the loaded CUDA runtime's major version differs from the headers' it was compiled with, and says to rebuild against one toolkit (-DCUDAToolkit_ROOT). HIP is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
The provider, the context and output limits, the thinking level as a default and as variants, and how they relate to the context check, fit_max_tokens and reasoning_budget_tokens. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
One line under the start banner: the config's engine options without the model's file paths, and the server's fields (host:port, an API key set or not, GPUs, draft subset, fit_max_tokens, thinking budget). The key itself is never printed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
… free The weight arena is the first large VRAM allocation and its size does not depend on the context, so when it fails something else holds the card (another program, an engine still exiting). The error now names the CUDA error and the card's free and total VRAM at that moment; a layer split's stage says the same for its card, with the MiB it needed. Error paths only. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
A web page of another site could point its own name at 127.0.0.1 and reach
the server as the same origin: without an API key it could read /status,
/metrics and /api/requests, change settings and run MCP tools. The server now
answers only Host names it knows: localhost (and *.localhost), any IP address
(rebinding needs a name), the bind address and, bound beyond this PC, the PC's
name, name.local, its LAN addresses and host.docker.internal; any port. No
Host header (HTTP/1.0) passes. "allowed_hosts" in the config and
STRATA_ALLOWED_HOSTS add names (".example.com" for a domain, "*" turns the
check off); trusted_origins' hosts count too. A refusal is a 403 naming the
setting and one log line.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
/unload and /load took a plain form POST from any site: they now take JSON from Strata's own page only, like /settings (_own_page). Without an API key, a POST to /v1/* that carries an Origin must come from Strata's own page, a page on localhost or an allowed host name, an origin in trusted_origins or cors_origins, or a non-web scheme (browser extensions, desktop apps), and then send JSON; others (and Origin: null) get 403, so a web page can no longer burn GPU time with a text/plain POST. Requests without an Origin (curl, the SDKs) are unchanged; with an API key, the key decides. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
It keeps 256 of the 512 experts, chosen on code data, so it is weaker outside code and in other languages; for general use, or when RAM allows, MODELS.md now points to the full models. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
A rebinding page cannot send the key, so with one configured the check only broke tunnels and proxies that pass their own name on (cloudflared does). Keyless setups keep it, with allowed_hosts; the 403 now also says that setting an API key turns the check off. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
… split A splitter may copy general.architecture into EVERY shard of a split while the qwen4exp.* hyperparameters live only in shard 1 (seen with the Huihui abliterated re-split of the UD-Q4_K_XL GGUF: general.architecture in shards 2-4, no qwen4exp.* there). NativeDense::load then ran check_architecture on shard 2 and failed with 'missing qwen4exp.block_count'. Run the full guard only when the shard is the first of the split (no split.no, or split.no == 0); later shards take the existing split-metadata consistency check whether or not they carry the architecture key. Verified end to end: with this change the engine loads the re-split GGUF (previously: refused), pack built with iq_pack.py --compat-bf16, coherent greedy output.
This was referenced Oct 3, 2026
Closed
Niko1221
added a commit
that referenced
this pull request
Oct 6, 2026
The native_dense split-shard arch guard from the same PR is #648's change and is left to it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Niko1221
added a commit
that referenced
this pull request
Oct 6, 2026
The native_dense split-shard arch guard from the same PR is #648's change and is left to it.
Owner
|
Heads-up: today we cleaned up this repository's commit history, and GitHub closed this PR automatically when GitHub cannot reopen it, because its branch is built on the old history. If you want to keep it going: start a new branch from the new Sorry for the noise, and thank you for the contribution. |
anon761
pushed a commit
to anon761/Strata-upstream
that referenced
this pull request
Oct 6, 2026
The native_dense split-shard arch guard from the same PR is Niko1221#648's change and is left to it.
Niko1221
pushed a commit
that referenced
this pull request
Oct 7, 2026
… split A splitter may copy general.architecture into EVERY shard of a split while the qwen4exp.* hyperparameters live only in shard 1 (seen with the Huihui abliterated re-split of the UD-Q4_K_XL GGUF: general.architecture in shards 2-4, no qwen4exp.* there). NativeDense::load then ran check_architecture on shard 2 and failed with 'missing qwen4exp.block_count'. Run the full guard only when the shard is the first of the split (no split.no, or split.no == 0); later shards take the existing split-metadata consistency check whether or not they carry the architecture key. Verified end to end: with this change the engine loads the re-split GGUF (previously: refused), pack built with iq_pack.py --compat-bf16, coherent greedy output. (PR #648)
Owner
|
Thank you, jollyroger1480. The architecture guard on the first shard only is in 0.1.40.2 with your authorship kept. Already closed with the history rewrite. |
sskver
pushed a commit
to sskver/Strata
that referenced
this pull request
Oct 7, 2026
The native_dense split-shard arch guard from the same PR is Niko1221#648's change and is left to it.
0xrjman
pushed a commit
to 0xrjman/Strata
that referenced
this pull request
Oct 7, 2026
… split A splitter may copy general.architecture into EVERY shard of a split while the qwen4exp.* hyperparameters live only in shard 1 (seen with the Huihui abliterated re-split of the UD-Q4_K_XL GGUF: general.architecture in shards 2-4, no qwen4exp.* there). NativeDense::load then ran check_architecture on shard 2 and failed with 'missing qwen4exp.block_count'. Run the full guard only when the shard is the first of the split (no split.no, or split.no == 0); later shards take the existing split-metadata consistency check whether or not they carry the architecture key. Verified end to end: with this change the engine loads the re-split GGUF (previously: refused), pack built with iq_pack.py --compat-bf16, coherent greedy output. (PR Niko1221#648)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
NativeDense::loadrancheck_architectureon every shard that carriesgeneral.architecture. A splitter may copy that key into all shards of a split while theqwen4exp.*hyperparameters live only in shard 1 —NativeDense::loadthen fails on shard 2 with:Seen with a re-split of the UD-Q4_K_XL GGUF (the Huihui abliterated release of Qwen3.8-Flash-Next:
general.architecturepresent in shards 2-4, noqwen4exp.*there).Change
Run the full architecture guard only when the shard is the first of the split (
split.noabsent or0); later shards take the existing split-metadata consistency check whether or not they carry the architecture key. Single-file GGUFs are unaffected (they always havesplit.no == 0or none).Verification
iq_pack.py --compat-bf16, resident mode, coherent greedy output from the 125B.strata-gguf <shard1> --checkstill passes (the guard on shard 1 is unchanged).Tested on the HIP backend (gfx1030 / RX 6950 XT, ROCm 7.1).