Repository navigation
Conversation
…060 XT, Niko1221#256 by Efeisot) as community-validated cards Cut down from Niko1221#254 and Niko1221#256 onto 0.1.30's AMD tables: - setup.py: AMD_ARCHS / AMD_NAMES / ROCM_INDEXES / AMD_CARDS list gfx1101 (TheRock gfx110X-dgpu) and gfx1200 (gfx120X-all); amd_gpus() takes the sysfs root so the detection can be tested. - cmake/hip_backend.cmake: gfx1101 and gfx1200 build without the "not validated" warning (a status line names them community-validated); gfx1102 (Niko1221#192: ctest only) keeps the warning. - docs/AMD_HIP.md (Community-validated cards: the owners' setups, ctest and numbers) and README. - tools/test_setup_amd.py: detection on a mocked KFD topology (every family, the CPU node, an iGPU and gfx1102 listed as not supported, product-name fallbacks, the ROCm index per family). Left out: the PRs' own cmake/device.cu arch checks (0.1.30's compiled_for covers them), the dp4a change (main has it), math_constants.h, the bench JSONs, the launcher, and both hipBLASLt tables (neither PR measured the prompt speed without its table). Co-Authored-By: Joseph Hohertz <jhohertz@gmail.com> Co-Authored-By: efeisot <efeproofkmr@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…D arch; the engine compiled for each)
setup refused AMD GPU lists at install and at start ("NVIDIA only for now"), although the engine's layer split runs
on two AMD cards (0.1.30, from a hand-written config).
- amd_parse_gpus(): --gpus "1,0" (setup's HIP numbering, the first card the main one) or "all" (every supported
card, the most VRAM first); every card must be in AMD_ARCHS, else a plain message with what can be used.
- install: --backend hip --gpus writes "gpu": [..] + "layer_split" like the NVIDIA path; build_engine_hip compiles
for the set of the chosen cards' archs (CMAKE_HIP_ARCHITECTURES "gfx1100;gfx1201" already worked) and records
it in BUILD.json; an engine whose archs cover the cards is not rebuilt; update_installed_engine keeps every arch.
- rocm_root(archs): cards of two families need a system ROCm (TheRock's wheels hold one family) - said plainly.
- start: --gpus on an AMD install is saved like on NVIDIA (refused when the engine has no code for a card);
--gpus all finds the AMD cards on a PC without NVIDIA cards that can share.
- docs/AMD_HIP.md, README; tools/test_setup_amd.py: lists, "all", refusals, the one-family wheel rule, the arch set
in the build and BUILD.json.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…sliced pin is opt-in (STRATA_ARENA_PIN_GIB=auto) - by default the arena pins as in 0.1.30; on this 64 GB PC the cap pinned 26 instead of 30 GiB and cost ~20% prompt speed and 2-4% IQ3_S decode Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…file reads, a partial pin of the RAM tier, bit-exact AVX2 multi-token kernels for Q4_K / Q5_1 / Q8_0 experts, decode-only tier stats Where the 288 ms/token at a 40 GiB budget went (UD-Q4_K_XL, --stats): of the 945 ms CPU pool call per verify round (3.46 tokens), 871 ms were the dispatch thread assembling missed experts from the mapped GGUF one after another (~391 blobs, 1.3 GB per round, ~1.5 GB/s: page faults from the SSD); the kernels were 42 ms gate/up and 30 ms down. At 24 GiB, 771 ms/round more went into the PCIe planning, which asked blob() - a file read with the GGUF in place - of every miss just to learn it was not pinned. - ExpertSource::prefetch: the dispatch hands the source the layer's missed experts before its jobs loop; FileExpertSource with the GGUF in place assembles them on STRATA_FETCH_THREADS threads (default 8) so the SSD sees a queue instead of one fault at a time. A no-op for every other source. Staging buffers being filled are never reused, and a blob asked while its fill runs waits for it. - The PCIe planning asks pinned() before blob(): the same decisions and bytes for every source. - --resident-budget-gib: when the driver refuses to page-lock the whole budget, the locked arena's hottest prefix (profile order) is registered with cudaHostRegister, as much as it takes (from the whole down in 2 GiB steps), so the GPU can read those misses over PCIe and the prompt path can DMA them; pinned() answers for that prefix only. STRATA_PARTIAL_PIN=0 is the A/B arm. - src/kernels/cpu/kq_avx2.cpp: ggml-cpu's AVX2 vec_dot for Q4_K x Q8_K, Q5_1 x Q8_1 and Q8_0 x Q8_0 with the token loop inside the block loop - the weight side decoded once per verify window, each token's integer chain and float operations exactly ggml's, so the results are ggml's to the bit for any group size (no Niko1221#152 rule). native_gu_rows / native_down_rows use them for those types (only UD-Q4_K_XL has them); STRATA_NO_KQ256 is the A/B arm. native_expert_parity checks them bit for bit against vec_dot for 1..8 tokens and times 4 tokens. - Stats: the decode's file tier alone (blobs, MB, MB/round, reading ms/round and GB/s per thread) and the prompt path's copies separately. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
# Conflicts: # serve/server.py # src/core/expert_source.cpp
…_K/Q5_1/Q8_0 multi-token kernels from two tokens on - Registering 30 GiB of a 40 GiB RAM tier left the driver unable to page-lock the prompt path's token id buffer afterwards (RTX 5070, WDDM: "prefill: the token id buffer"); the prefix is now capped at 24 GiB, the size a whole 24 GiB budget page-locks with, and STRATA_PARTIAL_PIN_GIB sets it. - A verify window's expert groups hold ~1.4 tokens on average; one token goes to ggml's own dot (the same bits, without the multi-token kernel's per-token arrays). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…ssed slices at once (PrefetchVirtualMemory) The copies' page faults read a few clusters each; one PrefetchVirtualMemory call for every role slice about to be copied lets the memory manager queue large reads at once, and the parallel copies then find the pages resident or in flight. Only the GGUF-in-place source's prefetch uses it; STRATA_FETCH_PVM=0 is the A/B arm. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…IN_GIB; the split's prompt chunk steps down - The expert arena was pinned at most 8 GiB whenever a layer split (or remote experts) ran, on every OS - a Windows/WDDM workaround that cost a Linux 4090 + 3060 split two thirds of its prompt speed (Niko1221#253, bettercallcaleb). The cap now applies only under WDDM (_WIN32, or WSL2 detected by /dev/dxg), for multi_gpu and expert_cache_remote alike. STRATA_ARENA_PIN_GIB overrides it on any OS: "0" pins the whole arena, N > 0 at most N GiB (arena_pin_cap_gib(), -1 when unset: the same name and meaning as the S2 branch's helper, file-local here). - The split's prompt chunk no longer stops the start when its buffers do not fit: a prompt path without a loan first gets the largest chunk whose buffers (Prefill::bytes_needed) leave 512 MiB free on its device, and an init that still fails steps down one size (8192 -> 6144 -> ... -> 2048 -> 1536 -> 1024 -> 512): Prefill::reset() frees what the failed init made, the loans are re-priced, and the failed allocation's sticky error is consumed (else the next launch check reported "out of memory"). - pinned.cu: the capped note no longer says "for CUDA1". - docs: MULTI_GPU.md, SECOND_GPU.md. Measured on amdbox (Linux, R9700 + RX 9070 XT, gpu [1, 0], Coder IQ1_M, 128 greedy tokens; tok/s): | cache | arm | 4K cold | 4K warm | 16K cold | 16K warm | decode | | auto (holds ~all) | 8 GiB cap | 678 | 1,355 | 1,709 | 1,880 | 44.5-51.5 | | | whole arena | 703 | 1,234 | 1,729 | 1,864 | 44.6-51.4 | | fixed 1500 on CUDA0 | 8 GiB cap | 608 | 1,101 | 1,287 | | 34.5-36.1 | | | whole arena | 716 | 1,160 | 1,803 | | 34.8-36.0 | (2 starts per arm for auto, 5 for fixed.) With most experts streamed the whole arena reads a 16K prompt 40% faster; with every expert already on the cards the two are within noise except 4K warm (-9%: the pinned share raises the auto lend to 90% and the streamed ring to 384 slots). Identity: the fixed-cache arms gave the same tokens in all 5 + 5 starts (4K, 16K, 4K again), and the auto arms 2 + 2. The step-down was exercised on the 9070 XT with --expert-cache 3000 --no-prefill-borrow --prefill 8192 ("need 4528 MiB, 2910 MiB free: 2048-token chunks"; with the init retry alone it went 8192 -> ... -> 2048, and 1536 ran all four requests). Known, also on main: that own-buffer configuration at 2048 makes hipBLAS pick hipBLASLt kernels whose code objects this ROCm 7.14 lacks (TensileLibrary_..._gfx1201-xnack+.co not found, "launchKernel failed") and the prompt hangs. The WDDM side (Windows, WSL2) is unchanged and not re-measured here. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…one when it holds every profiled expert + KV
Before any later stage loads, split-auto checks whether CUDA0 alone holds every profiled expert pair plus the whole
session (KV of every layer), the drafter + head, the verify windows and the reserve; if so it clears the split (one
GPU) and logs why, else it logs how much of the profile CUDA0 would hold and keeps the split. Default behaviour is
unchanged: the engine flag is opt-in, and the server passes it only for a config with "split_skip_if_fits": true
(and a GPU list). Not with remote expert caches.
Measured on amdbox (Coder IQ1_M, --expert-cache auto --prefill auto, 128 greedy tokens, gpu [1, 0] = R9700 first,
2 starts per arm, warm = the second request of each size):
| arm | 4K warm | 16K warm | decode |
| split (flag off) | 1,244 | 1,897 | 50.5-53.0 |
| --split-skip-if-fits (on) | 1,776 | 1,791 | 59.7-61.8 |
| R9700 alone | 1,787 | 1,801 | 59.9-61.3 |
The flag's run is the R9700-alone run (same tokens, all four requests); with the 9070 XT first it keeps the split
("would hold only 9.84 of the profile's 23.42 GiB"). On this pair the split still reads 16K prompts ~5% faster.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
# Conflicts: # src/core/pinned.cu
…aster on an AVX2-only CPU (Ryzen 9 3900X: gate/up -2.6..-4.7% at 1 token, down -3..-5% at 2 tokens) This reverts commit 9e9dce4. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…t an expert boundary; the Q4_K/Q5_1/Q8_0 multi-token kernels opt-in (STRATA_KQ256=1) - Locking the whole 40 GiB arena in the working set and then registering its prefix made the next device allocation fail (the prompt path's token id buffer); the registered prefix is page-locked by the driver already, so only the rest is locked now. 24 GiB registered out of a 40 GiB budget then starts and runs. - The registered prefix ends at an expert boundary: a blob that began inside it and ran past it was taken as page-locked by cudaMemcpyAsync and refused (an adaptive refill failed, with 16 GiB registered). - The prompt path's token id buffer error names the CUDA error. - kq_avx2 multi-token kernels: bit-exact, but no faster in the engine (the rows phases stayed at 72 ms per verify round at a 40 GiB budget; a window's expert groups hold ~1.4 tokens and ggml's per-token calls find the weights in L1), so they are opt-in with STRATA_KQ256=1; native_expert_parity still checks them bit for bit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…m-split down kernel (164 blocks), rs/lo/inject in up's prologue (STRATA_GR_V3=0 = old)
…); its shared memory is sized from the card's opt-in limit - one or two column halves, else the default kernels (Turing/HIP 64 KB fit with two; capability-based split as proposed by @Adamyno); the neutral knobs (DOWN_R2, UP_PF, KSPLIT) dropped (review)
…ure was generated, as its generator does; a partly missing set still fails; the open handle of a half-present fixture is closed Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…er on this layer's MoE input warms its predicted experts' pages - RouterLookahead: when the CPU pool starts layer l, a worker thread applies layer l+1's router (a BF16 host copy of ffn_gate_inp, 126 MB for 48 layers) to layer l's MoE input - the router's own input - for each token of the window, takes the top STRATA_LOOKAHEAD_K (default 10), drops the experts the GPU cache (host_res) or the RAM copy holds, and asks the source to warm the rest. It never waits: a prediction still running makes the next one skip. STRATA_LOOKAHEAD=0 is the A/B arm. - ExpertSource::warm / warms(): FileExpertSource with the GGUF in place issues one PrefetchVirtualMemory for the predicted slices (madvise(WILLNEED) elsewhere). Pages only: blob(), the experts computed and their bytes do not change (the same answers in every A/B run). Every other source: no-op, and the predictor is not started. - bf16_rows_dot_multi (kq_avx2.cpp, AVX2): the router read once for all tokens of the window. - Stats: experts warmed, how many of the decode's file-tier reads had been warmed, the predictor's time and skips. UD-Q4_K_XL at a 40 GiB budget (single-token-router version): ~48-54% of the file tier's reads had been warmed; reading time 850-968 -> 630-730 ms per verify round; decode 5.31/6.60 tok/s off vs 7.08/6.79/6.91 on. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
Add preliminary native v0.1.26 results, raw run JSON, scrubbed guest hardware facts, and the local API harness. Prompt and memory topology limitations are documented.
On the RTX 5070 / 64 GB PC the GPU computing a share of the misses over PCIe measured no faster than the CPU: 7.30 and 7.44 tok/s with 24 and 16 GiB of a 40 GiB budget registered, against 7.05-7.74 unpinned, and registering costs startup time and driver memory. The default budget mode is now locked-only; the routing-aware prefetch (multi-run A/B: 6.93 -> 7.91 tok/s mean, identical answers) stays on with the GGUF in place. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
… the mapped mode without experts.bin, --resident-budget-gib, the tier counters - docs/UNSLOTH_Q4.md: what UD-Q4_K_XL is (Hugging Face revision 38bb39e, the four shards with sizes and SHA-256), disk/RAM/GPU needs (111.3 GB download, 1.4 GB pack, no experts.bin), the iq_pack.py command and why --compat-bf16, the numerical policy (native experts/embedding/head/projections/PLE; the converted small tensors listed in conversions.json), the MTP draft layer, a server config (--native shard 1, --ple-gguf found by name, --resident-budget-gib N with N = RAM minus 20-24 GB), measured speed on an RTX 5070 / 64 GB PC (7-8.5 tok/s at 40 GiB; ~60 s to the first answer), the switches (STRATA_LOOKAHEAD=0, STRATA_KQ256=1, STRATA_PARTIAL_PIN=1, STRATA_FETCH_THREADS), scope and validation, credits (eddoursul Niko1221#245, gopinath87607 Niko1221#255, jagsan-cyber Niko1221#247). Experimental, not in setup. - docs/DETAILS.md: the mapped low-RAM mode reads a native pack's experts from the GGUF when it has no experts.bin (checked, identical answers on the Coder); --resident-budget-gib and the routing-aware prefetch; where the tier counters are (--stats, the server log, /metrics' ram_blobs / file_blobs / file_mb); a pointer from "Which model?". Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…plit GGUF, Q4_K/Q5_K/Q5_1/Q8_0 experts, RAM budget + SSD tier with routing prefetch), the low-RAM mode reads GGUF experts in place, Windows GGUF load 2x (Niko1221#230), server status race (Niko1221#266) and tokenizer (Niko1221#268) fixes, truncated tool calls (Niko1221#211/Niko1221#231), AMD intrinsics (Niko1221#262); setup requires it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…ATA_HIP_WMMA=1, gfx12 only) The v2 tensor-core design with gfx12's v_wmma_f32_16x16x16_f16: one block per (query, KV head), wave w owns dims [64w, 64w+64) (int8 scale group w), q and p split into FP16 hi + lo parts, int8 codes exact as FP16, scales in FP32, the four groups' q.k partials added in a fixed order, online softmax over 32-cell chunks. Masked (non-resident) cells get no weight, as in the decode kernel. FP32-level accuracy, not bitwise equal to the default kernel, hence opt-in. hip_prompt_attn_wmma = qsa_prompt_attn_parity built for HIP (skips off gfx12). NVIDIA builds unchanged (all new code is under __HIPCC__ / the HIP test section). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…ame (default on HIP; STRATA_HIP_ROUTER_OLD=1: old) On RDNA4 the router took 39 us per call in decode (16% of the decode GPU time): 512 dependent FP64 adds, ten selection passes with 20 block barriers, a renormalisation re-reading its weights from global memory. The sum's only use is inv = (float)(1.0 / sum): all terms are positive with the largest exactly 1, so the serial sum and a tree sum are within 2^-43 of each other; when the bracket tree*(1 -+ 2^-40) maps to one float, that float is the serial sum's (1/x and the cast are monotone), otherwise thread 0 runs the serial sum as before. The selection runs on wave 0 from registers (the argmax with the lowest index on ties is unique), the renormalisation from registers in rank order. hip_router_fast checks ids and weights bitwise against the portable kernel (65,536 rows with ties, near-ties, NaN rows, the forced serial path; and 256 experts / k 8). CUDA builds unchanged (HIP-only code). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
… unterminated Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…n's logits STRATA_DUMP_FIRST_LOGITS=<file>: the logits of the first window's first position (the token the prompt predicts), n_vocab float32, written once per run. Two engine variants run on the same prompt then give a first-token KL divergence: how far a prompt-path change moves the model's distribution, not only whether its top token changed. On a layer split the last stage's verifier writes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio
…t; STRATA_EMULATE_CC for tests - generate.cpp: before the expert arena loads, the engine names the card and its compute capability and checks that the build has code for it (device_code_error: cudaFuncGetAttributes on a kernel). A binary built for other GPUs used to fail at the first kernel, after the whole arena had loaded; now it stops in ~0.3 s and says why. - include/strata/core/emulate.hpp: STRATA_EMULATE_CC=75|80|86|89 makes the host's device queries (compute capability, shared memory per block) answer as that generation, for a PTX build of it JIT-compiled on a newer card - ggml's MMQ config, the QSA kernels' choice, fused_gr's shared memory. Tests only. - STRATA_QSA_WARP=1|select|attn: the pre-sm_80 QSA kernels on any card (A/B). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio
…iko1221#496) On an RTX 3060 Laptop (6 GB) the weights, the KV cache and the draft layer left 0.29 GiB, the default 700 MiB reserve and the draft head took it, the expert cache got 0 slots and the verify window refused to start. - setup: a single card under 8 GB (nvidia-smi below 7.5 GB; an 8 GB card lists 7.99) gets --vram-reserve-mib 300 unless one is given (images on the GPU keep theirs), marked in the config as setup's own choice so a later setup decides again. It prints what frees more VRAM if the start still stops: an 8K context, --draft-vocab en, other programs closed. The draft layer stays (the server needs --mtp and --spec). 8 GB and larger cards: the same config (golden test). - engine: a cache too small to lend the prompt path a 256-token chunk (its buffers plus the 128-slot floor; one slot without --prefill) makes the prompt path allocate its own on top. When the default reserve leaves less than that, the reserve shrinks to what leaves exactly that cache, down to 300 MiB (what comes after the cache is paid from the reserve: at 300 an emulated small card ended with 5 MiB free, LOW by the serve check). With no slot at all the start stops with how many MiB are short, what the session and the draft head take, and the options. A start whose cache could lend that chunk is sized as before. Checked with a VRAM ballast on the 5070 (IQ3_XXS, 32K): 258 slots needed, 0.71 GiB free -> "about 333 MiB more", and with --vram-reserve-mib 700 given, "733 MiB more" plus the reserve tip. - verify's refusal points at that line. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
peer_experts.cpp calls cudaDeviceCanAccessPeer, cudaDeviceEnablePeerAccess and cudaMemcpyPeerAsync and tests cudaErrorPeerAccessAlreadyEnabled; the HIP shim had none of them, so the Windows HIP build stopped there. Mapped to their hip* names. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…ives the tip Setup no longer writes a fixed 300 MiB reserve for cards under 8 GB: the engine's own rule (lower the reserve only as far as the smallest working cache needs, down to 300) chooses better, and a fixed 300 left an emulated small card with 5 MiB free. Setup keeps the tip (an 8K context, --draft-vocab en). When the engine lowered the reserve and the card ends below the 256 MiB LOW line, it prints one warning (requests may stall; close other programs that use the GPU, or lower --max-context) instead of advising a larger reserve, which would leave the cache no room. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…iko1221#452), Q5_0 experts on the GPU (Niko1221#473), IQ4_XS on AVX-2 (Niko1221#415), unbuffered loads on Windows (Niko1221#357 Niko1221#362), --peer-device (Niko1221#531), a 6 GB card starts (Niko1221#496), PR batch; setup requires it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…l's expert count (Niko1221#531 follow-up) Niko1221#531's peer-aware lookup indexed the residency table with the 512-expert constant NE instead of the model's own count. The Coder has 256 experts per layer, so from its second layer a resident expert was looked up in the wrong row, and a prompt long enough for the streamed walk stopped with "an expert is neither resident, streamed nor on the peer" (found by the 57K real-use check). The 512-expert models index the same row as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…kernels cover EVERY layer fused_layout() (STRATA_PF_FUSED=1) shrinks GU / H / Xq to the fused kernels' needs whenever fused_ring() is true, and fused_ring() was true when ANY layer of a native pack was covered. A pack with an uncovered layer - Unsloth UD-IQ4_XS: 5 layers with Q8_0 down projections - then ran MMQ for those layers at the full chunk in buffers sized for stream_all_min() - 1 tokens: on gfx1151 an 8K prompt answered "!!!!..." and a 64K prompt hung (--prefill 2048: an illegal memory access). Such a pack now keeps MMQ's buffers; its covered layers still take the fused kernels. UD-IQ4_XS 8K retrieval after the fix: 3/3 correct, 837.9 prompt tok/s. (The CUDA path shares this code.) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
The verify window waits for the adaptive tier's copies since Niko1221#463; STRATA_ADAPT_NOWAIT=1 starts it as 0.1.37 did (the non-blocking query), for timing that wait. Default unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…of KeyError: 'args' (Niko1221#549) installed_configs() and update_install() keep only JSON objects with "exe" and "args"; any other strata-*.json in the folder gets a one-line warning naming it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…1#545) It names the largest max_tokens that fits and the existing "fit_max_tokens": true config switch. The check itself is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…mmended VRAM reserve (3072 MiB) With the default 700 MiB reserve the expert cache fills an AMD card that also drives a KDE/Wayland desktop. When the desktop then needs VRAM (the browser --open starts, a new app), amdgpu moves GPU memory to system RAM (GTT), which the experts already fill: the OOM killer ends kwin_wayland / plasmashell (Niko1221#560, R9700), or the compositor fails to pin its framebuffer (Niko1221#516, 7900 XTX). --vram-reserve-mib 3072 / 4000 fixed it in both reports. A recommendation, nothing is changed: setup prints the tip on a Linux desktop session (WAYLAND_DISPLAY or DISPLAY) with the HIP backend and no reserve given, and the server prints it after the start when the engine reports less than 2048 MiB of VRAM free (INFO vram_free_mib) and the reserve is under 3072 - before --open starts the browser. INSTALL.md and AMD_HIP.md describe the symptom and the fix. No engine code. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…elps (Niko1221#530) A reply that ends with finish "length" inside the reasoning has no answer; the server log now says so and names reasoning_budget_tokens (per request, or in the config for every request). Nothing the client receives changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
… and timings use it (Niko1221#548, as Niko1221#536) Dev::put uploads with a pageable cudaMemcpy, which can return before its DMA lands, and the graph replays and the timing cases run on a cudaStreamNonBlocking stream that does not wait for the legacy stream: the graph could read the previous replay's logits (one argmax token off; 5 of 6 replays on an RTX 5060 Ti, now and then on a 5090). Dev::put now synchronizes the device after its copy, which covers run_graph_case and bench(). Test only; no engine code. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…en libcudart is older than the headers A CUDA 13 build that linked the system's CUDA 12 libcudart.so (a kit with a dangling libcudart.so symlink) read cudaDeviceProp shifted: sharedMemPerBlockOptin came back as 1, Niko1221#420's fits() then refused MMQ for every i-quant and the prompts ran ~4x slower, with a log line printing "1 bytes of shared memory per block". - ggml_cuda_info() (CUDA) reads cc, smpbo, smpb, nsm, warp size, integrated and cooperative launch with cudaDeviceGetAttribute, which is ABI-stable. A matched build reads the same values: the same MMQ choices, the same output. - The engine warns at start when the loaded CUDA runtime's major version differs from the headers' it was compiled with, and says to rebuild against one toolkit (-DCUDAToolkit_ROOT). HIP is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
The provider, the context and output limits, the thinking level as a default and as variants, and how they relate to the context check, fit_max_tokens and reasoning_budget_tokens. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
One line under the start banner: the config's engine options without the model's file paths, and the server's fields (host:port, an API key set or not, GPUs, draft subset, fit_max_tokens, thinking budget). The key itself is never printed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
… free The weight arena is the first large VRAM allocation and its size does not depend on the context, so when it fails something else holds the card (another program, an engine still exiting). The error now names the CUDA error and the card's free and total VRAM at that moment; a layer split's stage says the same for its card, with the MiB it needed. Error paths only. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
A web page of another site could point its own name at 127.0.0.1 and reach
the server as the same origin: without an API key it could read /status,
/metrics and /api/requests, change settings and run MCP tools. The server now
answers only Host names it knows: localhost (and *.localhost), any IP address
(rebinding needs a name), the bind address and, bound beyond this PC, the PC's
name, name.local, its LAN addresses and host.docker.internal; any port. No
Host header (HTTP/1.0) passes. "allowed_hosts" in the config and
STRATA_ALLOWED_HOSTS add names (".example.com" for a domain, "*" turns the
check off); trusted_origins' hosts count too. A refusal is a 403 naming the
setting and one log line.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
/unload and /load took a plain form POST from any site: they now take JSON from Strata's own page only, like /settings (_own_page). Without an API key, a POST to /v1/* that carries an Origin must come from Strata's own page, a page on localhost or an allowed host name, an origin in trusted_origins or cors_origins, or a non-web scheme (browser extensions, desktop apps), and then send JSON; others (and Origin: null) get 403, so a web page can no longer burn GPU time with a text/plain POST. Requests without an Origin (curl, the SDKs) are unchanged; with an API key, the key decides. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
It keeps 256 of the 512 experts, chosen on code data, so it is weaker outside code and in other languages; for general use, or when RAM allows, MODELS.md now points to the full models. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
A rebinding page cannot send the key, so with one configured the check only broke tunnels and proxies that pass their own name on (cloudflared does). Keyless setups keep it, with allowed_hosts; the 403 now also says that setting an API key turns the check off. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
Keeps the launcher small and Strata-driven; no dependencies, no dashboard, no extra service. - READY + Enter no longer starts directly: it opens a small load-options page (Context / Vision / Load / Back) with one shortcut line; only Load starts - Context chooser: 32K / 64K / 128K / 262K / Custom; the current value is read from the actual config and preselected; Custom takes an integer and is passed through unchanged (Strata validates, nothing is clamped) - Vision chooser: Off / Auto / GPU / CPU, mapped to --vision no/yes/gpu/cpu; the config's vision state is detected and preselected - Load: unchanged settings start the existing config directly; changed settings first run the CURRENT setup.py (--context/--vision, pack markers are kept, so nothing is rebuilt) and only then start the result; enabling vision lets Strata download/prepare the encoder in the log page - operation log keeps the one fixed status bar (LOADING ... spinner -> RUNNING ... READY, PREPARING / STOPPING, ERROR) and now also carries the context / vision detail on the LOADING / PREPARING rows - when an operation is alive but silent, only the single status line is repainted (no whole-terminal redraw, no duplicate lines, no flash) - Esc and B both return everywhere (fixed read-key normalization on Windows) - V4 GGUF verification, gguf_reader, dynamic setup.py support and the READY / PREPARE / BROKEN repair logic are unchanged Verified: py -3 -m py_compile, plus the synthetic state suite (89/89 checks incl. load options, context/vision planning, phase handover and a spinner-only redraw).
Ayshinko
force-pushed
the
feature/minimal-strata-tui
branch
from
October 3, 2026 21:58
9c66246 to
1ba813d
Compare
Owner
|
Heads-up: today we cleaned up this repository's commit history, and GitHub closed this PR automatically when GitHub cannot reopen it, because its branch is built on the old history. If you want to keep it going: start a new branch from the new Sorry for the noise, and thank you for the contribution. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A small interface, not another layer
This adds an optional terminal launcher for Strata with one deliberately narrow goal: make the existing setup / prepare / start workflow easier to use without building another management stack around it.
It is intentionally small.
There is no second server, no reverse proxy, no daemon, no frontend framework, no background dashboard, and no additional Python dependency.
The launcher is essentially a thin terminal view over functionality Strata already owns.
What it does
From one compact terminal screen a user can:
Prepare, start and stop automatically switch to an operation-log view, so there is no constantly repainting dashboard in the background.
Press Esc/B and you are back at the model list.
Strata remains the source of truth
A key design constraint was to avoid creating a second compatibility database.
The launcher reuses the current Strata checkout for model support and GGUF handling instead of trying to teach a UI its own version of Strata.
Where available it consumes the existing definitions/helpers from
setup.pyand uses Strata's owntools/gguf_reader.py.That means model discovery can validate the GGUF metadata and tensor directory — including architecture, split/shard information and routed expert tensors — without reading tens of gigabytes of tensor payload just to identify a model.
If Strata changes its supported families or formats, the launcher is designed to follow Strata rather than maintain a parallel handwritten whitelist.
The launcher also follows whether the current checkout still has Strata's custom
--variantbuilds: on a checkout without them, variant labels and flags are simply not offered.READY means actually ready
An installed config is shown as READY only when the artifacts the current Strata actually needs exist (the engine binary, the prepared pack's completion markers and tokenizer, the tokenizer directory, every GGUF shard the config points at, and the
--mtpdraft runtime). Only file existence is checked — never the multi-GB payloads.If a prepared pack was deleted, the config is never shown as READY: the raw GGUF is offered again as PREPARE and Enter re-runs the current
setup.py; if the source GGUF is gone too, the config is shown as BROKEN with the reason.What it deliberately does not do
This PR does not introduce:
If the launcher is never opened, normal Strata operation is unchanged.
The idea is closer to a small installer/menu utility than an application layered on top of Strata.
Why
Strata already has the difficult parts: model validation, preparation, runtime configuration, the server, Chat and Monitor.
This PR adds only the small missing interface between those pieces and a user sitting at a terminal.
The path is deliberately short:
model -> prepare -> start -> logs
I previously experimented with a significantly larger manager UI.
That experiment made the trade-off clear: for this job, less is better.
This version takes the opposite approach:
small surface area, existing Strata logic, zero extra runtime architecture.
Nothing in this PR is required for existing users. It is simply an optional convenience for users who prefer a compact terminal workflow.
Files
tools/strata_tui.py— the launcher (no third-party dependencies; run withpy -3 tools\strata_tui.py)START-TUI.bat/start-tui.sh— tiny Windows/Linux launchers (same style asSETUP.bat/setup.sh)