serve: commit only the tokens a verify window hands out by anon761 · Pull Request #652 · Niko1221/Strata

anon761 · 2026-10-03T15:36:20Z

What

A verify window committed every accepted position (a + 1) into the session, but hands out only the outputs below max_tokens and through the first end of turn. When an answer ended inside a window - a max_tokens cap, or an end of turn followed by accepted drafts - the session held drafted tokens the client never saw. The follow-up that sends the answer back then no longer starts with the live session, and the engine falls back to the turn-boundary checkpoint: the whole answer and the new message are read again.

The window now counts the outputs it hands out first and commits exactly their positions - the same state as an answer that ends on a window's last token. consumed (the live session's ids) follows the same count. Decode itself is unchanged: the drafts and the next window are as before, only the final window's commit can be shorter.

Measured

2x RTX 3090, Unsloth UD-Q4_K_XL, layer split auto, 262K context, --spec 4 --spec-min-p 0.5. A coding conversation of 8 turns over the OpenAI endpoint (greedy); every turn sends the whole history back with the answers verbatim, max_tokens alternating between short caps and room for a natural end. "Resumed" = the engine reused at least what it held after the previous turn (prompt + answer less the uncommitted last token), read from /metrics.

	follow-ups resumed	the misses' prompt time
`main`	5 of 7	530-770 ms instead of ~220 ms
this branch	7 of 7	-
this branch, same conversation after an ~8K-token code block	7 of 7	-

Both misses on main were answers cut by max_tokens inside a window (reused 30 (held 97), reused 753 (held 849)). Follow-up turns resume in 200-230 ms at 1K and at 9K tokens of context.

One kind of miss remains and is not the engine's: Qwen's chat template strips trailing whitespace from an assistant message, so an answer cut right after a \n comes back without it, and the session (which holds that \n) cannot rewind to the shorter prompt. That happened once in ~40 sampled follow-ups.

…060 XT, Niko1221#256 by Efeisot) as community-validated cards Cut down from Niko1221#254 and Niko1221#256 onto 0.1.30's AMD tables: - setup.py: AMD_ARCHS / AMD_NAMES / ROCM_INDEXES / AMD_CARDS list gfx1101 (TheRock gfx110X-dgpu) and gfx1200 (gfx120X-all); amd_gpus() takes the sysfs root so the detection can be tested. - cmake/hip_backend.cmake: gfx1101 and gfx1200 build without the "not validated" warning (a status line names them community-validated); gfx1102 (Niko1221#192: ctest only) keeps the warning. - docs/AMD_HIP.md (Community-validated cards: the owners' setups, ctest and numbers) and README. - tools/test_setup_amd.py: detection on a mocked KFD topology (every family, the CPU node, an iGPU and gfx1102 listed as not supported, product-name fallbacks, the ROCm index per family). Left out: the PRs' own cmake/device.cu arch checks (0.1.30's compiled_for covers them), the dp4a change (main has it), math_constants.h, the bench JSONs, the launcher, and both hipBLASLt tables (neither PR measured the prompt speed without its table). Co-Authored-By: Joseph Hohertz <jhohertz@gmail.com> Co-Authored-By: efeisot <efeproofkmr@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…D arch; the engine compiled for each) setup refused AMD GPU lists at install and at start ("NVIDIA only for now"), although the engine's layer split runs on two AMD cards (0.1.30, from a hand-written config). - amd_parse_gpus(): --gpus "1,0" (setup's HIP numbering, the first card the main one) or "all" (every supported card, the most VRAM first); every card must be in AMD_ARCHS, else a plain message with what can be used. - install: --backend hip --gpus writes "gpu": [..] + "layer_split" like the NVIDIA path; build_engine_hip compiles for the set of the chosen cards' archs (CMAKE_HIP_ARCHITECTURES "gfx1100;gfx1201" already worked) and records it in BUILD.json; an engine whose archs cover the cards is not rebuilt; update_installed_engine keeps every arch. - rocm_root(archs): cards of two families need a system ROCm (TheRock's wheels hold one family) - said plainly. - start: --gpus on an AMD install is saved like on NVIDIA (refused when the engine has no code for a card); --gpus all finds the AMD cards on a PC without NVIDIA cards that can share. - docs/AMD_HIP.md, README; tools/test_setup_amd.py: lists, "all", refusals, the one-family wheel rule, the arch set in the build and BUILD.json. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…sliced pin is opt-in (STRATA_ARENA_PIN_GIB=auto) - by default the arena pins as in 0.1.30; on this 64 GB PC the cap pinned 26 instead of 30 GiB and cost ~20% prompt speed and 2-4% IQ3_S decode Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…file reads, a partial pin of the RAM tier, bit-exact AVX2 multi-token kernels for Q4_K / Q5_1 / Q8_0 experts, decode-only tier stats Where the 288 ms/token at a 40 GiB budget went (UD-Q4_K_XL, --stats): of the 945 ms CPU pool call per verify round (3.46 tokens), 871 ms were the dispatch thread assembling missed experts from the mapped GGUF one after another (~391 blobs, 1.3 GB per round, ~1.5 GB/s: page faults from the SSD); the kernels were 42 ms gate/up and 30 ms down. At 24 GiB, 771 ms/round more went into the PCIe planning, which asked blob() - a file read with the GGUF in place - of every miss just to learn it was not pinned. - ExpertSource::prefetch: the dispatch hands the source the layer's missed experts before its jobs loop; FileExpertSource with the GGUF in place assembles them on STRATA_FETCH_THREADS threads (default 8) so the SSD sees a queue instead of one fault at a time. A no-op for every other source. Staging buffers being filled are never reused, and a blob asked while its fill runs waits for it. - The PCIe planning asks pinned() before blob(): the same decisions and bytes for every source. - --resident-budget-gib: when the driver refuses to page-lock the whole budget, the locked arena's hottest prefix (profile order) is registered with cudaHostRegister, as much as it takes (from the whole down in 2 GiB steps), so the GPU can read those misses over PCIe and the prompt path can DMA them; pinned() answers for that prefix only. STRATA_PARTIAL_PIN=0 is the A/B arm. - src/kernels/cpu/kq_avx2.cpp: ggml-cpu's AVX2 vec_dot for Q4_K x Q8_K, Q5_1 x Q8_1 and Q8_0 x Q8_0 with the token loop inside the block loop - the weight side decoded once per verify window, each token's integer chain and float operations exactly ggml's, so the results are ggml's to the bit for any group size (no Niko1221#152 rule). native_gu_rows / native_down_rows use them for those types (only UD-Q4_K_XL has them); STRATA_NO_KQ256 is the A/B arm. native_expert_parity checks them bit for bit against vec_dot for 1..8 tokens and times 4 tokens. - Stats: the decode's file tier alone (blobs, MB, MB/round, reading ms/round and GB/s per thread) and the prompt path's copies separately. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

# Conflicts: # serve/server.py # src/core/expert_source.cpp

…_K/Q5_1/Q8_0 multi-token kernels from two tokens on - Registering 30 GiB of a 40 GiB RAM tier left the driver unable to page-lock the prompt path's token id buffer afterwards (RTX 5070, WDDM: "prefill: the token id buffer"); the prefix is now capped at 24 GiB, the size a whole 24 GiB budget page-locks with, and STRATA_PARTIAL_PIN_GIB sets it. - A verify window's expert groups hold ~1.4 tokens on average; one token goes to ggml's own dot (the same bits, without the multi-token kernel's per-token arrays). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…ssed slices at once (PrefetchVirtualMemory) The copies' page faults read a few clusters each; one PrefetchVirtualMemory call for every role slice about to be copied lets the memory manager queue large reads at once, and the parallel copies then find the pages resident or in flight. Only the GGUF-in-place source's prefetch uses it; STRATA_FETCH_PVM=0 is the A/B arm. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…IN_GIB; the split's prompt chunk steps down - The expert arena was pinned at most 8 GiB whenever a layer split (or remote experts) ran, on every OS - a Windows/WDDM workaround that cost a Linux 4090 + 3060 split two thirds of its prompt speed (Niko1221#253, bettercallcaleb). The cap now applies only under WDDM (_WIN32, or WSL2 detected by /dev/dxg), for multi_gpu and expert_cache_remote alike. STRATA_ARENA_PIN_GIB overrides it on any OS: "0" pins the whole arena, N > 0 at most N GiB (arena_pin_cap_gib(), -1 when unset: the same name and meaning as the S2 branch's helper, file-local here). - The split's prompt chunk no longer stops the start when its buffers do not fit: a prompt path without a loan first gets the largest chunk whose buffers (Prefill::bytes_needed) leave 512 MiB free on its device, and an init that still fails steps down one size (8192 -> 6144 -> ... -> 2048 -> 1536 -> 1024 -> 512): Prefill::reset() frees what the failed init made, the loans are re-priced, and the failed allocation's sticky error is consumed (else the next launch check reported "out of memory"). - pinned.cu: the capped note no longer says "for CUDA1". - docs: MULTI_GPU.md, SECOND_GPU.md. Measured on amdbox (Linux, R9700 + RX 9070 XT, gpu [1, 0], Coder IQ1_M, 128 greedy tokens; tok/s): | cache | arm | 4K cold | 4K warm | 16K cold | 16K warm | decode | | auto (holds ~all) | 8 GiB cap | 678 | 1,355 | 1,709 | 1,880 | 44.5-51.5 | | | whole arena | 703 | 1,234 | 1,729 | 1,864 | 44.6-51.4 | | fixed 1500 on CUDA0 | 8 GiB cap | 608 | 1,101 | 1,287 | | 34.5-36.1 | | | whole arena | 716 | 1,160 | 1,803 | | 34.8-36.0 | (2 starts per arm for auto, 5 for fixed.) With most experts streamed the whole arena reads a 16K prompt 40% faster; with every expert already on the cards the two are within noise except 4K warm (-9%: the pinned share raises the auto lend to 90% and the streamed ring to 384 slots). Identity: the fixed-cache arms gave the same tokens in all 5 + 5 starts (4K, 16K, 4K again), and the auto arms 2 + 2. The step-down was exercised on the 9070 XT with --expert-cache 3000 --no-prefill-borrow --prefill 8192 ("need 4528 MiB, 2910 MiB free: 2048-token chunks"; with the init retry alone it went 8192 -> ... -> 2048, and 1536 ran all four requests). Known, also on main: that own-buffer configuration at 2048 makes hipBLAS pick hipBLASLt kernels whose code objects this ROCm 7.14 lacks (TensileLibrary_..._gfx1201-xnack+.co not found, "launchKernel failed") and the prompt hangs. The WDDM side (Windows, WSL2) is unchanged and not re-measured here. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…one when it holds every profiled expert + KV Before any later stage loads, split-auto checks whether CUDA0 alone holds every profiled expert pair plus the whole session (KV of every layer), the drafter + head, the verify windows and the reserve; if so it clears the split (one GPU) and logs why, else it logs how much of the profile CUDA0 would hold and keeps the split. Default behaviour is unchanged: the engine flag is opt-in, and the server passes it only for a config with "split_skip_if_fits": true (and a GPU list). Not with remote expert caches. Measured on amdbox (Coder IQ1_M, --expert-cache auto --prefill auto, 128 greedy tokens, gpu [1, 0] = R9700 first, 2 starts per arm, warm = the second request of each size): | arm | 4K warm | 16K warm | decode | | split (flag off) | 1,244 | 1,897 | 50.5-53.0 | | --split-skip-if-fits (on) | 1,776 | 1,791 | 59.7-61.8 | | R9700 alone | 1,787 | 1,801 | 59.9-61.3 | The flag's run is the R9700-alone run (same tokens, all four requests); with the 9070 XT first it keeps the split ("would hold only 9.84 of the profile's 23.42 GiB"). On this pair the split still reads 16K prompts ~5% faster. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

# Conflicts: # src/core/pinned.cu

…aster on an AVX2-only CPU (Ryzen 9 3900X: gate/up -2.6..-4.7% at 1 token, down -3..-5% at 2 tokens) This reverts commit 9e9dce4. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…t an expert boundary; the Q4_K/Q5_1/Q8_0 multi-token kernels opt-in (STRATA_KQ256=1) - Locking the whole 40 GiB arena in the working set and then registering its prefix made the next device allocation fail (the prompt path's token id buffer); the registered prefix is page-locked by the driver already, so only the rest is locked now. 24 GiB registered out of a 40 GiB budget then starts and runs. - The registered prefix ends at an expert boundary: a blob that began inside it and ran past it was taken as page-locked by cudaMemcpyAsync and refused (an adaptive refill failed, with 16 GiB registered). - The prompt path's token id buffer error names the CUDA error. - kq_avx2 multi-token kernels: bit-exact, but no faster in the engine (the rows phases stayed at 72 ms per verify round at a 40 GiB budget; a window's expert groups hold ~1.4 tokens and ggml's per-token calls find the weights in L1), so they are opt-in with STRATA_KQ256=1; native_expert_parity still checks them bit for bit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…m-split down kernel (164 blocks), rs/lo/inject in up's prologue (STRATA_GR_V3=0 = old)

@Adamyno

…); its shared memory is sized from the card's opt-in limit - one or two column halves, else the default kernels (Turing/HIP 64 KB fit with two; capability-based split as proposed by @Adamyno); the neutral knobs (DOWN_R2, UP_PF, KSPLIT) dropped (review)

…ure was generated, as its generator does; a partly missing set still fails; the open handle of a half-present fixture is closed Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…er on this layer's MoE input warms its predicted experts' pages - RouterLookahead: when the CPU pool starts layer l, a worker thread applies layer l+1's router (a BF16 host copy of ffn_gate_inp, 126 MB for 48 layers) to layer l's MoE input - the router's own input - for each token of the window, takes the top STRATA_LOOKAHEAD_K (default 10), drops the experts the GPU cache (host_res) or the RAM copy holds, and asks the source to warm the rest. It never waits: a prediction still running makes the next one skip. STRATA_LOOKAHEAD=0 is the A/B arm. - ExpertSource::warm / warms(): FileExpertSource with the GGUF in place issues one PrefetchVirtualMemory for the predicted slices (madvise(WILLNEED) elsewhere). Pages only: blob(), the experts computed and their bytes do not change (the same answers in every A/B run). Every other source: no-op, and the predictor is not started. - bf16_rows_dot_multi (kq_avx2.cpp, AVX2): the router read once for all tokens of the window. - Stats: experts warmed, how many of the decode's file-tier reads had been warmed, the predictor's time and skips. UD-Q4_K_XL at a 40 GiB budget (single-token-router version): ~48-54% of the file tier's reads had been warmed; reading time 850-968 -> 630-730 ms per verify round; decode 5.31/6.60 tok/s off vs 7.08/6.79/6.91 on. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

Add preliminary native v0.1.26 results, raw run JSON, scrubbed guest hardware facts, and the local API harness. Prompt and memory topology limitations are documented.

On the RTX 5070 / 64 GB PC the GPU computing a share of the misses over PCIe measured no faster than the CPU: 7.30 and 7.44 tok/s with 24 and 16 GiB of a 40 GiB budget registered, against 7.05-7.74 unpinned, and registering costs startup time and driver memory. The default budget mode is now locked-only; the routing-aware prefetch (multi-run A/B: 6.93 -> 7.91 tok/s mean, identical answers) stays on with the GGUF in place. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

… the mapped mode without experts.bin, --resident-budget-gib, the tier counters - docs/UNSLOTH_Q4.md: what UD-Q4_K_XL is (Hugging Face revision 38bb39e, the four shards with sizes and SHA-256), disk/RAM/GPU needs (111.3 GB download, 1.4 GB pack, no experts.bin), the iq_pack.py command and why --compat-bf16, the numerical policy (native experts/embedding/head/projections/PLE; the converted small tensors listed in conversions.json), the MTP draft layer, a server config (--native shard 1, --ple-gguf found by name, --resident-budget-gib N with N = RAM minus 20-24 GB), measured speed on an RTX 5070 / 64 GB PC (7-8.5 tok/s at 40 GiB; ~60 s to the first answer), the switches (STRATA_LOOKAHEAD=0, STRATA_KQ256=1, STRATA_PARTIAL_PIN=1, STRATA_FETCH_THREADS), scope and validation, credits (eddoursul Niko1221#245, gopinath87607 Niko1221#255, jagsan-cyber Niko1221#247). Experimental, not in setup. - docs/DETAILS.md: the mapped low-RAM mode reads a native pack's experts from the GGUF when it has no experts.bin (checked, identical answers on the Coder); --resident-budget-gib and the routing-aware prefetch; where the tier counters are (--stats, the server log, /metrics' ram_blobs / file_blobs / file_mb); a pointer from "Which model?". Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…plit GGUF, Q4_K/Q5_K/Q5_1/Q8_0 experts, RAM budget + SSD tier with routing prefetch), the low-RAM mode reads GGUF experts in place, Windows GGUF load 2x (Niko1221#230), server status race (Niko1221#266) and tokenizer (Niko1221#268) fixes, truncated tool calls (Niko1221#211/Niko1221#231), AMD intrinsics (Niko1221#262); setup requires it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…ATA_HIP_WMMA=1, gfx12 only) The v2 tensor-core design with gfx12's v_wmma_f32_16x16x16_f16: one block per (query, KV head), wave w owns dims [64w, 64w+64) (int8 scale group w), q and p split into FP16 hi + lo parts, int8 codes exact as FP16, scales in FP32, the four groups' q.k partials added in a fixed order, online softmax over 32-cell chunks. Masked (non-resident) cells get no weight, as in the decode kernel. FP32-level accuracy, not bitwise equal to the default kernel, hence opt-in. hip_prompt_attn_wmma = qsa_prompt_attn_parity built for HIP (skips off gfx12). NVIDIA builds unchanged (all new code is under __HIPCC__ / the HIP test section). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…ame (default on HIP; STRATA_HIP_ROUTER_OLD=1: old) On RDNA4 the router took 39 us per call in decode (16% of the decode GPU time): 512 dependent FP64 adds, ten selection passes with 20 block barriers, a renormalisation re-reading its weights from global memory. The sum's only use is inv = (float)(1.0 / sum): all terms are positive with the largest exactly 1, so the serial sum and a tree sum are within 2^-43 of each other; when the bracket tree*(1 -+ 2^-40) maps to one float, that float is the serial sum's (1/x and the cast are monotone), otherwise thread 0 runs the serial sum as before. The selection runs on wave 0 from registers (the argmax with the lowest index on ties is unique), the renormalisation from registers in rank order. hip_router_fast checks ids and weights bitwise against the portable kernel (65,536 rows with ties, near-ties, NaN rows, the forced serial path; and 256 experts / k 8). CUDA builds unchanged (HIP-only code). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

… unterminated Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…n's logits STRATA_DUMP_FIRST_LOGITS=<file>: the logits of the first window's first position (the token the prompt predicts), n_vocab float32, written once per run. Two engine variants run on the same prompt then give a first-token KL divergence: how far a prompt-path change moves the model's distribution, not only whether its top token changed. On a layer split the last stage's verifier writes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

…t; STRATA_EMULATE_CC for tests - generate.cpp: before the expert arena loads, the engine names the card and its compute capability and checks that the build has code for it (device_code_error: cudaFuncGetAttributes on a kernel). A binary built for other GPUs used to fail at the first kernel, after the whole arena had loaded; now it stops in ~0.3 s and says why. - include/strata/core/emulate.hpp: STRATA_EMULATE_CC=75|80|86|89 makes the host's device queries (compute capability, shared memory per block) answer as that generation, for a PTX build of it JIT-compiled on a newer card - ggml's MMQ config, the QSA kernels' choice, fused_gr's shared memory. Tests only. - STRATA_QSA_WARP=1|select|attn: the pre-sm_80 QSA kernels on any card (A/B). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

… and timings use it (Niko1221#548, as Niko1221#536) Dev::put uploads with a pageable cudaMemcpy, which can return before its DMA lands, and the graph replays and the timing cases run on a cudaStreamNonBlocking stream that does not wait for the legacy stream: the graph could read the previous replay's logits (one argmax token off; 5 of 6 replays on an RTX 5060 Ti, now and then on a 5090). Dev::put now synchronizes the device after its copy, which covers run_graph_case and bench(). Test only; no engine code. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…en libcudart is older than the headers A CUDA 13 build that linked the system's CUDA 12 libcudart.so (a kit with a dangling libcudart.so symlink) read cudaDeviceProp shifted: sharedMemPerBlockOptin came back as 1, Niko1221#420's fits() then refused MMQ for every i-quant and the prompts ran ~4x slower, with a log line printing "1 bytes of shared memory per block". - ggml_cuda_info() (CUDA) reads cc, smpbo, smpb, nsm, warp size, integrated and cooperative launch with cudaDeviceGetAttribute, which is ABI-stable. A matched build reads the same values: the same MMQ choices, the same output. - The engine warns at start when the loaded CUDA runtime's major version differs from the headers' it was compiled with, and says to rebuild against one toolkit (-DCUDAToolkit_ROOT). HIP is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

The provider, the context and output limits, the thinking level as a default and as variants, and how they relate to the context check, fit_max_tokens and reasoning_budget_tokens. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

One line under the start banner: the config's engine options without the model's file paths, and the server's fields (host:port, an API key set or not, GPUs, draft subset, fit_max_tokens, thinking budget). The key itself is never printed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

… free The weight arena is the first large VRAM allocation and its size does not depend on the context, so when it fails something else holds the card (another program, an engine still exiting). The error now names the CUDA error and the card's free and total VRAM at that moment; a layer split's stage says the same for its card, with the MiB it needed. Error paths only. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

A web page of another site could point its own name at 127.0.0.1 and reach the server as the same origin: without an API key it could read /status, /metrics and /api/requests, change settings and run MCP tools. The server now answers only Host names it knows: localhost (and *.localhost), any IP address (rebinding needs a name), the bind address and, bound beyond this PC, the PC's name, name.local, its LAN addresses and host.docker.internal; any port. No Host header (HTTP/1.0) passes. "allowed_hosts" in the config and STRATA_ALLOWED_HOSTS add names (".example.com" for a domain, "*" turns the check off); trusted_origins' hosts count too. A refusal is a 403 naming the setting and one log line. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

/unload and /load took a plain form POST from any site: they now take JSON from Strata's own page only, like /settings (_own_page). Without an API key, a POST to /v1/* that carries an Origin must come from Strata's own page, a page on localhost or an allowed host name, an origin in trusted_origins or cors_origins, or a non-web scheme (browser extensions, desktop apps), and then send JSON; others (and Origin: null) get 403, so a web page can no longer burn GPU time with a text/plain POST. Requests without an Origin (curl, the SDKs) are unchanged; with an API key, the key decides. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

It keeps 256 of the 512 experts, chosen on code data, so it is weaker outside code and in other languages; for general use, or when RAM allows, MODELS.md now points to the full models. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

A rebinding page cannot send the key, so with one configured the check only broke tunnels and proxies that pass their own name on (cloudflared does). Keyless setups keep it, with allowed_hosts; the 403 now also says that setting an API key turns the check off. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

A verify window committed every accepted position (a + 1) into the session, but handed out only the outputs below max_tokens and through the first end of turn. When an answer ended inside a window, the session held drafted tokens the client never saw. The follow-up that sends the answer back then no longer started with the live session and fell back to the turn-boundary checkpoint: the answer and the new message were read again. The window now counts the outputs it hands out first and commits exactly their positions - the same state as an answer that ends on a window's last token.

q8atnight · 2026-10-03T18:36:32Z

Tested on 2× RTX 3090 (sm_86, NVLink), Qwen3.8-Flash-Next IQ3_S, --spec 4 MTP, conversation cache on (--conversation-cache-mib 12288 --conversation-cache-slots 4), --peer-device 1, v0.1.38 + our patches with this PR merged.

Exactness: our greedy gate vs a stock v0.1.38 single-GPU reference: PASS (byte-identical), 20K-token long gate: PASS.
Speed: unchanged (prefill 2807/3154/2996 t/s at 8K/32K/128K, decode 120/111/107 — same as without it within run noise).
Follow-up turn: a 23K-token chat continued with ~1.5K new tokens: TTFT 1.22 s vs 1.33 s without the PR (same prompts, same night), consistent with the resume no longer falling back to re-reading the tail.

We're carrying it in our build. Thanks for the clean, small fix.

Niko1221 · 2026-10-04T12:35:41Z

Thank you, and thanks @q8atnight for the gate run. This changes the session state a follow-up continues from, so it needs our identity gate. It is planned for a later release rather than 0.1.39.

…iko1221#652) Upstream PR Niko1221#652 by anon761 (Niko1221#652), commit cf14881, cherry-picked; the one conflict was the adaptive-tier thread line, which keeps ac922's release_inherited_pin (POWER: a new thread starts on the host core). A window committed every accepted position (a + 1) but handed out only the outputs below max_tokens and through the first end of turn, so an answer cut inside a window left drafted tokens in the session that the client never saw, and the follow-up fell back to the turn checkpoint. 4 GPUs, 8 two-turn chats (thinking off, first answer cut at 22-58 tokens, the follow-up sends it back verbatim): before, one follow-up missed the live session (14 reused + 69 read, 458 ms); now all 8 resume (~180 ms, the new message read). Decode texts on three 384-token answers identical. (cherry picked from commit cf14881) Co-Authored-By: anon761 <anon@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ZHWKWn7UYZV64KwpGrWuN

…shed README.md: everything the fork adds (the AC922 summary and numbers, the DGX Spark note) is one block above upstream's README, which follows unchanged - the diff against upstream/main is additions at the top only, so upstream's README merges without conflicts. docs/IBM_AC922.md: the launch configuration's numbers after the v0.1.39 merge and the layer-split VRAM fix (llama-benchy, 4 GPUs prefill 1,967-6,353 tok/s, decode mean 83.4; 2 GPUs 1,845-3,785, 73.4; single prompts up to 123K; a 248,934-token prompt at 7,299 tok/s with --max-context 262144); the 2026-10-04 work (Q5_K fused gate/up, PLE early landing and threaded gather, host grouping event, STRATA_PREFILL_ADAPT, the m8n8k4 prompt attention now the Volta default, async split commit, PR Niko1221#652, the draft-head booking); the run section updated (--prefill 3072 --prefill-ring 320 on 4 GPUs, auto + ring 480 on 2; STRATA_ATTN_WMMA no longer needed); quality rows for the current kernels (32K KL 0.0220, 2K 0.0154, byte-identical across the merge); dropped ideas; a section on merging upstream. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ZHWKWn7UYZV64KwpGrWuN

blange48 · 2026-10-04T16:49:21Z

Feedback from production: we have run this since 0.1.38 and now on 0.1.39 (rig: 4 x RTX 5080 16 GB (PCIe Gen3, one card at x8), Linux, IQ3_S, 262K context, int8 K/V with --kv-resident 32768, layer split 12,24,36 with --trim-stage-weights, "parallel"/--batch 8 --batch-groups 4, MTP spec 4), behind an agent client (OpenCode) and chat users.

Good: no regression seen. Exactness checks with --pcie-frac 0 --adapt-every 1000000 and STRATA_IQ_MT_MIN=1 all stay identical: tools/batch_test.py with 8 and 3 slots, and tools/parking_test.py (97/97 tokens, 141 MB of K/V reused on the second park).
Two long conversations alternating come back at 0.31-0.38 s to the first token.
Not measured: a before/after on the exact case you fixed (an answer ending inside a window, then its follow-up). We applied it as a correctness fix and didn't isolate the gain.

Niko1221 · 2026-10-06T10:11:02Z

Heads-up: today we cleaned up this repository's commit history, and GitHub closed this PR automatically when main was force-pushed (that is the "base branch was force-pushed" event above). It is not a rejection.

GitHub cannot reopen it, because its branch is built on the old history. If you want to keep it going: start a new branch from the new main, cherry-pick your own commits onto it (git fetch upstream && git switch -c my-fix upstream/main && git cherry-pick <your commits>), and open a new PR. Or simply leave it: when we take your change for a release we port it onto the new main ourselves and keep you as its author.

Sorry for the noise, and thank you for the contribution.

… window hands out

Upstream merged Niko1221#863, Niko1221#732, Niko1221#869, Niko1221#931 and Niko1221#846 (our copies dropped) and took its own form of Niko1221#970 (Niko1221#804). Kept: Niko1221#851, Niko1221#652, Niko1221#855, Niko1221#813, Niko1221#789. A pool turns off what it cannot run yet: --pipeline-windows, --batch-mtp, --kv-grow, session files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0157fjixQD2nRxvCvtXq66DD

… window hands out

KarlGrier · 2026-10-09T08:15:33Z

Test result on v0.1.41, in case this is ported. The bug is still there.

We carry this PR's change in a local build on v0.1.41 (fb58e0db). A local switch puts back v0.1.41's own commit, so the same binary ran with and without the change. Follow-up chats over the OpenAI endpoint, each turn sending the whole history back; a follow-up counts as "resumed" when it reuses everything the engine held after the previous turn. Two runs per setting, A B B A, 2026-10-09:

	plain follow-ups resumed	with thinking
v0.1.41's commit	9 of 14, 9 of 14 (each miss re-read 57-159 tokens of the previous answer)	4 of 5, 3 of 5
this PR's commit	14 of 14, 14 of 14	4 of 5, 5 of 5

The one remaining thinking-mode miss is expected. It follows a reply cut at its 200-token cap, whose thinking the client then leaves out of the history, so the new prompt is shorter than the engine's state (23 tokens re-read). When we first took it in (v0.1.38, together with three other fixes), decode speed and time per verify round were unchanged within 0.4 %.

Setup: RTX 5080 16 GB + RTX 4060 Ti 8 GB (helper expert cache), Ryzen 9 9950X3D, 96 GB DDR5, Linux 7.0, CUDA 13.3, Qwen3.8-Flash-Next GSQ-RCO IQ3_S, --spec 4 --kv int8, the server's default sampling (temperature 1.0).

Not tested: a layer split, --batch slots, upstream's --conversation-cache (our build parks conversations with its own cache), and Windows.

Measured and drafted with Claude Code.

Upstream 0.1.45 took in ten of the PRs we carried: Niko1221#567 (prompt encoder), Niko1221#1164 (sibling borrow), Niko1221#1166, Niko1221#1368, Niko1221#1367, Niko1221#1120 (split overlap), Niko1221#1912, Niko1221#1888, Niko1221#1949, Niko1221#1305. Their versions are kept and our copies dropped: verify.cpp's duplicate prelaunch/handoff block, verify_kernels.hpp, the qsa prompt kernels, gr_upmix_parity, the tokenizer and its tests (tools/test_strata_tokenizer_upstream.py is gone: upstream's test file is the one now), the second quantize_scatter declaration. README credits updated. Pool parts kept on top: - conversation cache: upstream renamed the borrow guard (pin -> guard); clear() resets it. - prefill: the remote_next hand-off beside upstream's hand-off pieces/worker (a pool stage has no pieces: the whole chunk goes to the workers). - generate.cpp: --pool-affinity measure over a node's own layers; the node's dense-range load beside STRATA_HC_REQ8; the carve on primary_device over the pool range; checkpoints keep pool_id (STRATA_CKPT_POOL/THREADS' parallel save is skipped on a pool PC); the workers' retain after eviction and after a resume; the borrow/restore of parked conversations on the workers with upstream's --conversation-cache-park-gain; Niko1221#652's emit count in upstream's merged windows. - expert_source: our budget retry beside upstream's lend budget. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0157fjixQD2nRxvCvtXq66DD

Niko1221 and others added 30 commits October 1, 2026 00:35

Merge branch 'f-131-srv' into r131-q4

a348c7a

Merge branch 'f-q4-on242' into r131-q4

b93a00c

# Conflicts: # serve/server.py # src/core/expert_source.cpp

Merge branch 'f-131-amd' into r131-amd

6926e0b

# Conflicts: # src/core/pinned.cu

decode: hyper-connection read in 2 kernels - norm folded into a strea…

de3f045

…m-split down kernel (164 blocks), rs/lo/inject in up's prologue (STRATA_GR_V3=0 = old)

docs: add community benchmark submission guide

8b9bb4e

docs: add RTX 5090 community benchmark results

b4f70e7

Bench: RTX 3090 / dual RTX 3090 on EPYC 7453

e30cd9b

Add preliminary native v0.1.26 results, raw run JSON, scrubbed guest hardware facts, and the local API harness. Prompt and memory topology limitations are documented.

Merge branch 'f-q4-on242' into r131-final

c4175bd

Merge branch 'f-q4-on242' into r131-final

652437b

HIP (S6): router_top10.cu - drop a nested #if that left the HIP block…

78700ca

… unterminated Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

Niko1221 and others added 13 commits October 3, 2026 01:57

Merge branch 'f-138-fixA' into f-138-prs

b20cdf5

Merge branch 'f-138-sec' into f-138-prs

e4e0cee

Merge branch 'f-138-fixB' into f-138-prs

99f3dbd

blange48 mentioned this pull request Oct 4, 2026

Several conversations at once: batch slots, a pipelined layer split, per-stage dense weights #559

Closed

This was referenced Oct 4, 2026

inference-oom: audit MTP context semantics and index reuse jomcgi-org/homelab#6834

Closed

inference-oom: persistent prefix-state store (host RAM, NVMe, S3) with TTL jomcgi-org/homelab#6840

Closed

Niko1221 closed this Oct 6, 2026

Niko1221 force-pushed the main branch from 1735d64 to 82f46a8 Compare October 6, 2026 09:48

fragtion added a commit to fragtion/StrataPool that referenced this pull request Oct 6, 2026

from upstream PR Niko1221#652: serve: commit only the tokens a verify…

7853854

… window hands out

fragtion added a commit to fragtion/StrataPool that referenced this pull request Oct 6, 2026

from upstream PR Niko1221#652: serve: commit only the tokens a verify…

fa4db59

… window hands out

k0valik mentioned this pull request Oct 15, 2026

replay (review only): hub.lumenfield.work-eelgaev-Strata-AC922-ac922 — filtered squash of 920 commit(s) from https://github.com/eelgaev/Strata-AC922 k0valik/Strata#4

Open

Navigation Menu

Repository navigation

serve: commit only the tokens a verify window hands out - #652

serve: commit only the tokens a verify window hands out#652
anon761 wants to merge 629 commits into
Niko1221:mainfrom
anon761:serve-commit-emitted

anon761 commented Oct 3, 2026

Uh oh!

q8atnight commented Oct 3, 2026

Uh oh!

Niko1221 commented Oct 4, 2026

Uh oh!

blange48 commented Oct 4, 2026

Uh oh!

Niko1221 commented Oct 6, 2026

Uh oh!

KarlGrier commented Oct 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

anon761 commented Oct 3, 2026

What

Measured

Uh oh!

q8atnight commented Oct 3, 2026

Uh oh!

Niko1221 commented Oct 4, 2026

Uh oh!

blange48 commented Oct 4, 2026

Uh oh!

Niko1221 commented Oct 6, 2026

Uh oh!

KarlGrier commented Oct 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants