Repository navigation
bench: community report, RTX 5090 + Ryzen 9 9950X3D, IQ3_S at the full 262K window; prefill auto:32768 A/B; conversation-cache 32 GiB demo - #440
Closed
Ambolio wants to merge 511 commits into
Conversation
…rompt/token path)
…mlock succeeded else 'loaded (not locked)', neutral comments + --help (review)
Under WSL2 the NVIDIA driver can pin/map only about 1 GiB of host memory in total (dmesg: "dxgk: create_existing_sysmem: establish_gpadl failed: -122"). With --expert-cache-device1/2 the startup ran out of it twice: - CUDA2's context is created after CUDA0's weights and MTP, whose mapped host buffers have spent the budget: "CUDA2 experts: cudaSetDevice(2) failed: out of memory". STRATA_EARLY_REMOTE_CONTEXTS=1 now creates every secondary context up front, like CUDA1's (opt-in: the default order is unchanged). - With three contexts up, the 260 MiB mapped native embedding no longer fits: "native embedding: cannot pin 260 MiB". It now falls back to a device copy on the current GPU instead of failing the start. Measured in Docker Desktop (WSL2), RTX 5070 Ti + 2x RTX 5060 Ti, Swift 1.5 IQ3_XXS: starts and serves with both helper caches (5000 experts each). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Display both rates in t/s with independent graph scales and a darker prefill curve. Expose prefill_tok_s_mean using the engine's existing rate, excluding cached tokens. Co-authored-by: Codex (gpt-6-astra-medium) <noreply@openai.com>
…ixed 90 GB IQ3_S and IQ3_XXS asked for 262K were cut to 128K on any machine under 90 GB of RAM. The limit is now counted: the experts' arena, plus the context's 8-bit KV (in RAM with KV streaming, ~13.7 KB a token), plus 24 GB of room. - A 64 GB PC still gets 128K for both models, as before. - A box with ~84 GB keeps the 262K it asked for. A Colab A100-40G (83.5 GiB) runs IQ3_S at 262K, with images, in 56 GiB.
The canonical path zeroes session state from put_input at position 0, but a native pack breaks out of that loop before the first put_input and runs verify windows instead, leaving cudaMalloc'd state uninitialized. On a reused allocation that produced NaN/overflow and degenerate identical-token output. Zero the state explicitly when native_pack is set. Also removes an unused argmax() helper that emitted a compiler warning, and ignores the local .pi/ harness directory.
…gative control (Niko1221#148; CMake on the backend variables, CUDA-only) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…mode (Niko1221#167 follow-up); no .pi/ ignore line Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…g until the watchdog (Niko1221#224); CMake warns when sm_120 is built with CUDA < 13.0 (Niko1221#220, Niko1221#224); iq_pack writes native_experts.txt under a temporary name, so a stop part-way leaves no partial pack (Niko1221#172) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…iko1221#197), faster prompt kernels (Niko1221#187, Niko1221#188, Niko1221#207), correctness fixes (Niko1221#154); setup requires it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…r/YaRN) The artifact pins theta 1e7 with no rope.scaling keys, so the engine had exactly one rope: inv[i] = theta**(-2i/n_rot), no freq_scale, no correction. Contexts past the trained 262,144 had no coherent path. rope_scaling.hpp adds the one process config - none | linear | YaRN, llama.cpp's types - and build_rope_table grows an overload that folds the whole scaled angle (ggml's rope_yarn: interpolation, the corr_dims ramp along the pairs, and the mscale magnitude correction) into the float64 host table. The rotation kernels cannot tell a scaled table from an unscaled one, so nothing downstream changes: for the default decode path and the indexer's pooled key, scaling IS the table. The math is transcribed from the vendored ggml (ggml.c corr_dims, ggml-cuda/ rope.cu rope_yarn/ramp, MIT headers kept), not re-derived; the one deliberate deviation is bookkeeping - the ramp takes the PAIR index ggml's i0/2 would produce, because every Strata rope kernel speaks in pairs. For type None the new builder calls the original loop verbatim, and the parity test holds it bit-identical; linear(4) is checked bit-exact against a float64 reference and equal row-for-row to none at p/4; YaRN against a float64 transcription of the ggml spec, plus the structural checks a tolerance cannot fake: cos_tab[0] == mscale (position 0 is angle 0 for every pair), first pair extrapolates / last interpolates at a far position (cos for the first, sin for the last - near zero sin IS the angle, which is where interpolation happens), and an observability assertion that the fixtures see scaling at all. Co-Authored-By: Claude <noreply@anthropic.com>
… sites The table path got scaling for free (it lives in the table contents); the three ANALYTIC paths computed the angle in-kernel and so need the config explicitly. Each wrapper now takes the RopeScaling and passes the resolved constants - freq_scale, the corr_dims pair, ext_factor, attn_factor - as kernel arguments. They are process constants set before session_init, so baking them at CUDA-graph capture is correct by the same argument as every other kernel argument: only per-token values must ride device memory, and positions still do (st.pos_dev). The shared float32 arithmetic is rope_scaled_angle (rope_scaling.hpp), a line-for-line rope_yarn: one definition for native_rope, the native indexer's pooled-key rotation and prefill's rope, instead of three transcriptions that could drift. The spare indexer key keeps its zero angle; under YaRN the mscale rides in through cos(0) on both pooled keys and indexer queries, so top-k is unmoved. rope_parity adds the check this whole wiring answers to: native vs table on the same input, yarn and none alike - which caught the first draft passing scaling.mscale() into the helper that applies the log term itself, a double magnitude correction visible as ~7% on every rotated dim. Co-Authored-By: Claude <noreply@anthropic.com>
… defaults Nine flags, llama.cpp's names: --rope-scaling none|linear|yarn, --rope-scale, --rope-freq-base, --rope-freq-scale, --yarn-orig-ctx, --yarn-ext-factor, --yarn-attn-factor, --yarn-beta-fast, --yarn-beta-slow. Ranges die in the validation block at second zero; the config itself resolves after the model file has had its say and before session_init builds the table or captures a graph (rope_scaling.hpp's one-writer rule). Precedence: CLI over the model file's qwen4exp.rope.* keys over the struct defaults - today's artifact ships no rope keys, so that channel is a no-op defaults path for future fine-tunes. A scaled run prints its resolved summary (type, factor, freq_scale, base, mscale, context vs trained) to stderr, which is what the bench logs quote, and warns when scaling is on with the context still inside the trained range - YaRN's magnitude correction applies everywhere, so that run is not the stock model. Co-Authored-By: Claude <noreply@anthropic.com>
CONTEXTS grows 383K/512K, labelled in the menu, and the setup refuses a context past the trained 262,144 without --rope-scaling (the engine would refuse it too). The factor defaults to the smallest that covers the chosen context (1.5 at 393K, 2 at 524K) and lands in the engine args like every other flag. Co-Authored-By: Claude <noreply@anthropic.com>
DETAILS.md gains the 'Context extension' subsection - what linear and yarn do, the setup path (383K/512K in the context menu pick the factor themselves), the engine flags with llama.cpp's names and their defaults, why the scaling is fixed for the whole run (K sits in the cache post-RoPE), and the one honest caveat: inside the trained range a scaled run is a slightly different model, and the recall numbers are text-only at one depth. Troubleshooting grows the row for the refusal a context past 262,144 now raises without the flag; README points the context question at it. Co-Authored-By: Claude <noreply@anthropic.com>
… the GGUF rope keys The defaults collided with the opt-outs: "none" was both the untouched --rope-scaling default and the explicit switch-off, and 1.0 the same for --rope-scale, so qwen4exp.rope.scaling.type/factor from the model file won even against an explicit flag - the opposite of the stated precedence (and of docs/DETAILS.md). Now the empty string and 0 mean the flag is absent (the same sentinel idiom as --rope-freq-base 0 and --yarn-orig-ctx 0), the model file fills only what no explicit flag named, and `none` / `1` are the opt-outs they read as. The resolution moves above the weight pipeline: the config needs only the CLI and the model file's metadata, its one hard constraint (set before session_init builds the table and the graphs capture the constants) is unchanged, and a bad rope key now dies before the shards are scanned. Checked on the bench machine against a synthetic GGUF carrying rope.scaling.type yarn / factor 4: the flags absent -> yarn 4; --rope-scaling none -> no scaling; --rope-scale 1 -> yarn, factor 1, mscale 1; explicit linear 2 -> linear 2; a GGUF type other than none/linear/yarn dies before the weight scan. The real artifact (no rope keys) starts and generates unchanged, and rope_parity stays green. Co-Authored-By: Claude <noreply@anthropic.com>
The first end-to-end check that a scaled run recalls past the trained end: 8 of 8 probes found the needle at mid-depth, temperature 0 - 1k/32k/128k on the stock model (arm A), 262k and 512k prompts on linear factor 2 (arm B) and yarn factor 2 (arm C), 512k being 2.01x the trained 262,144. The yarn 262k probe was repeated green after the rebase onto 0.1.20, next to the per-arm engine and server logs and the launch recipes. What it does not establish is written down in the README: one depth, one needle per length, and no long-document perplexity with and without scaling at 262K yet - the one open measurement. Co-Authored-By: Claude <noreply@anthropic.com>
…test The rebase onto 0.1.24 had brought the QSA indexer's C-2 chunk append, whose kernels rotated the pooled keys inline - cosf/sinf of the unscaled angle, no mscale - so a scaled prefill would have written indexer keys under a different rotation than the per-token append the decode path uses (rope_scaling.hpp's one-rotation rule, broken silently). The batch path now resolves freq_scale/corr_dims/mscale from the same RopeScaling the single append reads and rotates through the shared rope_scaled_angle; the raw freq_base parameter is gone from its signature. qsa_parity pins the contract the C-2 rework claimed: the batch append's whole end state (pooled, spare, tail ring, block_pos) is BIT-IDENTICAL to the sequential single appends', under none and yarn 2 alike, and the scaling is observable in the batched keys. Co-Authored-By: Claude <noreply@anthropic.com>
The comment said the setup adds the flag when the chosen context needs it; the code called fail(). `START-HERE.bat --setup --context 393216` - the command the docs give - stopped instead of configuring. Now an omitted --rope-scaling with a past-trained context resolves the config itself: yarn (llama.cpp's extension method, recall-tested here at 512K; one question when interactive, the silent default with --yes) and the smallest covering factor (1.5 at 384K, 2 at 512K). An explicit none is still refused for such a context, with the reason and the way out spelled out instead of a bare stop; explicit linear/yarn/scale pass verbatim; inside the trained 262144 nothing turns on by itself. The resolution moved AFTER the RAM reduction (the §4 ordering): the factor must cover the context the engine will serve, so a 384K choice reduced to 128K by the RAM limit gets no scaling at all, and an explicit selection that survives the reduction says so in a warning. The rope config is now two pure functions (covering_factor, resolve_rope) with its own unit tests (tools/test_setup_rope.py); the engine args take the resolved values; help, module docstring, the context menu note, README and docs/DETAILS.md describe the implemented behavior. Co-Authored-By: Claude <noreply@anthropic.com>
The triple-check pass found what the menu could not: --context is not limited to CONTEXTS, and the factor ladder (1.5, 2.0) ended at 512K, so `--context 600000` died with a bare traceback instead of a setup message. The ladder extends (3, 4) and a context past its top refuses cleanly, naming --rope-scale as the way out. The interactive explanation block also stays quiet under --yes, whose ok() line announces the choice already. Co-Authored-By: Claude <noreply@anthropic.com>
…rved The factor ladder (smallest of 1.5/2/3/4 covering the chosen context) is gone. The automatic factor is now the final context over the trained 262144, clamped at 1 - max(1.0, final / trained) - computed AFTER the RAM reductions, so it always matches the context the engine will serve: 512K + IQ3_S under 90 GB of RAM + yarn, no factor: final 128K, factor 1 no reduction: 384K -> 1.5, 512K -> 2 a chosen method inside the trained range: factor 1, not 2 Factor 1 removes the automatic expansion - the trained angles stand - but it is not a switch for rope as a whole: an explicitly chosen method's settings keep their defined behavior. A user-supplied --rope-scale is still kept verbatim when a reduction changes the context; the ok() line now says where the factor came from (final context / trained, or as requested), and help, docstring and docs/DETAILS.md drop the "smallest covering factor" wording. Co-Authored-By: Claude <noreply@anthropic.com>
…he window) The implementation landed with the derived-factor change; per the item's own scope note this is verification and tests, not a second one. The table from the item is now pinned case by case - 131072 and 262144 with yarn and no factor resolve to 1, an explicit --rope-scale 2 stays 2 inside the window, 393216/524288 keep 1.5/2 - and linear is checked under the same rule the old fallback applied to it too. On the real engine the resolved combo runs as "rope scaling yarn, factor 1 (freq_scale 1, mscale 1.000000)" and a greedy 12-token run is token-identical to the stock baseline - evidence on one path, not a bit-identity claim. Co-Authored-By: Claude <noreply@anthropic.com>
…es bits Item 8 verification. The parity section now runs all three scaling configs - none, linear 2 (angle shrink only), yarn 2 (ramp + mscale) - and "bit-identical" is asserted on the BIT representations (memcmp of the state words), not float !=, which would pass a NaN-payload or -0.0 flip. Each scaled variant must also differ from the none baseline in pooled-key bits, so the parity cannot pass because both sides ignore the scaling. Green on this tree: 0 of 33409 state words differ in every config; 128 (linear) and 192 (yarn) pooled words move. Co-Authored-By: Claude <noreply@anthropic.com>
v0.1.29's new draft_kv site (the 0.1.26 batched draft prompt pass) kept upstream's float freq_base form through the rebase, which both misses the scaling on the draft K/V and does not compile against the RopeScaling signature the other four sites carry. It passes rope_scaling() now, like the prompt K, Q and indexer-query sites. Co-Authored-By: Claude <noreply@anthropic.com>
…correction (TODO 22) The note fired for every scaled run inside the trained 262144 and its text named YaRN's magnitude correction even for linear, whose resolved mscale is exactly 1. Split in two: the in-window warning now only says that no position needs the extension while the resolved scaling still applies to every angle, and the magnitude explanation became its own note, printed only when YaRN is resolved and its magnitude multiplier differs from 1 (yarn factor 1 inside the window stays quiet; linear never sees it). The main rope scaling line is unchanged, the numbers it prints already carry the resolved mscale. Diagnostics only. Checked on the rebased tree: none prints nothing, linear 2 prints the in-window note without the magnitude line, yarn 2 prints both (mscale 1.069315), yarn factor 1 prints the in-window note only, and yarn 2 past the trained window prints the magnitude note without the in-window one. Co-Authored-By: Claude <noreply@anthropic.com>
…1221#414) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…keeps the old kernel (Niko1221#371) The host check compared the major version only (>= 7) while the device code traps below sm_75, so an sm_70 card took the m16n8k8 kernel and hit __trap(). sm_75 and newer: the same choice as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…section (Niko1221#408, Efs-O) Opt-in; without it the encoder gets the engine's environment as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…21#369, dag08) open_sized seeded every layer's cursor at slot 0, so all layers wrote the same slots and the startup check failed; the profile fill ended at the first full layer. Now each layer starts in its own range, a full layer skips only its own pairs, and per-layer mode keeps slots of the largest blob (its ranges ignore the rank a sized slot was cut for). The default (shared) cache is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…o-half split (Turing; Niko1221#375, kenh0u) Opt-in path only; the default read is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
With STRATA_GR_V3=1 "fused GR multi max-T=8 LDS launch and changing graph replay" failed: the test compares the multi-token read with the single-token kernel bit for bit, and v3 sums `down` in another order (row group x stream blocks). It also compared `lo`, the default kernels' workspace between down and up, which v3 keeps in shared memory and never writes. Under v3 the test now checks R_out, rs, inject and mixed within 2e-6 of the largest reference value and leaves `lo` out; without it, bit for bit as before. Passes both ways. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio
…1#406 Niko1221#364 Niko1221#384) setup.main() with --yes, and with Enter for every question, on 64 GB + 32 GB, 32 GB + 2x24 GB, 47 GB + 2x16 GB, 96 GB + 16 GB and 128 GB + 24 GB, for the recommended size, Q2_0, IQ3_XXS, IQ3_S and UD-Q4_K_XL: the run config recorded by this setup (tools/test_setup_golden.json), so the choices that follow cannot change the default. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…che 0 --adapt-swaps 0 (Niko1221#410) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…safety check (Niko1221#403) FileExpertSource::pin_cache_complement read the available RAM twice: the budget was clamped to the first reading (less the 4 GiB headroom) and the resident-memory check then failed it on a second reading a few MB lower, so a 64 GB PC with setup's --resident-budget-gib 40 could exit at start. With a budget, the check now reuses the reading the budget was sized from, and a clamped budget stops 256 MiB under the room. An unclamped budget holds the same experts as before (the lend region is off with a budget, so `budget` there is unused). If the budget still cannot be kept (no RAM reading, an allocation that fails), generate.cpp warns and serves every expert the GPU does not hold from the files through the OS file cache (--mmap-experts) instead of exiting; the complement is left unbuilt on every failure path. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…with a warning (Niko1221#364 Niko1221#384) The low-RAM mode's resident variant has no layer split yet, and the engine refused the pair. When the residency comes only from setup's soft --resident-experts (no --resident-cpu-experts, no --resident-budget-gib, no remote caches), it now prints a WARNING - the experts the GPUs do not hold are read through the OS file cache and RAM may fill up during long prompts - and continues exactly as --mmap-experts: the resident flags are cleared before anything reads them, so the source, the placement and the per-stage caches are the mapped mode's (Niko1221#364's hand-edited config). An explicit --resident-cpu-experts, a budget, or remote caches are still refused. The explicit --expert-cache reduced under a layer split is now a WARNING that names --vram-reserve-mib. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…RAM budget are kept with their risk (Niko1221#406 Niko1221#403 Niko1221#364 Niko1221#384) The recommended answers stay the defaults, and --yes / Enter write the same config as before (tools/test_setup_golden.py, five PCs x five sizes). What a user asks for beyond the recommendation is kept and its risk said: - Niko1221#406: the RAM rule for IQ3_XXS / IQ3_S (experts + KV + 24 GB) is part of the recommended context (the smaller of it and the GPU's rule), no longer a cap: an explicit --context, a pick in the list or the earlier install's context is kept with a note ("256K with IQ3_S needs ~78 GB of RAM by setup's estimate; this PC has 64. Kept as you chose ..."), and the list marks the sizes past it. The low-RAM mode keeps its KV cache in VRAM, so there the context does not count against RAM. - Niko1221#364 Niko1221#384: the low-RAM mode recommends one GPU (the resident variant has no layer split) and asks: 2 = all the chosen GPUs with the experts read through the OS file cache (--mmap-experts, layer_split auto; faster in both reports, RAM can fill up during long prompts). An explicit --gpus (or the earlier install's cards) is kept; --yes keeps one GPU. The GPUs' share counts every card. The one-GPU text no longer sends users to --low-ram off. At start, --gpus on a --resident-experts config (and offer_together's "y", whose default is now n for such a config) switches it to --mmap-experts with a note instead of an engine refusal. - S4: --resident-budget-gib N for UD-Q4_K_XL (default: the RAM less 24 GB, as before); more is kept with a note, and a given N is not lowered for KV streaming. - confirm_risk(msg, explicit, yes): one helper for the stops a user may take on - asked with default n, and with --yes an explicit flag is the consent (--yes alone keeps the stop): UD-Q4_K_XL under 48 GB (--model), the RAM floor (--model), the paging question (--model, besides --low-ram off), a card under 8 GB named with --gpus. - --kv-streaming on|off|auto (auto: the RAM test, as before); on past the RAM test with a note, never with k8v4 or under WSL. The refusals that stay are the impossible ones: rope none past 262144, disk space, engine version, driver < 580, no AVX2, a size outside the family. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
… --resident-budget-gib / --kv-streaming (Niko1221#406 Niko1221#403 Niko1221#364 Niko1221#384) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
), setup recommends instead of forcing (Niko1221#406 Niko1221#403 Niko1221#364 Niko1221#384), fixes Niko1221#352 Niko1221#365 Niko1221#369 Niko1221#371 Niko1221#375 Niko1221#393 Niko1221#408 Niko1221#414; setup requires it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…the HIP build) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…uto:32768 A/B; conversation-cache demo Co-Authored-By: Claude Code <noreply@anthropic.com>
Owner
|
Heads-up: today we cleaned up this repository's commit history, and GitHub closed this PR automatically when GitHub cannot reopen it, because its branch is built on the old history. If you want to keep it going: start a new branch from the new Sorry for the noise, and thank you for the contribution. |
Owner
|
Thanks for the report. It is in 0.1.40.2 as bench/results/2026-10-02-community-rtx-5090-iq3s, with your authorship kept. Raw dumps and logs were trimmed (see TRIMMED.md in the folder); the README, summaries and configs are unchanged. The row is in bench/results/COMMUNITY.md. |
caolonghao
added a commit
to caolonghao/Strata
that referenced
this pull request
Oct 20, 2026
…wo-V100 measurement), UD-IQ4_XS, context ladder to 1M, concurrency The first measured layer split over two V100 (docs/NVIDIA_V100.md lists it as unmeasured), on PCIe cards without NVLink (device ID 10DE:1DB5; the driver's "SXM2" name is a VBIOS label): engine 0.1.40.2 source-built with CUDA 12.8 (sm_70 + STRATA_EXPERIMENTAL_SM60), 2x Xeon Silver 4316, 125 GB RAM. Main sweep (fresh prompts, 3 runs): 4K/32K prompt 1,277.8/2,355.3 tok/s, decode 77.3/80.3. Context ladder to 1M (yarn up to 4.0): no VRAM wall, caches still 99.7% at 1M, decode -5.6% from 512K to 1M at the same 450K prompt. Concurrency: queueing flat at ~66 tok/s total, 8-slot two-GPU pipeline 117.7 tok/s (+78%). Also: --prefill auto:32768 measured -32% at 32K on this split (Niko1221#433/Niko1221#440's +21-35% is single-card data; Niko1221#834 saw the same reversal), and a cold start can exceed the 900 s READY watchdog on slow storage. Co-Authored-By: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Community benchmark report under
bench/results/2026-10-02-community-rtx-5090-iq3s/, results only (no engine changes), in thedocs/COMMUNITY_BENCHMARKS.mdformat.Hardware: RTX 5090 32 GB (Windows 11, WDDM, driver 617.14). The 4K desktop sits on a second GPU, so the 5090 carries no graphics clients — relevant to #279:
--vram-reserve-mib 100shows no stalls here. Ryzen 9 9950X3D (AVX-512), 96 GB RAM, NVMe.Software: engine 0.1.33 release binary (BUILD.json: CUDA 13.0, sm_75/86/89/120+PTX), serve wrapper 0.1.30.
Model: Qwen3.8-Flash-Next GSQ-RCO IQ3_S (2 GGUF shards + PLE), MTP
rt, shipped expert profile, vision on CPU (16 threads).Configuration:
--expert-cache auto --prefill auto:32768 --spec 4 --spec-min-p 0.70 --mtp rt --max-context 262144 --kv int8 --kv-resident 65536 --vram-reserve-mib 100 --pcie-frac 0.35 --conversation-cache-mib 32768 --conversation-cache-slots 8,fit_max_tokenson. Expert cache 12,439 slots / 23.61 GiB. Full config inconfig.json.Findings:
cache_n = 0in every measured run — greedy, 256 out, 1 warm-up + 3): prefill 999 / 2,439 / 5,045 / 6,004 tok/s at 1K/4K/32K/128K; decode 144 / 149 / 142 / 125; drafts accepted 75-76%.--prefill auto:32768(theauto:32768form from the help; PR prefill auto: chunks up to 32768 (was 8192), +15% on a 32K prompt #282's idea): 32K 4,168 -> 5,045 (+21%), 128K 4,450 -> 6,004 (+35%). Larger than prefill auto: chunks up to 32768 (was 8192), +15% on a 32K prompt #282's +15% at IQ2_XS — consistent with their note that bigger experts gain more. Decode unchanged.needle_bench.py9 of 9 found at 32k/128k/256k x depths 10/50/90 — the 256k rows are 261,669-261,670 prompt tokens, the edge of the 262,144 window, on one 5090.conversation_cache_demo.py): the engine default is off (--conversation-cache-mib 0, 4 slots). With 32 GiB / 8 slots, follow-ups on two interleaved ~65k-token conversations cost 0.6-1.2 s instead of 11-13 s (10-20x); the engine restores 66,251 tokens in 82 ms and a live 160,978-token agent session in 186 ms; after an LRU eviction the partial path still reused 30,560 tokens and read 25,500 fresh at 5,002 tok/s. Parked bytes ~ 30 KB/token; 8 conversations (largest 162k tokens, 3.85 GB) sat in ~20.8 GB.auto:32768the engine warns "0 MiB of VRAM free with everything loaded — add--vram-reserve-mib 612". The prompt-path loan (7,736 slots / 14.64 GiB) peaks during prompt processing; idle free is 485 MiB and no stalls appeared across the full sweep. Is the warning conservative for this shape, or should the loan cap account for the reserve?Limitations: one machine, single stream, greedy, prompts built from repo text; TTFT measured client-side on a FIFO server (includes queue time); the baseline per-run JSON was overwritten by the A/B run (its per-run lines are in
bench.out); no quality suite beyond the needle checks.Measurements and write-up put together with Claude Code on my machine and checked by me.
🤖 Generated with Claude Code