[Benchmark] RTX 5070 12 GB + 32 GB RAM on Windows 10, Flash-Next IQ2_XS low-RAM mode (0.2.3.1)

bench/results/2026-10-23-community-rtx-5070-qwen3-8-flash-next-iq2-xs/README.md

@@ -0,0 +1,77 @@
+# Community benchmark on NVIDIA GeForce RTX 5070
+
+Measured on 2026-10-23 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run.
+
+## Hardware and software
+
+- GPU and VRAM: NVIDIA GeForce RTX 5070, 12 GiB VRAM
+- CPU: Intel(R) Core(TM) i5-14600KF (14 cores, 20 threads); installed RAM: 31.8 GiB (2 x 16 GB DDR4-3600); storage: the model files are on an aigo P2000 NVMe drive, which read at about 1.4-1.6 GB/s in an earlier measurement on this machine
+- PCIe: the GPU is on PCIe 5.0 x16; Resizable BAR is off
+- OS: Windows 10 Pro 22H2 (10.0.19045), Chinese locale; GPU driver: 591.74; CUDA: 13.1 (driver)
+- Strata: release 0.2.3.1 (project 0.1.48.1), commit 7ddc17a3; engine 0.1.48.1 built from that commit on this PC (CUDA 13.0, sm_120, MSVC 2022, `STRATA_PORTABLE=ON`), not the release package's binary
+- Background workloads and power limits: not recorded; the GPU also drives the display
+
+## Model and configuration
+
+- Model and quantization: qwen3.8-flash-next-iq2_xs (the original Flash-Next IQ2_XS pack, 35.5 GB of experts)
+- Context: 32768 tokens; parallel slots: 1
+- Settings: the daily configuration of this PC (low-RAM mode, `--mmap-experts`, because the experts do not fit in 31.8 GiB of RAM), with one change: the optional control-vector file that setup offers (`--control-vector-scaled`, not a speed setting) was left out so the numbers compare with other reports; thinking off for the suite; temperature 0
+
+```text
+strata.exe --pack iq2_xs --native Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf --ple-gguf Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00002-of-00002.gguf --expert-profile expert-profile.bin --expert-cache auto --prefill auto --spec 4 --mtp rt --max-context 32768 --kv int8 --mmap-experts --pcie-frac 0.55 --spec-min-p 0.70
+```
+
+Environment of the config: `STRATA_FETCH_THREADS=16`
+
+## Method
+
+Run by `python tools/strata_bench.py` against the HTTP API of a Strata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests.
+
+- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs
+- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each
+- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`)
+- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests
+
+How it was run on this PC; neither changes what is measured:
+
+- The server was started separately with the release's `serve/server.py` and the tool was pointed at it with `--url http://127.0.0.1:8080 --config <the config>`, the same way as the 0.1.45 report of this PC, so the tool did not save the engine log itself. `engine-log-excerpt.txt` was cut from the benchmark run's part of the engine log with the tool's own `log_excerpt()` and `redactor()`, and local folders were replaced with placeholders.
+- The tool was run with `PYTHONUTF8=1`: on a Chinese-locale Windows it still reads `CMakeLists.txt` with the default code page (`tools/strata_bench.py`, the `project(strata VERSION ...)` lookup) and stops with `UnicodeDecodeError: 'gbk' codec` (reported in #2792).
+
+## Results
+
+| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range |
+| --- | ---: | ---: | ---: | ---: | --- | --- | --- |
+| short chat | 46 (43-51) | 0 | 100 (100-100) | 9 | not measured | 55.8 (14.6-66.8) | 0.73 (0.53-2.12) |
+| 4K prompt, fresh | 4019 (4019-4019) | 0 (0-0) | 24 cap | 3 | 87.0 (56.9-96.7) | not measured | 46.27 (41.63-70.81) |
+| 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 101.3 (66.7-105.3) | not measured | 158.67 (152.52-240.99) |
+| agent session, follow-up turns | 14601 (14581-14621) at the last turn | see below | up to 64 | 3 | see below | 5.7 (0.7-14.4) | 62.82 (41.85-80.74) |
+
+Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token.
+
+### Agent session (conversation cache)
+
+Cache hit over the follow-up turns: 69.5 % (69.4-69.5); earlier-turn tokens read again per session: 73 (73-144) (earlier turns that the cache did not hold; the 13729 (13729-13730) new tokens of the files are read once); first turn (nothing cached) first token: 24.91 s (24.12-29.21).
+
+| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s |
+| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| 0 | 609 | 0 | 609 | 20.9 | 29.207 | 9.8 |
+| 1 | 4109 | 641 | 3468 | 55.3 | 62.821 | 9.4 |
+| 2 | 5642 | 4173 | 1469 | 31.4 | 46.985 | 2.0 |
+| 3 | 9075 | 5635 | 3440 | 64.5 | 53.442 | 0.7 |
+| 4 | 11828 | 9120 | 2708 | 52.0 | 52.172 | 1.2 |
+| 5 | 14601 | 11884 | 2717 | 34.1 | 80.738 | 2.5 |
+
+Per-request data: `results.json`. Engine log lines from the benchmark run: `engine-log-excerpt.txt` (see Method).
+
+## Correctness and limitations
+
+Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 2161 s.
+
+## Notes
+
+- **Against 0.1.45 on this PC** (the report this one replaces, same settings and tool, 2026-10-18): 4K prompt 58.3 tok/s, 16K prompt 127.5 tok/s, agent follow-up turn first token 48.2 s, agent decode 6.4 tok/s. On 0.2.3.1 the 4K median is higher (87.0) but the ranges overlap (56.9-96.7 against 51.7-98.9); the 16K prompt is slower (101.3, range 66.7-105.3 against 124.3-173.4); the agent turns are not faster.
+- **Low RAM and a slower drive.** The RTX 5070 Ti report with 31.6 GiB of RAM (#2791, 0.1.44.1, Windows 11, WD SN770) reads a 4K prompt at 270 tok/s and a 16K prompt at 448 tok/s. The GPU (16 GB against 12 GB) and the drive differ, so this run alone does not say which one matters more.
+- **What earlier measurements on this PC showed** (0.1.39, with Windows performance counters during a 20K-token prompt): the drive read only 0.2-0.5 GB/s while about 300 MB of RAM stayed available, so the prompt path was limited by page faults on the mapped `experts.bin`, not by the drive's speed. Those pages also push the decode's cached experts out of RAM, which matches the slow decode in the agent turns (5.7 tok/s). Opt-in changes for this setup and their measurements: #3065.
+- **VRAM.** With these settings the engine logs 337 MiB of VRAM free with everything loaded, and warns that the card is nearly full; the suite completed without an error.
+- **The short-chat decode** (median 55.8 tok/s) includes the first run right after the warm-up (14.6-21.0 tok/s): in low-RAM mode the first requests after a start read many experts from the drive.
+- Not tested here: a context above 32K (`--long`), `--clients`, a RAM budget (`--resident-budget-gib`), the release package's own engine binary, other models.

bench/results/2026-10-23-community-rtx-5070-qwen3-8-flash-next-iq2-xs/engine-log-excerpt.txt

@@ -0,0 +1,80 @@
+[strata] 2026-10-23 17:55:24 engine started: <engine>\strata.exe --serve --pack <models>\packs\iq2_xs --native <models>\models\IQ2_XS\Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf --ple-gguf <models>\models\IQ2_XS\Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00002-of-00002.gguf --expert-profile <strata-0.1.45>\data\expert-profile.bin --expert-cache auto --prefill auto --spec 4 --mtp <models>\mtp\rt --max-context 32768 --kv int8 --mmap-experts --pcie-frac 0.55 --spec-min-p 0.70
+strata generate: profile <strata-0.1.45>\data\expert-profile.bin: 24576 ranked pairs, built for 24576 slots
+strata mtp: draft layer loaded, 839 MiB of VRAM (experts 675, dense 111), files read in 0.24 s (3307 MiB/s)
+strata generate: GPU 0: NVIDIA GeForce RTX 5070, compute capability 12.0
+strata generate: expert cache auto: 6.03 GiB free, 700 MiB reserved (+143 MiB for the draft head) -> 3701 slots
+strata generate: only 635 MiB free once the slots are written (reserve 700 MiB); shrinking the expert cache
+strata generate: expert cache 3780 slots, 5.08 GiB of VRAM; policy is
+strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
+strata generate: pre-filled 3780 of 3780 slots from the profile in 1.4 s (3951 MB/s); slot 0 verified
+strata generate: R4 hit path ON - resident experts are computed on the GPU
+strata generate: token graph hit path: 3780 resident experts, decided on the device
+strata: the CPU prefill share is not armed by default: the experts it would read are 27.9 GiB of file pages and only 19.7 GiB of RAM is free to keep them. STRATA_PREFILL_CPU_SHARE=auto turns it on anyway
+strata: prefill plan: chunk 8192, ring 96 kept: file set 32.1 GiB does not fit the page cache 12.4 GiB -> about 137.7 GB read from the drive per 32768-token prompt (4 chunks); no option does better (the set cannot fit: fewest passes win)
+strata: prefill plan inputs: 33.0 GiB experts, 5.1 GiB in the GPU caches, 0.0 GiB RAM copy (0 lend slots kept), file tier 27.9 GiB; 12.4 GiB available for the page cache; PCIe 12.0 GB/s; drive 1.50 GB/s (assumed); a 32768-token prompt
+strata serve: the prompt path borrows 2816 CUDA0 cache slots (3.77 GiB)
+strata mtp: draft head over 106299 tokens (137.9 MiB)
+strata serve: 337 MiB of VRAM free with everything loaded
+strata serve: that is little room for the verify windows' buffers under WDDM; if the engine exits right after this line, --prefill auto:8192 (a smaller prompt chunk borrows fewer cache slots) or a larger --vram-reserve-mib leaves more
+strata serve: WARNING: only 337 MiB of VRAM free at the start of this request (below 1024 MiB): when the card is this full the driver can page GPU memory to system RAM, decode falls to a few tok/s and a window may never finish (#2065). Add --vram-reserve-mib 1408 to the config's args, or lower --max-context / --batch
+strata verify: capturing the 6-token window (337 MiB of VRAM free)
+strata verify: capturing the 3-token window (327 MiB of VRAM free)
+strata verify: capturing the 1-token window (311 MiB of VRAM free)
+strata verify: capturing the 4-token window (293 MiB of VRAM free)
+strata verify: capturing the 2-token window (277 MiB of VRAM free)
+strata serve: decode expert cache hit rate: 37.7% (4035 hits / 10716 lookups)
+strata serve: expert tiers: GPU 4035 hits this request; since the start RAM 0 blobs, files 0 blobs 18633.4 MB read
+strata serve: decode expert cache hit rate: 60.1% (235 hits / 391 lookups)
+strata serve: expert tiers: GPU 235 hits this request; since the start RAM 0 blobs, files 0 blobs 30085.1 MB read
+strata serve: decode expert cache hit rate: 60.6% (238 hits / 393 lookups)
+strata serve: expert tiers: GPU 238 hits this request; since the start RAM 0 blobs, files 0 blobs 44342.5 MB read
+strata serve: decode expert cache hit rate: 57.8% (1882 hits / 3257 lookups)
+strata serve: expert tiers: GPU 1882 hits this request; since the start RAM 0 blobs, files 0 blobs 60129.2 MB read
+strata serve: decode expert cache hit rate: 58.8% (26220 hits / 44590 lookups)
+strata serve: expert tiers: GPU 26220 hits this request; since the start RAM 0 blobs, files 0 blobs 88506.5 MB read
+strata serve: decode expert cache hit rate: 69.4% (35483 hits / 51145 lookups)
+strata serve: expert tiers: GPU 35483 hits this request; since the start RAM 0 blobs, files 0 blobs 115436.9 MB read
+strata serve: decode expert cache hit rate: 61.5% (30476 hits / 49527 lookups)
+strata serve: expert tiers: GPU 30476 hits this request; since the start RAM 0 blobs, files 0 blobs 145735.1 MB read
+strata serve: decode expert cache hit rate: 52.9% (22663 hits / 42805 lookups)
+strata serve: expert tiers: GPU 22663 hits this request; since the start RAM 0 blobs, files 0 blobs 175808.6 MB read
+strata serve: decode expert cache hit rate: 73.6% (37309 hits / 50680 lookups)
+strata serve: expert tiers: GPU 37309 hits this request; since the start RAM 0 blobs, files 0 blobs 199174.6 MB read
+strata serve: decode expert cache hit rate: 69.1% (34363 hits / 49740 lookups)
+strata serve: expert tiers: GPU 34363 hits this request; since the start RAM 0 blobs, files 0 blobs 226448.7 MB read
+strata serve: decode expert cache hit rate: 52.1% (22955 hits / 44057 lookups)
+strata serve: expert tiers: GPU 22955 hits this request; since the start RAM 0 blobs, files 0 blobs 257122.2 MB read
+strata serve: decode expert cache hit rate: 74.6% (39320 hits / 52689 lookups)
+strata serve: expert tiers: GPU 39320 hits this request; since the start RAM 0 blobs, files 0 blobs 280624.7 MB read
+strata serve: decode expert cache hit rate: 67.5% (35337 hits / 52346 lookups)
+strata serve: expert tiers: GPU 35337 hits this request; since the start RAM 0 blobs, files 0 blobs 308138.4 MB read
+strata serve: decode expert cache hit rate: 59.5% (8574 hits / 14402 lookups)
+strata serve: expert tiers: GPU 8574 hits this request; since the start RAM 0 blobs, files 0 blobs 350392.2 MB read
+strata serve: decode expert cache hit rate: 74.7% (9243 hits / 12366 lookups)
+strata serve: expert tiers: GPU 9243 hits this request; since the start RAM 0 blobs, files 0 blobs 389506.3 MB read
+strata serve: decode expert cache hit rate: 83.4% (10581 hits / 12694 lookups)
+strata serve: expert tiers: GPU 10581 hits this request; since the start RAM 0 blobs, files 0 blobs 427433.9 MB read
+strata serve: decode expert cache hit rate: 87.5% (13598 hits / 15545 lookups)
+strata serve: expert tiers: GPU 13598 hits this request; since the start RAM 0 blobs, files 0 blobs 503008.3 MB read
+strata serve: decode expert cache hit rate: 83.9% (10014 hits / 11939 lookups)
+strata serve: expert tiers: GPU 10014 hits this request; since the start RAM 0 blobs, files 0 blobs 578166.6 MB read
+strata serve: decode expert cache hit rate: 91.4% (14442 hits / 15796 lookups)
+strata serve: expert tiers: GPU 14442 hits this request; since the start RAM 0 blobs, files 0 blobs 652834.5 MB read
+strata serve: decode expert cache hit rate: 61.0% (12640 hits / 20733 lookups)
+strata serve: expert tiers: GPU 12640 hits this request; since the start RAM 0 blobs, files 0 blobs 685675.3 MB read
+strata serve: decode expert cache hit rate: 56.0% (19786 hits / 35363 lookups)
+strata serve: expert tiers: GPU 19786 hits this request; since the start RAM 0 blobs, files 0 blobs 735612.3 MB read
+strata serve: decode expert cache hit rate: 66.6% (21043 hits / 31618 lookups)
+strata serve: expert tiers: GPU 21043 hits this request; since the start RAM 0 blobs, files 0 blobs 780461.3 MB read
+strata serve: decode expert cache hit rate: 80.8% (23560 hits / 29150 lookups)
+strata serve: expert tiers: GPU 23560 hits this request; since the start RAM 0 blobs, files 0 blobs 822446.0 MB read
+strata serve: decode expert cache hit rate: 82.8% (31203 hits / 37703 lookups)
+strata serve: expert tiers: GPU 31203 hits this request; since the start RAM 0 blobs, files 0 blobs 864264.7 MB read
+strata serve: decode expert cache hit rate: 79.7% (30497 hits / 38246 lookups)
+strata serve: expert tiers: GPU 30497 hits this request; since the start RAM 0 blobs, files 0 blobs 908571.6 MB read
+strata serve: decode expert cache hit rate: 59.5% (9262 hits / 15572 lookups)
+strata serve: expert tiers: GPU 9262 hits this request; since the start RAM 0 blobs, files 0 blobs 939876.7 MB read
+strata serve: decode expert cache hit rate: 75.7% (26768 hits / 35346 lookups)
+strata serve: expert tiers: GPU 26768 hits this request; since the start RAM 0 blobs, files 0 blobs 984763.0 MB read
+strata serve: decode expert cache hit rate: 71.2% (23485 hits / 32986 lookups)
+strata serve: expert tiers: GPU 23485 hits this request; since the start RAM 0 blobs, files 0 blobs 1029102.6 MB read