Skip to content

bench/results: RTX 5070 Ti (16 GiB) with 31.6 GiB of RAM — IQ2_XS at 128K in low-RAM mode - #2791

Closed
Lithivm wants to merge 1 commit into
Niko1221:mainfrom
Lithivm:bench/rtx5070ti-32gb-iq2-xs
Closed

Lithivm wants to merge 1 commit into
Niko1221:mainfrom
Lithivm:bench/rtx5070ti-32gb-iq2-xs

Conversation

@Lithivm

@Lithivm Lithivm commented Oct 16, 2026

Copy link
Copy Markdown

Adds a results-only report for an RTX 5070 Ti (16 GiB) with 31.6 GiB of system RAM — the same GPU as 2026-10-04-community-rtx-5070ti-windows and 2026-10-06-community-rtx5070ti-5900x, but at half their RAM, which in bench/results/ is a combination that is not covered yet.

Only three files under bench/results/2026-10-17-community-rtx-5070-ti-qwen3-8-flash-next-iq2-xs/ (README.md, results.json, engine-log-excerpt.txt). No engine or code changes, as the guide asks for a results-only submission.

Hardware

  • RTX 5070 Ti, 16,303 MiB VRAM, power limit 300 W (not reduced), PCIe Gen5 x16
  • Ryzen 7 9700X (8C/16T, AVX-512), 31.6 GiB RAM
  • WD_BLACK SN770 1 TB NVMe (model files and OS on the same drive)
  • Windows 11 Pro 10.0.26200, driver 616.64, CUDA 13.0
  • Not an isolated machine (proxy, Steam, ComfyUI MCP bridge, antivirus; the GPU also drives the display)

Model and configuration

  • ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF @ ed59f92, IQ2_XS, both shards with their SHA-256
  • Context 131,072, KV q4_0, KV streaming off, low-RAM mode (--mmap-experts), expert cache auto (~5,000 slots / ~6.8 GiB VRAM), --prefill auto, MTP --spec 4 --spec-min-p 0.5, --vram-reserve-mib 1600
  • Strata 0.1.44.1 / engine 0.1.44.1, release binary, zip install (not a git checkout)
  • Thinking off for the suite; the suite's requests set temperature 0

Results (medians, ranges over 3 runs)

Configuration Actual prompt tokens Runs Prompt tok/s Decode tok/s TTFT s
short chat (100 tok cap) 46 9 not measured 57.2 (36.8-87.0) 0.54 (0.42-1.06)
4K prompt, fresh 4019 3 270.0 (226.5-270.2) not measured 14.94 (14.94-17.80)
16K prompt, fresh 16058 3 447.8 (446.7-518.1) not measured 35.94 (31.07-36.03)
agent session, follow-up turns 14572 3 see report 45.2 (32.3-51.4) 12.79 (10.82-17.17)

Total suite time 420 s.

What the report is about — and its limitation

This install cannot keep the 36 GB expert arena in 31.6 GiB of RAM, so unlike the two 64 GiB 5070 Ti reports it runs the low-RAM mode and serves the experts through the OS file cache. The measured boundary:

  • decode lands at ~85-87 tok/s once the cache is warm, roughly half the 64 GiB figures (102 tok/s)
  • prompt processing is 270-448 tok/s against 1,992-3,523 tok/s at 64 GiB — 4-8x slower, because the prompt path re-reads lent experts from the SSD

So on a 32 GiB machine the prompt/TTFT side, not decode, is what decides whether the setup is usable (a 16K prompt is 36 s before the first token). The chat row's 36.8-87.0 range is the low-RAM cache warming across the nine back-to-back requests, not run-to-run noise; the report says so in the Results section.

Also recorded because the guide asks for them: storage, PCIe link, power limit and the background load are filled in, and two things the tool got wrong are corrected — it reports the OS as "Windows 10 (10.0.26100)" (actually Windows 11 Pro 10.0.26200) and its own README template asks for facts it cannot read (CUDA version, commit on a zip install).

One tool bug found while preparing this

tools/strata_bench.py:564 reads CMakeLists.txt with read_text() and no encoding, so on a Chinese/Japanese Windows (GBK code page) it raises UnicodeDecodeError: 'gbk' codec can't decode byte 0xbf in position 2 on the file's UTF-8 BOM and exits before any measurement. setup.py:4649 reads the same file with encoding="utf-8". PYTHONUTF8=1 works around it. Happy to file it as an issue with the traceback if that is more useful than fixing it in this PR — I left it out of this PR to keep it results-only.

…K, low-RAM mode

Results-only submission from tools/strata_bench.py (suite 1, 3 runs per test).

The same GPU as 2026-10-04-community-rtx-5070ti-windows and
2026-10-06-community-rtx5070ti-5900x, but on 31.6 GiB of system RAM instead of 64 GiB,
so this install cannot keep the experts in RAM and runs the low-RAM mode
(--mmap-experts) with them served through the OS file cache. Decode is about half and
prompt processing 4-8x slower than the 64 GiB figures; that boundary is what the
report measures.

No engine changes: three files under bench/results/ only.
@Niko1221

Copy link
Copy Markdown
Owner

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

  1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
  2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
  3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
  4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

@Niko1221 Niko1221 closed this Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants