Repository navigation
Community benchmark 2026-10-11: RTX 5070 Ti (PCIe x8 dock) and RX 7900 XTX (Windows, eGPU over USB4) - 0.1.43, A/B of the new defaults and the #1630 prefill cap #1977
Description
Community benchmarks: RTX 5070 Ti (PCIe x8 dock) and RX 7900 XTX (Windows, eGPU over USB4) — engine 0.1.43, with A/B runs
Four tools/strata_bench.py --long runs on one machine on 2026-10-11, engine 0.1.43, each arm three runs, medians with ranges. Two installations sharing one host, one card at a time. Both cards are in docks, and the link widths matter — flagged per COMMUNITY_BENCHMARKS.md.
The machine
- Beelink GTi14 mini PC: Intel Core Ultra 9 185H (6P+8E, 22 threads, AVX2, no AVX-512), 96 GB DDR5, NVMe, Windows 11 (10.0.26300).
- RTX 5070 Ti 16 GB in the Beelink EX Pro expansion dock — PCIe 5.0 x8 (the dock's link width by design; engine probe 25.5 GB/s host-to-device). Drives four displays (~2.6 GB VRAM held by the desktop). Driver 617.42, CUDA 13.0 build.
- RX 7900 XTX 24 GB (gfx1100) in an AOOSTAR AG02 dock (ASM2464PDX) on a Thunderbolt 4 / USB4 port, x16 slot inside the dock. No displays on it. Driver 32.0.31041.1004, HIP build (bundled ROCm 10.2.0a20260930, hipBLASLt 100500 with the gfx1100 tuning table).
- Model: Qwen3.8-Flash-Next IQ3_XXS, 262144-token context, one slot. Sources: upstream
v0.1.43(7695859a) + a local config layer (deviations tabled in each README); no engine source changes.
1. RTX 5070 Ti (Windows): the 0.1.42/43 speed defaults measure larger here than the published tail-skip figure
Baseline (defaults on: tail skip 7, measured PCIe share, CPU prompt share) vs the same day with all three opted out (STRATA_ROUTE_TAIL_SKIP=0, STRATA_PCIE_FRAC_DEFAULT=old, STRATA_PREFILL_CPU_SHARE=0):
| Metric | defaults on | defaults off | change |
|---|---|---|---|
| chat decode tok/s | 63.0 (52.4-68.6) | 48.8 (38.3-58.4) | −22.5% when opted out |
| chat first token (s) | 0.60 | 0.75 | +25.3% slower when opted out |
| agent decode tok/s | 72.3 (43.0-94.0) | 57.7 (30.8-74.7) | −20.2% when opted out |
| 4K / 16K / 32K prompt tok/s | 1614.7 / 2383.0 / 2384.8 | 1607.9 / 2452.8 / 2429.4 | within noise (±3%) |
The defaults are worth +29% chat decode and +25% agent decode on this card, with the prompt side unchanged. The per-run ranges overlap (chat 52.4-68.6 against 38.3-58.4), so the claim rests on the medians and on the direction being the same in every pairing: all 9 defaults chat runs exceed the opted-out arm's median, 6 of 9 exceed its best run, and no run of the opted-out arm exceeds the best defaults run. Published +13-16% was tail skip alone; the three defaults together measure larger. Attribution per default was not done (one policy switch, three env vars). Answer drift not measured in these runs (speed-only suite); the published KL figures apply to the tail-skip component.
2. RX 7900 XTX (Windows, eGPU over USB4): the --prefill auto cap (#1630) costs ~24% prompt here, not 14%
Baseline with explicit --prefill 8192 vs the same day with stock auto (engine caps at 6144 on Windows AMD). One variable; everything else identical:
| Metric | 8192 | auto → 6144 | change |
|---|---|---|---|
| 16K prompt tok/s | 308.6 | 236.4 | −23.4% |
| 32K prompt tok/s | 322.0 | 244.0 | −24.2% |
| 16K / 32K first token (s) | 52.1 / 99.6 | 68.0 / 131.4 | +31% / +32% |
| 4K prompt tok/s | 174.3 | 175.8 | unchanged (chunks under 6144, as expected) |
| chat decode tok/s | 45.6 | 44.3 | −2.9%, ranges overlap |
| agent decode tok/s | 74.9 | 71.9 | −4.0% |
| agent re-read tokens/session | 4 | 144 (144-145 in all 3 runs) | chunk boundaries at 6144 misalign with turn boundaries |
The cap costs more prompt speed than the published −14% (645 vs 748 tok/s, internal card) — consistent with expert streaming being the bottleneck and USB4 being narrower than internal PCIe. Decode is essentially unchanged: the smaller RAM complement at 6144 (25.86 vs 26.61 GiB, −455 experts) did not show up as a measurable decode cost.
Question: would a RAM-aware cap (or a documented "explicit 8192 is fine above ~64 GB RAM") be reasonable? This box ran 8192 since 0.1.41 without incident; commit headroom at start was 75.6 GiB of a 143.7 GiB limit. Safety on smaller-RAM Windows AMD boxes was not tested.
3. Tool note: strata_bench.py mislabels the GPU on a dual-GPU machine
The local GPU probe reads nvidia-smi first, so both AMD runs were written as "NVIDIA GeForce RTX 5070 Ti" in the report header, folder name, and results.json (machine.gpu, gpu_vram_mib, gpu_driver), even though --url pointed at the AMD server. Fixed by hand in the attached files (README headers, folder names, results.json fields; machine.engine_info was correct — it comes from the server). Suggest: when --url is used, prefer the GPU the server reports (or probe rocm-smi too).
What was not tested
- Correctness / answer drift in these runs (speed suite only).
- Per-default attribution on NVIDIA.
- Whether 8192 is safe on smaller-RAM Windows AMD boxes.
--clientsconcurrency, long soaks, Coder/Swift variants.
Full per-request data and settings tables are in the four README folders (sanitized; local paths, names and keys checked to be absent):
Activity
vitaliso commentedon Oct 12, 2026
Update 2026-10-12: the prefill ladder on the same RX 7900 XTX — 8192 → auto:16384 → auto:32768
Follow-up to §2 above. Same box, same model (Qwen3.8-Flash-Next IQ3_XXS, 262144 ctx), engine 0.1.43, one flag changed per arm; everything else is the working configuration of that install. Each arm is a separate
tools/strata_bench.py --longrun against a live server (--url), 3 runs per test, medians with ranges. The long-prompt arm uses our own harness (fresh prompt, engine restarted before every arm, ABBA order, numbers taken from the engine log, warm-up request off):coding-bench.ps1 -NoWarmup -ContextTokens 131072, ABBA 3+3, fresh 142,816-token prompts.8192 → auto:16384
auto:16384 → auto:32768
Prompt-side ranges do not overlap at any of the three steps. The +23.0% at a fresh 131K prompt matches what #1389's reporter measured at 74K.
Memory and safety per arm (engine startup lines)
Commit limit on this box is 143.7 GiB (96 GB RAM + page file); 427 MiB of VRAM free after load in all three arms. The engine still prints the #1630 warning for chunks ≥ 8192; no crash in these suites. One engine exit happened in a discarded run that had other 200K-token requests interleaved with a bench (no crash line in the log, not reproduced in the clean suites).
Reading for the #1630 cap
On this machine the 6144 cap is paid on every fresh prompt ≥ 16K, and each doubling of the chunk buys prompt speed (≈3 GiB commit and ≈1 GiB resident RAM per doubling, lent slots ≈×2) while decode stays inside noise. Lend coverage stayed full (
N of N) at every step, so the #1389 failure mode (partial coverage → experts read from the pack during the prompt) did not appear here. That suggests framing the cap by available RAM and commit rather than a fixed 6144 — for example allow 16384/32768 when commit headroom and lend coverage allow it, and keep the cap for small-RAM boxes.Not tested
--parallel) serving, where lent slots would contend with a concurrent decode.Per-request data and settings tables in the four attached files (sanitized: local paths removed; GPU label corrected by hand per §3 of this issue):
--prefill 8192: https://gist.github.com/vitaliso/5d6755c5771571216a8fed5427307f17--prefill auto:16384: https://gist.github.com/vitaliso/d80e9342f0e6990b90dcca9be0046ed4--prefill auto:32768: https://gist.github.com/vitaliso/81eccb2df03a3c2ade348a8cff1a1770Niko1221 commentedon Oct 19, 2026
Thanks for taking the time to report this.
We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to read properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.
To reopen it takes about two minutes:
/reopen. A check runs and either reopens the issue or lists what is still missing.The rules are in CONTRIBUTING.md.