[noon-at-cgn]

Title

Issue: Related: none (results-only PR, no engine change). Nearby: #1598, #1599, #1600, #1601, #1614, #1636, #1637 (my open PRs, listed below), #1190 (resident split lend regions, by another contributor), #1253 (batch MTP on a layer split, another contributor; same design as #1636), #1011 (shared KV pool, closed by the 2026-10-06 history rewrite).

Summary

I run Qwen3.8-Flash-Next at UD-Q4_K_XL (104 GB) on a used server I built for about $2,000. A 2016 Xeon and two RTX 3080 20 GB cards read a 104K-token prompt at a median 2,996 tok/s and decode at a median 86 tok/s, with two 262,144-token lanes sharing one 524,288-token KV pool. One person measuring, one machine, 2026-10-08.

My build:

• Board: MACHINIST X99-MR9A PRO MAX (one socket)
• CPU: Intel Xeon E5-2696 v4 (2016), 22 cores / 44 threads, up to 3.7 GHz, 55 MB L3, AVX2, no AVX-512
• RAM: 4 x 32 GB DDR4-2400 (Hynix) = 128 GB
• GPUs: 2 x RTX 3080 20 GB, each PCIe Gen3 x16, each limited to 220 W
• Storage: Samsung 512 GB NVMe (the model lives here) and 2 x Seagate 8 TB hard disks (other containers)
• OS: Proxmox VE 9.2 host. Strata runs in an LXC with 90 GiB RAM and 24 threads
• Cost: about $2,000 for the whole build

Community harness (benchmark.py, unmodified), 3 runs per length, 256-token greedy output, median (range):

Prompt tokens	Prompt tok/s	Decode tok/s
4,096	718 (708-740)	81.2 (58.6-85.9)
25,000	2,219 (2,149-2,242)	84.0 (79.3-85.5)
51,000	2,585 (2,515-2,601)	90.6 (89.0-91.8)
103,999	2,996 (2,945-3,016)	86.3 (85.0-88.4)
127,000	3,080 (3,066-3,104)	80.7 (78.1-90.8)
two streams, aggregate (my ab.py, 12 rounds x 2 identical arms, n = 24)		89.5 (75.4-97.2)

The two screenshots in the report show peak instantaneous performance. Prefill, 3,124 tok/s at 65,536 of 104,067 tokens (peak instantaneous). Decode, 109.4 tok/s (peak instantaneous).

This does not reproduce from upstream main plus the open PRs. The engine is my fork's build on the 0.1.40.3 base (strata-w7, sha256 b1df9608...). It depends on four changes that are in neither main nor #1598-#1601: the shared KV pool (--kv-pool-tokens, my open PR #1614), --batch-mtp on a layer split (my open PR #1636) and --adapt-async beside --batch slots (my open PR #1637, stacked on #1636), and #1190 (resident split lend regions; an open PR by evaanp, not mine, not in main, currently conflicting). With a layer split and prefill borrowing, the RAM copy of the lend regions that #1190 keeps is required for the "no disk reads while serving" behaviour (start line: 5396 lent slots, all kept in RAM, 15.76 GiB). I did not measure how much the numbers depend on #1190.

To get the binary I measured, build branch repro/w7 (commit 428d9910134aaaae9ffd8c8e262cec39504221e5) on noon-at-cgn/Strata: it is upstream v0.1.40.3 plus my patch stack. Everything in it that is not in upstream main is listed in repro-w7-commits.txt in the report folder. The sha256 of my binary is from a build of the earlier ids (ac5fc3f on up-merge 9098e89); repro/w7 has the same source (empty git diff against ac5fc3f) but every commit has a new id, because the wrong author e-mail on my own commits is corrected. A rebuild from repro/w7 gives the same source, not necessarily the same binary bytes.

Configuration versus patch, as measured:

Piece	Kind	Effect on this box
--prefill auto:16384, STRATA_PREFILL_RING=384, STRATA_PREFILL_EQUAL=1	upstream options in main	the largest prefill gain: 25K / 51K / 104K reads +53% / +51% / +49% together (medians of my reads)
--ple-io direct	upstream option	speed neutral, frees RAM: the engine's locked memory went from 26.8 GiB to 0.0 GiB, and container MemAvailable from 4.6-13.8 GiB to 30.8-32.7 GiB (same pre-merge binary, 2026-10-07; readings in the README)
#1598 --aux-cpus	my open PR	decode +10% (solo 72 -> 80, two streams 82 -> 91 tok/s); prefill unaffected
#1599 STRATA_SPLIT_MTP_BATCH	my open PR	prefill +3% to +8%; decode effect not resolved (about -3% pooled)
#1601 --memory-limit-mib	my open PR	not a speed feature
#1600 lend sizing	my open PR (waits on #1190)	not needed here (reachable only with STRATA_SPLIT_RING)
#1190 resident split lend regions	open PR by evaanp, not mine, not in main	required for the "no disk reads while serving" behaviour with a layer split and prefill borrowing; not measured separately
#1614 shared KV pool, #1636 --batch-mtp on a split, #1637 --adapt-async beside slots	my open PRs	in the binary; the two-stream numbers depend on them; not measured separately on this binary (measured on 0.1.41, see Newer upstream)
the rest of my fork (77 commits not in main, plus 9 merge commits; 7 of the 77 are in main under other commit ids)	always on in the binary: the AVX2 Q8_K quantizer (also in upstream), the O(n) window plan (#1181), the flag-B fold (a48f52c/1194892 on repro/w7, earlier ids 530afec/d68cf5e; changes the captured window graph at --pcie-frac 0.1, STRATA_VERIFY_FLAGB=1 restores the old wait), per-slot vision tables (#1242), the parking fixes (#1163 and c55444d = #1503, earlier id 460b0a7), the serve give-way rule (#1288)	not measured separately; I did not measure the effect of the flag-B fold and of the window plan separately

All seven of my open PRs (#1598-#1601, #1614, #1636, #1637) are in this table.

What changed

• One report folder, bench/results/2026-10-08-community-2x-rtx3080-20gb-ud-q4-k-xl-512k-pool/: README.md (my build, model and exact command, method, results, the peak screenshots, 0.1.41 numbers, what did not help), benchmark.py (the unmodified harness from 2026-09-30-community-rtx-5090), run_harness.py (adds the API-key header, since the harness has no option for one), ab/ (my own two-stream script and its output), data/ (results, summary, telemetry, small request records, the 0.1.41 probe output in w10-0.1.41/), repro-w7-commits.txt (every commit of repro/w7 that is not in v0.1.40.3, with both ids), start-command.txt, strata-q4xl-prod.json, BUILD.json, machine.txt, model-provenance.json, images/, TRIMMED.md.
• bench/results/COMMUNITY.md: one row for the new folder.
• No engine, server or test change.

Extra Notes

Source and patches. All branches are on https://github.com/noon-at-cgn/Strata. Production source: repro/w7 428d991 (= up-merge 9098e89, upstream v0.1.40.3 d5ea713 merged into my stack, plus the opt/04 commit ac5fc3f; the binary does not embed its commit, the match to those earlier ids is by my build notes and by time). The binary has the 0.1.40.x-based branches of the patches that are now my open PRs (the PRs are written against 0.1.41), with merge results of those branches as text merges against upstream main fb58e0d, nothing compiled:

• shared KV pool, [Engine] shared KV pool for all lanes (--kv-pool-tokens): 256k lane moves 10x faster, 3 GiB less pinned RAM #1614 (pr/kv-pool-141, head 400fc50): binary has kv-shared-pool 9c533de (b346dd6 on repro/w7), copy pr/kv-pool 3aefa11; base 82f46a8, 13 commits, 393 commits behind main, conflicts only in code that upstream already has
• batch MTP on a layer split, batch-mtp: work with a layer split, +15% two-stream decode, one group under --batch-groups auto #1636 (pr/batch-mtp-141, head 2c08077; same design as open Enable serial multi-GPU batch MTP #1253 by ilumn): binary has batch-mtp-split 384d67e (0aec553), copy pr/batch-mtp-split 484b49c, with pr/mtp-shared-draft-head 08ef4a3; base 82f46a8, 3 commits, 393 commits behind main, merges clean
• --adapt-async beside --batch slots, batch: the adaptive expert tier keeps adapting in batch windows (--adapt-async 1 beside slots) #1637 (pr/adapt-async-141, head 4dba66a; stacked on batch-mtp: work with a layer split, +15% two-stream decode, one group under --batch-groups auto #1636): binary has batch-adapt 8d0c106 (7c011e0), commits 347420d (822a0d3) and 3e9cecb (c2ce009); base 82f46a8, 393 commits behind main; 347420d applies cleanly, 3e9cecb conflicts with upstream 13eee45

Newer upstream: 0.1.41. My numbers are on engine 0.1.40.3 plus my patches (v0.1.40.3, d5ea713). Upstream main is now 0.1.41 and has 128 more commits than up-merge. I built my stack on 0.1.41 as branch up-141 (head 10baecb, code head db925c1, notes in docs/UP141_NOTES.md on that branch) and ran it as strata-w10 on the same machine with the same production config. With my own probes (read96, ab.py), not the harness, and compared with strata-w7 from the same probes (medians of two restarts; strata-w10 is one start): prompt reads 25K 2,048 vs 2,128.5, 51K 2,386 vs 2,553, 104K 2,706 vs 2,940 tok/s (3.8%, 6.6%, 8.0% lower; I do not know why). Decode: one stream 79.6 and 78.7 vs 79.7 and 80.4 tok/s, two streams 88.3 and 86.9 vs 90.8 and 90.7 tok/s; the two strata-w10 values are the two identical arms of one run, so the restart-to-restart spread of up to about 5% is not covered and I do not claim a decode change. On 0.1.41, --batch-groups auto is on by default for a layer split with --batch above 1, which runs two pipelined groups of one slot where batch MTP does not draft (upstream issue #1413); my up-141 commit db925c1 makes auto resolve to 1 when --batch-mtp or --adapt-async 1 is given (start log: batch-groups auto: 1 group of 2 slots). #1253 is similar batch-MTP-on-a-split work by another contributor; my open PR for it is #1636.

More strata-w10 measurements from later the same day (two slots, greedy ab.py prompts, same binary, only the named flags differ; "pipelined groups" is strata-w10 started with the flags upstream 0.1.41 would pick, not the upstream binary), medians: two-stream decode with batch MTP 89.2 tok/s (3 restarts) against 77.3 without it (2 restarts) and 77.7 for the pipelined groups (1 restart). The pipelined groups read a long prompt beside a running decode 2.06x faster than my serial batch windows (2,508 against 1,217 tok/s). With the adaptive tier, two-stream decode is 89.2 tok/s with the asynchronous tier, 74.7 with the blocking tier and 45.2 with the tier off. The shared KV pool pins 3.1 GiB less RAM at two lanes (72.0 against 75.1 GiB), and two-stream decode is 89.2 with the pool and 87.7 without it, inside the restart-to-restart spread.

What did not help (same day, same box): a 280 W cap (+2% prefill, decode 0), STRATA_STAGER_SLEEP=0, STRATA_AUX_STAGER=0, STRATA_MMVQ_IL=0, deeper PLE I/O, ring 512, layer split 24 (-3%), --prefill auto:32768 (+6% at 104K but the RAM is too tight), pipelined windows and two-lane pipelining.

Method. Run on the machine that hosts the engine, against the production engine that was already running (not restarted, status idle for 32 minutes before the first request). The two-stream and one-stream numbers come from ab.py, my script, not the harness; its two identical arms are an A/A control, and the one-stream pair differs by 4%, which is the method's noise.

Prepared with an AI assistant (Claude Code) from the raw measurements; the numbers are the engine's and the client's own output, unedited.

🤖 Generated with Claude Code

[Niko1221]

Thank you, in v0.1.42 with your commits squashed into one, and a row in bench/results/COMMUNITY.md. One line of the README about how it was prepared was dropped; the numbers are yours.