Proposal: reproducible end-to-end benchmark / regression harness #281

Activity

1. tuxevil commented on Aug 29, 2026

   

   tuxevil

   on Aug 29, 2026

   Author

   More actions

   A useful external baseline came out of the same testing, and I think it strengthens the case for keeping prefill and decode as separate first-class measurements in the proposed harness.

   I ran the same Qwen3.6-35B-A3B family through Ollama/llama.cpp on the same machine:

   • Quadro RTX 4000, 8 GB, sm_75
   • Ryzen 9 7945HX
   • 96 GB DDR5
   • Ollama qwen3.6:35b
   • quantization reported by Ollama: Q4_K_M
   • model placement: approximately 74% CPU / 26% GPU

   I used a deterministic needle-retrieval prompt containing 3,226 input tokens and requesting only a short identifier as the answer.

   Ollama reported:

   prompt eval count:     3226
   prompt eval duration:  10.485 s
   prompt eval rate:      307.67 tok/s
   
   eval count:            175
   eval duration:         11.225 s
   eval rate:             15.59 tok/s
   

   The retrieval itself was correct.

   For comparison, the FreeToken context sweep on the same hardware used a very similar ~3.3K-token prompt:

   runtime / quant	prompt tokens	prefill	decode
   Ollama Q4\_K\_M, CPU+GPU	3226	307.67 tok/s	15.59 tok/s
   FreeToken Q4\_K offload	3295	188.10 tok/s	39.45 tok/s
   FreeToken IQ3\_S offload	3295	138.91 tok/s	45.44 tok/s

   The GGUFs are not identical (Q4_K_M vs Q4_K), so I would not treat this as a strict engine-vs-engine benchmark.

   However, the magnitude and direction of the result are interesting:

   • Ollama/llama.cpp Q4_K_M was ~1.64x faster than FreeToken Q4_K at fresh-context prefill.
   • FreeToken Q4_K was ~2.53x faster at autoregressive decode.
   • FreeToken IQ3_S was ~2.91x faster at decode.

   The Ollama decode number also appears stable: an earlier long-generation run on the same CPU/GPU placement generated 4,055 tokens at 16.07 tok/s, versus 15.59 tok/s here.

   I also forced Ollama CPU-only for the same 3,226-token prompt:

   prefill: 89.40 tok/s
   decode:   7.52 tok/s
   

   So partial GPU placement helps llama.cpp substantially, but it still produces a very different performance profile from FreeToken expert offload.

   This seems like a good concrete example of why a single aggregate "tokens/sec" result is insufficient for comparing serving configurations.

   Depending on the workload, the conclusion can reverse:

   • large fresh prompt + very short answer -> the faster prefill path can dominate;
   • moderate/long generation -> FreeToken's much faster decode quickly dominates;
   • cached-prefix workloads introduce a third performance regime again.

   It also suggests that an optional external-baseline mode could eventually be useful in a community benchmark format — not necessarily something FreeToken itself has to run, but a JSON schema capable of representing measurements from another engine would make reports much easier to compare.

   One additional observation: during Ollama autoregressive decode I observed the RTX 4000 staying below roughly 50 W in nvidia-smi, whereas FreeToken offload in my earlier runs held the GPU near full utilization and roughly 116-119 W. I did not collect synchronized telemetry for the Ollama run, so I would treat that as qualitative rather than benchmark data, but it is consistent with the very different decode behavior.

   For me this external baseline reinforced that the proposed harness should keep at least these measurements independent:

   1. cold/fresh-context prefill throughput,
   2. autoregressive decode throughput,
   3. cached-prefix TTFT,
   4. total request latency.

   Collapsing them into one throughput number would hide exactly the trade-off seen here.
2. benwilson commented on Aug 31, 2026

   

   benwilson

   on Aug 31, 2026

   More actions

   +1, and one requirement such a harness should build in from the start: greedy decoding is not reproducible when thinking is on. Three identical temperature: 0 requests produced three different reasoning traces (206 / 218 / 288 completion tokens); the first divergence was 210 characters in, on a near-tie ("Function reverse linked list" vs "Function to reverse linked list"). The same prompt with chat_template_kwargs: {"enable_thinking": false} was byte-identical across runs (91 tokens each), and explicit temperature/top_k/seed are honoured over --sampling-defaults model (omitting them samples, as documented).

   Environment: FreeToken git 4b94bdc (HEAD 2026-08-30, no local patches), installed with pip install --no-build-isolation "freetoken[accel] @ git+…@4b94bdc"; torch 2.11.0+cu130, CUDA toolkit 13.0, driver 595.71.05; RTX 3090 24 GB (sm_86) on a rented vast.ai host (503 GiB RAM); image pytorch/pytorch:2.11.0-cuda13.0-cudnn9-devel; checkpoint dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 @ be794b99 (standard compressed-tensors NVFP4 layout, byte-identical quant config to the RadixArk stock checkpoint — not the FTW repack).

   Reads like a nondeterministic kernel (atomics in the offload MoE combine, or the Triton NVFP4 path) flipping near-tied argmax picks — content-heavy reasoning text has many near-ties, peaked code output few. For a regression harness that means either thinking off, or comparing on scores rather than token hashes, and a note in the docs that temp-0 ≠ reproducible here. Happy to re-run the repro on --moe-backend hybrid or another backend if that helps localise it.
3. benwilson commented on Aug 31, 2026

   

   benwilson

   on Aug 31, 2026

   More actions

   A second data point for the harness question, this time the exact case in your motivation — ft bench bw recommended hybrid; here real inference agreed at short context and could not tell the difference at long context, for a reason the harness would want to measure explicitly.

   Box: RTX 3090 (sm_86) on a rented host, PCIe 4.0 x16 (bw: linear H2D 23.8 / D2H 20.1 GB/s), 32-core CPU (STREAM read 80.6 GB/s), 503 GiB RAM; FreeToken 4b94bdc, torch 2.11.0+cu130; dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 (compressed-tensors NVFP4), --nvfp4-backend triton --moe-cache-auto --max-seq-len-override 262144 --kv-reserve-tokens 262144 --max-running-requests 1.

   ft bench bw --dtype nvfp4: CPU-MoE 65.8 GB/s vs PCIe-gather 25.1 → ratio 2.62× → hybrid; overlapped 43.4 + 21.6 → 33.2% of misses fetched over PCIe.

   Probe (same on both backends; the ad-hoc thing your harness would replace): streaming /v1/chat/completions, temperature 0, enable_thinking: false, 400-token answer, a fresh nonce on every prompt so the radix cache cannot help; decode = tokens/(last − first token time).

   prompt	offload (n=3)	hybrid + profile (n=3)	hybrid, no profile (fetch cap 1)
   ~70 tokens	24.5–24.7 tok/s	31.6–31.9 (+29%)	28.3–30.5
   75.7k tokens	11.4–13.0 (median 12.6)	12.4–13.1	9.5 / 29.1 (n=2, unstable)
   TTFT @ 75.7k	~70 s (≈1,075 tok/s prefill)	~70 s	~70 s

   Two observations for the harness design:

   1. Scheduler counters and end-to-end disagree at depth. For the same 75.7k runs the Decode batch … gen throughput lines average 21.3 tok/s (offload) vs 26.5 (hybrid) — the MoE gain is real inside the batch — but the client sees ~12.5 on both. Short prompts agree within 5%. So something per decode step outside the batch grows with context and dominates at 75k; a harness that reports only the engine counter would call hybrid a win here, one that reports wall-clock would call it a wash. Both numbers are needed, and the gap itself is worth a row.
   2. The profile has to be findable or hybrid quietly degrades. Saving bw with -o and not setting FREETOKEN_BENCHBW_PATH gave --moe-hybrid-max-fetch auto: no usable ft bench bw profile … fixed fetch cap of 1 — one WARNING line, then a different backend configuration than the one benchmarked. A harness should assert which profile (if any) the serve loaded.

   Raw probe logs and the bw JSON are kept; happy to re-run under whatever protocol the harness settles on.
4. benwilson commented on Sep 1, 2026

   

   benwilson

   on Sep 1, 2026

   More actions

   Concurrency data for the harness proposed here, since the field this issue asks for (concurrent requests / throughput scaling) has no published values for the Flash-Next class that I can find — the paper's numbers and #324's are single-stream, and #257's 5090/4090 figures are --max-running-requests 1.

   Box: RTX 5090 (sm_120, 32 GB), driver 590.48.01, torch 2.11.0+cu130, PCIe 4.0 x16, 501 GiB host RAM, rented vast.ai. FreeToken 4b94bdc. RadixArk/Qwen3.8-Flash-Next-NVFP4 @ 7b71922. --moe-backend offload --moe-cache-auto --nvfp4-backend triton --max-seq-len-override 262144 --kv-reserve-tokens 262144 --max-running-requests 8 --cuda-graph-max-bs 8. Graphs captured [1,2,4,8]; moe_cache_size=2606, num_pages=4097, KV 262,208 tok = 6.19 GiB, 2.88 GiB free after capture.

   Method: N streams launched together, each with its own nonce prefix so the radix cache cannot serve one stream from another; enable_thinking:false; per-stream tok/s measured first-token→last-token so TTFT is excluded from decode; aggregate is total tokens over the whole batch's wall clock. 8/8 streams completed at every level, zero failures.

   2000-token prompts, 400-token outputs:

   concurrency	per-stream tok/s	aggregate tok/s	TTFT s
   1	27.5	18.8	6.8
   2	18.5	27.2	7.7
   4	9.6	32.3	8.0
   8	6.1	39.9	15.1

   Aggregate keeps climbing to 8 (2.1×) but per-stream falls 4.5× and TTFT more than doubles.

   Two methodology points I'd suggest the harness bake in, both learned by getting them wrong first:

   1. Output length has to be long enough for decode to dominate, or the curve flatters badly. The identical sweep with 300-token prompts / 23-token outputs read 22.5 → 18.3 tok/s per stream — a −19% cost at 8× concurrency, versus the −78% above. Short outputs are TTFT-dominated and barely decode.
   2. Warm every concurrency level before measuring. CuTe kernels are shape-specialised, so the first request at each batch size pays its own JIT. On the b12x backend the first request took 2m20s wall for 10 output tokens (input throughput (token/s): 0.18 on a 28-token prompt). A harness that measures cold will report compilation as inference.

   Also relevant to scope: I could not get --nvfp4-backend flashinfer (b12x) to serve more than one request at all on this card — filed as #335 — so any benchmark matrix should record the NVFP4 backend alongside the MoE backend, since on sm_120 they are not interchangeable.
5. tuxevil commented on Sep 2, 2026

   

   tuxevil

   on Sep 2, 2026

   Author

   More actions

   Thanks — these are very useful data points, and they actually help narrow the scope rather than expand it.

   A few things I would now consider requirements for the first performance harness:

   • report client-wall TTFT / decode throughput separately from the engine's internal scheduler throughput;
   • explicitly distinguish cold prefill, cached-prefix reuse, and decode;
   • warm each configuration / concurrency level before measuring, so JIT/compilation is not counted as inference;
   • record the actual MoE/backend configuration in effect, including whether a bench bw profile was found and used;
   • avoid treating temperature: 0 + thinking as byte-level deterministic. Exact-output regression cases should probably run with thinking disabled, while reasoning-enabled cases should use semantic/protocol invariants instead.

   The long-context result is especially interesting. I just ran into a related effect on the 8 GB Turing box from the original post: increasing Qwen3.6 KV capacity from ~32k to ~64k kept decode reasonably healthy, but reduced expert-cache residency enough that fresh-context TTFT for a ~12k prompt went from roughly tens of seconds to ~100 s. Cached-prefix turns were a completely different regime again.

   That reinforces the idea that context length, expert-cache residency, TTFT, and decode throughput need to be recorded together — a single tok/s number can give a very misleading picture.

   For the first PR I would still keep the scope small: performance/serving measurements + machine-readable output. Concurrency and behavioral/API regression can build on the same schema afterwards.

   Thanks for running these — this is exactly the kind of independent hardware data I hoped the harness would make easier to compare.
6. benwilson commented on Sep 2, 2026

   

   benwilson

   on Sep 2, 2026

   More actions

   Your Turing observation replicates at the other end of the hardware range, which I think makes the "record context length + residency + TTFT + decode together" requirement worth treating as non-negotiable.

   Same checkpoint and engine as my last comment, but on a 96 GB RTX PRO 6000, changing only --kv-reserve-tokens (524288 -> 2097152, i.e. 8 seats at the model's full 262,144 window):

   	524k pool	2M pool
   moe_cache_size	22,338	7,983
   per-stream @ 8 conc, 2k prompts	57.4 tok/s	43.4 tok/s
   TTFT @ 8 conc, 245k prompts	—	287 s
   TTFT, same prefix resent (cached)	—	5.7 s

   So: 4x the KV cost 64% of expert residency but only 24% of shallow throughput — and then showed up almost entirely as cold TTFT at depth, exactly the regime split you saw at 8 GB. Cached-prefix turns are a different regime again, and the log is explicit about why: Prefill batch, #new-seq: 7, #new-token: 8192, #cached-token: 751744 (~99% hit).

   One hard number for your first bullet, since it is worse than it sounds: at 8 streams x 245,867 tokens my client-side probe measured 1.4 tok/s per stream while Decode batch, #running-req: 8 ... gen throughput (token/s): 325-360 for the same requests — ~30x disagreement. The client span is first-token-to-last-token, which at depth includes the GPU chunk-prefilling the other streams; it reports queueing as slow decode. Neither number is wrong, they answer different questions, and a harness that reports only one will mislead in whichever direction it picked. Worth stating in the schema which one a field is.

   Realistic agentic shape on the same box, 8 sessions with their own ~110k contexts, 3 turns each: TTFT 139 s cold (234 s worst), then 5.7 s per turn, decode ~30-36 tok/s per session from the scheduler lines. The cold fill is a once-per-session admission cost bounded by ~3,300 tok/s aggregate prefill.

   Agreed on keeping the first PR to performance + machine-readable output.
7. tuxevil commented on Sep 2, 2026

   

   tuxevil

   on Sep 2, 2026

   Author

   More actions

   That replication across such different hardware is very useful. I think it settles two schema decisions for the first pass:

   • distinguish client_wall_* measurements explicitly from engine_* scheduler telemetry rather than calling either simply “decode throughput”;
   • record KV capacity, MoE residency and prefix-cache state alongside TTFT, since they clearly describe different serving regimes rather than just tuning details.

   The ~30x client/engine divergence at deep concurrent context is a particularly strong example of why the field semantics need to be explicit.

   And the 287 s cold -> 5.7 s cached TTFT result makes me think cold admission and cached continuation should probably be separate benchmark cases rather than statistics of the same case.

   I think we have enough methodology convergence now to stop growing the issue and turn the performance portion into a small first PR.