feat(bench): add reproducible serving performance harness

[tuxevil]

Summary

• Add benchmarks/bench_serving.py, a reproducible client-wall performance harness for an already-running FreeToken server.
• Calibrate prompt sizes through /v1/messages/count_tokens.
• Measure streamed /v1/chat/completions TTFT, decode throughput, effective prefill throughput, and total latency.
• Capture model, server, hardware, KV/MoE/Mamba cache geometry, MoE residency, and cache budget.
• Benchmark fresh, identical-prefix, and small-suffix prefix-cache behavior.
• Add CPU-only unit tests and usage documentation.

Motivation

FreeToken currently has serving benchmarks, but no small, generic harness that clearly separates client-observed performance from engine-internal telemetry.

This harness records measurements at the HTTP client boundary. Fields prefixed with client_wall_ include queueing, prefill, scheduling, detokenization, and SSE delivery.

Design

The harness:

1. Validates that the server is ready.
2. Collects metadata from /health, /v1/models, /v1/cache/status, and /v1/stats.
3. Performs configurable warmup runs.
4. Calibrates deterministic prompts to approximately 512, 1024, 2048, and 4096 tokens.
5. Measures fresh prefill performance.
6. Measures steady-state single-stream decode performance.
7. Measures prefix-cache reuse through an ordered fresh → identical → small_suffix sequence.
8. Writes machine-readable schema version 1 JSON and prints a human-readable summary.

The benchmark uses explicit greedy sampling with thinking disabled so runtime measurements are not affected by reasoning nondeterminism.

Decode throughput follows the existing serving benchmark convention:

(completion_tokens - 1) / (last_token_time - first_token_time)

Scope

This first version intentionally:

• connects to an existing server;
• measures one serving configuration at a time;
• uses standard-library HTTP/SSE handling;
• does not add new server endpoints;
• does not start or stop the server;
• does not run concurrency or backend sweeps;
• does not parse engine logs;
• does not perform behavioral evaluation.

These capabilities can build on the schema in later PRs.

Validation

• PYTHONPATH=python:. uv run --no-project --with pytest pytest -q tests/benchmarks/test_bench_serving.py

  • 13 tests passed

• uv run --no-project --with ruff ruff check benchmarks/bench_serving.py tests/benchmarks/test_bench_serving.py

• uv run --no-project --with ruff ruff format --check benchmarks/bench_serving.py tests/benchmarks/test_bench_serving.py

• python -m py_compile benchmarks/bench_serving.py

• git diff --check

The full repository test suite was not collected in this environment because FreeToken's runtime dependencies, including PyTorch, are not installed.

[earlvanze]

We exercised this benchmark shape on RadixArk/Qwen3.8-Flash-Next-NVFP4 and found a useful follow-up for the harness: a concurrency sweep that makes admission and queueing explicit.

In our RTX 5090 run, c8 delivered 115.249 aggregate output tok/s and 32.428 median steady decode tok/s/request. A separate c16 run with both --max-running-requests=16 and --cuda-graph-max-bs=16 admitted all 16 requests and reached 123.079 aggregate output tok/s, but median steady decode fell to 11.614 tok/s/request. Without recording the active-request limit, it is easy to confuse a true c16 run with c16 offered to an eight-slot worker, where half the requests queue. Aggregate throughput alone also hides the per-session latency collapse.

Would you be open to an extension that adds configurable concurrency levels and records, per level:

• offered concurrency and configured active-request/CUDA-graph limits
• admitted versus queued requests, where observable
• total prompt and completion tokens plus batch wall time
• client-wall effective prefill throughput
• aggregate delivered output throughput
• median/p95 TTFT and per-request steady decode throughput
• machine-readable JSON results using the existing schema conventions

We can prepare the patch against this branch if that scope fits your intent for #341. Published evidence and the exact c1/c2/c4/c8/c16 records are here:
https://tekiz.ai/benchmark-data/qwen38-flash-next-rtx5090-nj-20260920.json

The c16 prefill figure in that artifact is explicitly client-wall/effective rather than engine-internal telemetry, consistent with this PR's measurement boundary.

[tuxevil]

Thanks — this is exactly the kind of follow-up the schema was intended to support.

The c8/c16 results make the case clearly: aggregate throughput alone hides the per-session latency collapse, and without recording the active-request and CUDA graph limits it is easy to mislabel an offered concurrency level as an admitted one.

I would welcome a focused concurrency extension built on top of this harness. The fields you listed fit well with the current measurement boundary, especially:

• offered versus admitted concurrency;
• max_running_requests and cuda_graph_max_bs;
• queued requests where the server exposes enough information to distinguish them;
• batch wall time and total prompt/completion tokens;
• aggregate client-wall output throughput;
• per-request TTFT p50/p95 and steady-state decode throughput.

My preference would be to keep #341 focused on the single-stream serving protocol and schema foundation, then implement the concurrency sweep as a follow-up PR based on this branch. That should let us settle the admission/queue semantics explicitly instead of making them implicit in the first version.

The RTX 5090 data is a very useful reference case. In particular, the contrast between c8 and the fully admitted c16 run is exactly the kind of result this harness should make easy to reproduce and compare.