bench: RTX 5090 Laptop GPU follow-up for 0.1.40.2 and 0.1.40.3

bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/BUILD.json

@@ -0,0 +1,13 @@
+{
+  "source": "local",
+  "version": "0.1.40.3",
+  "archs": [
+    120
+  ],
+  "vision": "gpu",
+  "cuda_dirs": [
+    "<cuda-13.3>\\bin"
+  ],
+  "src": "38215a61b1d7b518",
+  "vision_src": "1fb2b3fb797ec402"
+}

bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/README.md

@@ -0,0 +1,100 @@
+# Community benchmark: RTX 5090 Laptop GPU (24 GB), engine 0.1.40.3
+
+Measured in the night of 2026-10-07/08 by [wolffahrer](https://github.com/wolffahrer) on the same Windows 11 gaming
+laptop (Schenker XMG NEO 16, A25) as [2026-10-06-community-rtx-5090-laptop](../2026-10-06-community-rtx-5090-laptop/README.md)
+(engine 0.1.40.1, PR #1263). This is a follow-up: same machine, same model files, same configuration, same scripts,
+same order of requests. Only the Strata version changed. 0.1.40.2 was measured the same night with the identical
+procedure; its numbers are in a short section below.
+
+Median decode throughput with **0.1.40.3** was **138.1 tok/s at 4,096 prompt tokens, 144.4 tok/s at 32,768, and
+131.0 tok/s at 128,000**. Median prompt throughput was 1,949.7, 2,893.3 and 2,780.1 tok/s. All nine speed requests
+succeeded with zero reused tokens. Against 0.1.40.1 this is within run-to-run variation (see the comparison table).
+
+## Hardware and software
+
+Unchanged from the 0.1.40.1 entry unless noted:
+
+- NVIDIA GeForce RTX 5090 **Laptop** GPU, 24,463 MiB reported VRAM, 175 W limit (laptop Dynamic Boost). Sampled draw
+  during this run median 163.9 W, peak sample 179.4 W; GPU temperature peaked at 84 °C. PCIe Gen 4 x8 under load.
+- AMD Ryzen 9 9955HX3D, `--pool-workers 10`. 64 GB RAM (61.68 GiB usable). Samsung SSD 990 PRO 1TB for model and pack.
+- Windows 11 Home, build 26200. NVIDIA driver 610.88. CUDA toolkit 13.3.
+- Strata **v0.1.40.3**, tag commit `d5ea7133741e67743c0e886bb426c0ce8d69cf6c`, built from the release source archive
+  (sha256 `6fdca77a04d8535c598c6602990789892d999000f8b7e956a27ed0dc602b5cba`) with the unmodified installer for CUDA
+  architecture 120, GPU vision helper included. See [BUILD.json](BUILD.json).
+- Background: a WSL2 VM with an idle local agent stack; the regular inference server was stopped, so the GPU was
+  dedicated to Strata and its vision helper. Not an isolated OS.
+
+## Model and configuration
+
+Identical to the 0.1.40.1 entry: `ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF` **Q2_0**, context 131,072, INT8 KV with
+32,768 resident cells, expert cache `auto` (again **11,742 slots, 15.12 GiB VRAM**, 495 MiB VRAM free with everything
+loaded), prefill `auto` (borrows 2,082 slots), MTP `--spec 4 --spec-min-p 0.7`, `--pcie-frac 0`, `--mmap-experts`, GPU
+vision, conversation cache 6 GiB / 2 slots, `reasoning_budget_tokens: 12000`, `reasoning_loop_recovery: "stop"`.
+Complete configuration with local paths replaced by placeholders: [strata-q2_0.json](strata-q2_0.json). Startup
+choices and every request's timing are in [engine.log](engine.log).
+
+## Method
+
+Same scripts as the 0.1.40.1 entry: [benchmark.py](benchmark.py) (4K/32K/128K synthetic code-explanation prompts,
+three runs each, greedy, reasoning off, 256-token cap), the unchanged `tools/needle_bench.py` (32k/128k at depths
+10/50/90) and [coding_check.py](coding_check.py) (reasoning `low`). One warm-up request excluded. The server was healthy
+12.1 s after start. Memory and GPU state were sampled once per second by [monitor.py](monitor.py); the summary is in
+[memory-summary.json](memory-summary.json) (298 samples over 393.0 s; the raw `telemetry.jsonl` is left out to keep
+the folder small).
+
+```text
+.venv\Scripts\python.exe serve\server.py --engine strata --config strata-q2_0.json --port 18096
+.venv\Scripts\python.exe benchmark.py --root <strata> --pack <strata-data>\packs\q2_0 --url http://127.0.0.1:18096 --out results
+.venv\Scripts\python.exe tools\needle_bench.py --url http://127.0.0.1:18096 --lengths 32k,128k --depths 10,50,90 --out needles.json
+.venv\Scripts\python.exe coding_check.py --url http://127.0.0.1:18096 --out coding-check.json
+```
+
+## Results (0.1.40.3)
+
+Median **[minimum–maximum]** of three runs. Every request generated 256 tokens; no failed or cancelled requests.
+
+| Prompt tokens | Reused | Prompt tok/s | Decode tok/s | TTFT seconds | Total seconds |
+| ---: | ---: | ---: | ---: | ---: | ---: |
+| 4,096 | 0 | 1,949.7 [1,805.6–1,958.6] | 138.1 [125.5–148.5] | 2.133 [2.124–2.302] | 4.083 [3.848–4.329] |
+| 32,768 | 0 | 2,893.3 [2,888.9–2,906.8] | 144.4 [140.1–145.3] | 11.384 [11.330–11.427] | 13.145 [13.145–13.177] |
+| 128,000 | 0 | 2,780.1 [2,752.8–2,801.1] | 131.0 [123.2–140.4] | 46.246 [45.884–46.702] | 48.513 [47.827–48.540] |
+
+Raw per-run records: [results.json](results.json); aggregates: [summary.json](summary.json). Decode expert-cache hit
+rates were 90.2–98.7% for the nine speed requests. Peak GPU memory 23,221 MiB; host RAM in use (system-wide, includes
+the WSL2 VM) 12.74 GiB before load, 54.88 GiB peak; page file unchanged at 0.50 GiB.
+
+**Across versions on this laptop** (median decode / prompt tok/s; same procedure, not interleaved, one run of three
+requests per length each):
+
+| Prompt tokens | 0.1.40.1 (2026-10-06) | 0.1.40.2 (2026-10-07 23:16) | 0.1.40.3 (2026-10-08 00:50) |
+| ---: | ---: | ---: | ---: |
+| 4,096 | 136.9 / 1,952.5 | 139.3 / 1,960.0 | 138.1 / 1,949.7 |
+| 32,768 | 137.7 / 2,877.8 | 141.6 / 2,890.0 | 144.4 / 2,893.3 |
+| 128,000 | 134.6 / 2,764.1 | 137.2 / 2,764.1 | 131.0 / 2,780.1 |
+
+Decode differences of a few percent are inside the min–max ranges of single runs; prompt throughput is flat.
+We read this as "no measurable speed change on this hardware" for 0.1.40.2 and 0.1.40.3, and no regression.
+
+## Correctness and limitations
+
+- **Long-context recall:** all six needles found (prompt lengths 32,895–32,896 and 125,868–125,870 tokens). One 32K and
+  two 128K cases reused 16,384 prompt tokens. See [needles.json](needles.json).
+- **Coding check (0.1.40.3):** **10/10 tests passed**, 823 generated tokens, 7.9 s. See [coding-check.json](coding-check.json).
+- **Coding check (0.1.40.2), a note on greedy decoding:** the 0.1.40.2 run scored **0/10**. The answer text was
+  empty after 2,567 generated tokens, all of them reasoning (the raw response of that run was not kept). We re-ran the
+  check three times per version on a freshly started server and kept the raw responses: 0.1.40.2 scored 0, 10, 10 and
+  0.1.40.1 scored 0, 10, 6. Both zeros were the first request after the start, with **identical** looping reasoning in
+  both versions ("If 'IV' with invalid char? 'IV' pair. ..."), ended by the configured
+  `reasoning_loop_recovery: "stop"` (`finish_reason: length`, 2,048 tokens). The result varies between runs in both
+  versions, so we do not read the 0/10 as a 0.1.40.2 regression but as a property of greedy decoding with reasoning
+  `low` on this Q2_0 model and prompt.
+- Not evaluated: long outputs, sampled decoding, thinking speed, vision, tool use, concurrency, sustained thermal load.
+  Laptop GPU clocks can vary with temperature and power state.
+
+Besides these synthetic runs we checked both versions with our own agent task suite (multi-step tool use under a local
+agent, German prompts, 44 tasks x 3 runs per version). Pass rates were within two runs of 0.1.40.1 on every set
+(total 121, 122 and 123 of 132 runs for 0.1.40.1, 0.1.40.2 and 0.1.40.3); those tasks are private and not part of this
+report.
+
+The author documents local AI on consumer hardware (in German) on the KI SOUVERÄN channels:
+[YouTube](https://www.youtube.com/channel/UCE6Ch6g6Bo8v4ROpYDzOOxA) and [Telegram](https://t.me/lokale_ki).

bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/benchmark.py

@@ -0,0 +1,198 @@
+#!/usr/bin/env python3
+"""Serial, fresh-prompt Strata benchmark (Windows-capable).
+
+Derived from bench/results/2026-09-30-community-rtx-5090/benchmark.py: same synthetic prompts, nonces,
+token targets, greedy decoding, reasoning off and 256-token output cap, so the numbers are comparable.
+Changes: runs with Strata's own Windows venv; a failed or cancelled run is recorded with its error and
+excluded from the statistics instead of stopping the suite; peak host RAM and GPU memory are sampled
+every 0.5 s during each request.
+
+    .venv\\Scripts\\python.exe benchmark.py --root . --pack <data-dir>\\packs\\q2_0 --url http://127.0.0.1:18096 --out results
+"""
+import argparse
+import hashlib
+import json
+import statistics
+import subprocess
+import sys
+import threading
+import time
+import urllib.request
+from pathlib import Path
+
+import psutil
+
+
+def get(url):
+    with urllib.request.urlopen(url, timeout=30) as response:
+        return json.load(response)
+
+
+class Peak:
+    """Samples system-wide host RAM in use (total - available) and GPU memory used while a request runs."""
+
+    def __init__(self, interval=0.5):
+        self.interval, self.ram, self.vram, self._stop = interval, 0, 0, threading.Event()
+
+    def _gpu_mib(self):
+        try:
+            out = subprocess.check_output(["nvidia-smi", "--query-gpu=memory.used", "--format=csv,noheader,nounits"],
+                                          text=True, timeout=5)
+            return int(out.strip().splitlines()[0])
+        except Exception:
+            return 0
+
+    def _run(self):
+        while not self._stop.is_set():
+            vm = psutil.virtual_memory()
+            self.ram = max(self.ram, vm.total - vm.available)
+            self.vram = max(self.vram, self._gpu_mib())
+            self._stop.wait(self.interval)
+
+    def __enter__(self):
+        self._thread = threading.Thread(target=self._run, daemon=True)
+        self._thread.start()
+        return self
+
+    def __exit__(self, *exc):
+        self._stop.set()
+        self._thread.join()
+
+    def as_dict(self):
+        return {"peak_host_ram_used_gib": round(self.ram / 2**30, 2), "peak_gpu_memory_used_mib": self.vram}
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument("--root", type=Path, default=Path("."))
+    parser.add_argument("--pack", type=Path, required=True)
+    parser.add_argument("--url", default="http://127.0.0.1:18096")
+    parser.add_argument("--out", type=Path, default=Path("results"))
+    parser.add_argument("--targets", default="4096,32768,128000")
+    parser.add_argument("--runs", type=int, default=3)
+    args = parser.parse_args()
+    args.out.mkdir(parents=True, exist_ok=True)
+    sys.path[:0] = [str(args.root), str(args.root / "tools")]
+    from strata_tokenizer import Tokenizer
+    from serve.frontend import ChatTemplate, openai_to_messages
+    directory = args.pack / "tokenizer"
+    vocab = json.loads((directory / "vocab.json").read_text(encoding="utf-8"))
+    tokens = [None] * len(vocab)
+    for token, number in vocab.items():
+        tokens[number] = token
+    tokenizer = Tokenizer(tokens, (directory / "merges.txt").read_text(encoding="utf-8").splitlines(),
+                          json.loads((directory / "token_type.json").read_text(encoding="utf-8")))
+    template = ChatTemplate(directory / "chat_template.jinja")
+    initial = get(args.url + "/v1/status")
+    (args.out / "initial-status.json").write_text(json.dumps(initial, indent=2) + "\n", encoding="utf-8")
+    rows = []
+
+    def request(content, maximum=256):
+        return {"model": "strata", "messages": [{"role": "user", "content": content}],
+                "temperature": 0, "reasoning_effort": "none", "max_tokens": maximum,
+                "stream": True, "stream_options": {"include_usage": True}}
+
+    def count(req):
+        messages, tools, kwargs = openai_to_messages(req)
+        return len(tokenizer.encode(template.render(messages, tools, **kwargs), parse_special=True))
+
+    def save():
+        (args.out / "results.json").write_text(json.dumps(rows, indent=2) + "\n", encoding="utf-8")
+
+    def perform(label, req, expected=None):
+        body = json.dumps(req).encode()
+        (args.out / (label + "-request.json")).write_bytes(body + b"\n")
+        print("start", label, "tokens", expected, flush=True)
+        chunks, texts, first, finish = [], [], None, None
+        started = time.perf_counter()
+        try:
+            with Peak() as peak:
+                wire = urllib.request.Request(args.url + "/v1/chat/completions", data=body,
+                                              headers={"Content-Type": "application/json"})
+                with urllib.request.urlopen(wire, timeout=1800) as response:
+                    for line in response:
+                        if not line.startswith(b"data: "):
+                            continue
+                        payload = line[6:].strip()
+                        if payload == b"[DONE]":
+                            break
+                        chunk = json.loads(payload)
+                        chunks.append(chunk)
+                        for choice in chunk.get("choices", []):
+                            delta = choice.get("delta", {})
+                            text = (delta.get("content") or "") + (delta.get("reasoning_content") or "")
+                            if text:
+                                first = first if first is not None else time.perf_counter() - started
+                                texts.append(text)
+                            finish = choice.get("finish_reason") or finish
+            elapsed = time.perf_counter() - started
+            engine = get(args.url + "/metrics")["requests"][0]
+            usage = next((chunk["usage"] for chunk in reversed(chunks) if "usage" in chunk), {})
+            row = {"label": label, "status": "ok", "expected_prompt_tokens": expected,
+                   "request_sha256": hashlib.sha256(body).hexdigest(),
+                   "client_ttft_s": first, "client_elapsed_s": elapsed, "finish_reason": finish,
+                   "usage": usage, "engine": engine, "memory": peak.as_dict(), "text": "".join(texts)}
+            if expected is not None and engine["prompt_tokens"] != expected:
+                raise RuntimeError(f"token count mismatch: {expected} vs {engine['prompt_tokens']}")
+            if not texts:
+                raise RuntimeError(f"no text received for {label}")
+        except Exception as error:  # recorded, excluded from the statistics
+            row = {"label": label, "status": "failed", "error": repr(error), "expected_prompt_tokens": expected,
+                   "request_sha256": hashlib.sha256(body).hexdigest(),
+                   "client_elapsed_s": time.perf_counter() - started, "text": "".join(texts)}
+        (args.out / (label + "-raw.json")).write_text(json.dumps({"chunks": chunks, "row": row}, indent=2) + "\n",
+                                                      encoding="utf-8")
+        rows.append(row)
+        save()
+        print("done", label, row["status"], json.dumps({k: row.get(k) for k in ("client_ttft_s", "client_elapsed_s", "memory")}),
+              flush=True)
+        return row
+
+    perform("warmup", request("Reply with exactly the word READY.", 16))
+    # Each nonce occurs before the filler so previous long prompts cannot be reused.
+    filler = "\n".join(f"def task_{i:05d}(value: int) -> int: return (value * {(i % 97) + 1} + {i}) % 100003"
+                       for i in range(12000))
+    ending = ("\n\nWrite a detailed explanation of the code above. Discuss deterministic integer transforms, "
+              "modulo arithmetic, testing, naming, complexity, and maintainability. Write at least 600 words.")
+    for target in map(int, args.targets.split(",")):
+        for run in range(1, args.runs + 1):
+            prefix = f"Benchmark nonce: series-{target}-trial-{run}.\nReview this synthetic Python module:\n"
+            lo, hi = 0, len(filler)
+            while lo < hi:
+                middle = (lo + hi + 1) // 2
+                if count(request(prefix + filler[:middle] + ending)) <= target:
+                    lo = middle
+                else:
+                    hi = middle - 1
+            req = request(prefix + filler[:lo] + ending)
+            actual = count(req)
+            if actual < target - 20 or actual > target:
+                raise RuntimeError(f"could not construct target {target}: {actual}")
+            perform(f"tokens-{target}-run-{run}", req, actual)
+    measured = [row for row in rows if row["label"] != "warmup" and row["status"] == "ok"]
+    groups = {}
+    for target in map(int, args.targets.split(",")):
+        subset = [r for r in measured if r["label"].startswith(f"tokens-{target}-")]
+        failed = [r["label"] for r in rows if r["label"].startswith(f"tokens-{target}-") and r["status"] != "ok"]
+        if not subset:
+            groups[str(target)] = {"runs_ok": 0, "failed": failed}
+            continue
+        values = {
+            "prompt_tokens": [r["engine"]["prompt_tokens"] for r in subset],
+            "cached_tokens": [r["engine"].get("reused", 0) for r in subset],
+            "output_tokens": [r["engine"]["engine_generated"] for r in subset],
+            "client_ttft_s": [r["client_ttft_s"] for r in subset],
+            "client_elapsed_s": [r["client_elapsed_s"] for r in subset],
+            "prefill_tok_s": [(r["engine"]["prompt_tokens"] - (r["engine"].get("reused") or 0)) / (r["engine"]["prompt_ms"] / 1000) for r in subset],
+            "decode_tok_s": [r["engine"]["engine_generated"] / (r["engine"]["decode_ms"] / 1000) for r in subset],
+            "peak_host_ram_used_gib": [r["memory"]["peak_host_ram_used_gib"] for r in subset],
+            "peak_gpu_memory_used_mib": [r["memory"]["peak_gpu_memory_used_mib"] for r in subset],
+        }
+        groups[str(target)] = {"runs_ok": len(subset), "failed": failed,
+                               **{key: {"median": statistics.median(v), "min": min(v), "max": max(v)} for key, v in values.items()}}
+    (args.out / "summary.json").write_text(json.dumps(groups, indent=2) + "\n", encoding="utf-8")
+    print("summary", json.dumps(groups), flush=True)
+
+
+if __name__ == "__main__":
+    main()