diff --git a/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/BUILD.json b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/BUILD.json new file mode 100644 index 00000000000..3606f7f4ba9 --- /dev/null +++ b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/BUILD.json @@ -0,0 +1,13 @@ +{ + "source": "local", + "version": "0.1.40.3", + "archs": [ + 120 + ], + "vision": "gpu", + "cuda_dirs": [ + "\\bin" + ], + "src": "38215a61b1d7b518", + "vision_src": "1fb2b3fb797ec402" +} \ No newline at end of file diff --git a/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/README.md b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/README.md new file mode 100644 index 00000000000..098a2b49c14 --- /dev/null +++ b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/README.md @@ -0,0 +1,100 @@ +# Community benchmark: RTX 5090 Laptop GPU (24 GB), engine 0.1.40.3 + +Measured in the night of 2026-10-07/08 by [wolffahrer](https://github.com/wolffahrer) on the same Windows 11 gaming +laptop (Schenker XMG NEO 16, A25) as [2026-10-06-community-rtx-5090-laptop](../2026-10-06-community-rtx-5090-laptop/README.md) +(engine 0.1.40.1, PR #1263). This is a follow-up: same machine, same model files, same configuration, same scripts, +same order of requests. Only the Strata version changed. 0.1.40.2 was measured the same night with the identical +procedure; its numbers are in a short section below. + +Median decode throughput with **0.1.40.3** was **138.1 tok/s at 4,096 prompt tokens, 144.4 tok/s at 32,768, and +131.0 tok/s at 128,000**. Median prompt throughput was 1,949.7, 2,893.3 and 2,780.1 tok/s. All nine speed requests +succeeded with zero reused tokens. Against 0.1.40.1 this is within run-to-run variation (see the comparison table). + +## Hardware and software + +Unchanged from the 0.1.40.1 entry unless noted: + +- NVIDIA GeForce RTX 5090 **Laptop** GPU, 24,463 MiB reported VRAM, 175 W limit (laptop Dynamic Boost). Sampled draw + during this run median 163.9 W, peak sample 179.4 W; GPU temperature peaked at 84 °C. PCIe Gen 4 x8 under load. +- AMD Ryzen 9 9955HX3D, `--pool-workers 10`. 64 GB RAM (61.68 GiB usable). Samsung SSD 990 PRO 1TB for model and pack. +- Windows 11 Home, build 26200. NVIDIA driver 610.88. CUDA toolkit 13.3. +- Strata **v0.1.40.3**, tag commit `d5ea7133741e67743c0e886bb426c0ce8d69cf6c`, built from the release source archive + (sha256 `6fdca77a04d8535c598c6602990789892d999000f8b7e956a27ed0dc602b5cba`) with the unmodified installer for CUDA + architecture 120, GPU vision helper included. See [BUILD.json](BUILD.json). +- Background: a WSL2 VM with an idle local agent stack; the regular inference server was stopped, so the GPU was + dedicated to Strata and its vision helper. Not an isolated OS. + +## Model and configuration + +Identical to the 0.1.40.1 entry: `ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF` **Q2_0**, context 131,072, INT8 KV with +32,768 resident cells, expert cache `auto` (again **11,742 slots, 15.12 GiB VRAM**, 495 MiB VRAM free with everything +loaded), prefill `auto` (borrows 2,082 slots), MTP `--spec 4 --spec-min-p 0.7`, `--pcie-frac 0`, `--mmap-experts`, GPU +vision, conversation cache 6 GiB / 2 slots, `reasoning_budget_tokens: 12000`, `reasoning_loop_recovery: "stop"`. +Complete configuration with local paths replaced by placeholders: [strata-q2_0.json](strata-q2_0.json). Startup +choices and every request's timing are in [engine.log](engine.log). + +## Method + +Same scripts as the 0.1.40.1 entry: [benchmark.py](benchmark.py) (4K/32K/128K synthetic code-explanation prompts, +three runs each, greedy, reasoning off, 256-token cap), the unchanged `tools/needle_bench.py` (32k/128k at depths +10/50/90) and [coding_check.py](coding_check.py) (reasoning `low`). One warm-up request excluded. The server was healthy +12.1 s after start. Memory and GPU state were sampled once per second by [monitor.py](monitor.py); the summary is in +[memory-summary.json](memory-summary.json) (298 samples over 393.0 s; the raw `telemetry.jsonl` is left out to keep +the folder small). + +```text +.venv\Scripts\python.exe serve\server.py --engine strata --config strata-q2_0.json --port 18096 +.venv\Scripts\python.exe benchmark.py --root --pack \packs\q2_0 --url http://127.0.0.1:18096 --out results +.venv\Scripts\python.exe tools\needle_bench.py --url http://127.0.0.1:18096 --lengths 32k,128k --depths 10,50,90 --out needles.json +.venv\Scripts\python.exe coding_check.py --url http://127.0.0.1:18096 --out coding-check.json +``` + +## Results (0.1.40.3) + +Median **[minimum–maximum]** of three runs. Every request generated 256 tokens; no failed or cancelled requests. + +| Prompt tokens | Reused | Prompt tok/s | Decode tok/s | TTFT seconds | Total seconds | +| ---: | ---: | ---: | ---: | ---: | ---: | +| 4,096 | 0 | 1,949.7 [1,805.6–1,958.6] | 138.1 [125.5–148.5] | 2.133 [2.124–2.302] | 4.083 [3.848–4.329] | +| 32,768 | 0 | 2,893.3 [2,888.9–2,906.8] | 144.4 [140.1–145.3] | 11.384 [11.330–11.427] | 13.145 [13.145–13.177] | +| 128,000 | 0 | 2,780.1 [2,752.8–2,801.1] | 131.0 [123.2–140.4] | 46.246 [45.884–46.702] | 48.513 [47.827–48.540] | + +Raw per-run records: [results.json](results.json); aggregates: [summary.json](summary.json). Decode expert-cache hit +rates were 90.2–98.7% for the nine speed requests. Peak GPU memory 23,221 MiB; host RAM in use (system-wide, includes +the WSL2 VM) 12.74 GiB before load, 54.88 GiB peak; page file unchanged at 0.50 GiB. + +**Across versions on this laptop** (median decode / prompt tok/s; same procedure, not interleaved, one run of three +requests per length each): + +| Prompt tokens | 0.1.40.1 (2026-10-06) | 0.1.40.2 (2026-10-07 23:16) | 0.1.40.3 (2026-10-08 00:50) | +| ---: | ---: | ---: | ---: | +| 4,096 | 136.9 / 1,952.5 | 139.3 / 1,960.0 | 138.1 / 1,949.7 | +| 32,768 | 137.7 / 2,877.8 | 141.6 / 2,890.0 | 144.4 / 2,893.3 | +| 128,000 | 134.6 / 2,764.1 | 137.2 / 2,764.1 | 131.0 / 2,780.1 | + +Decode differences of a few percent are inside the min–max ranges of single runs; prompt throughput is flat. +We read this as "no measurable speed change on this hardware" for 0.1.40.2 and 0.1.40.3, and no regression. + +## Correctness and limitations + +- **Long-context recall:** all six needles found (prompt lengths 32,895–32,896 and 125,868–125,870 tokens). One 32K and + two 128K cases reused 16,384 prompt tokens. See [needles.json](needles.json). +- **Coding check (0.1.40.3):** **10/10 tests passed**, 823 generated tokens, 7.9 s. See [coding-check.json](coding-check.json). +- **Coding check (0.1.40.2), a note on greedy decoding:** the 0.1.40.2 run scored **0/10**. The answer text was + empty after 2,567 generated tokens, all of them reasoning (the raw response of that run was not kept). We re-ran the + check three times per version on a freshly started server and kept the raw responses: 0.1.40.2 scored 0, 10, 10 and + 0.1.40.1 scored 0, 10, 6. Both zeros were the first request after the start, with **identical** looping reasoning in + both versions ("If 'IV' with invalid char? 'IV' pair. ..."), ended by the configured + `reasoning_loop_recovery: "stop"` (`finish_reason: length`, 2,048 tokens). The result varies between runs in both + versions, so we do not read the 0/10 as a 0.1.40.2 regression but as a property of greedy decoding with reasoning + `low` on this Q2_0 model and prompt. +- Not evaluated: long outputs, sampled decoding, thinking speed, vision, tool use, concurrency, sustained thermal load. + Laptop GPU clocks can vary with temperature and power state. + +Besides these synthetic runs we checked both versions with our own agent task suite (multi-step tool use under a local +agent, German prompts, 44 tasks x 3 runs per version). Pass rates were within two runs of 0.1.40.1 on every set +(total 121, 122 and 123 of 132 runs for 0.1.40.1, 0.1.40.2 and 0.1.40.3); those tasks are private and not part of this +report. + +The author documents local AI on consumer hardware (in German) on the KI SOUVERÄN channels: +[YouTube](https://www.youtube.com/channel/UCE6Ch6g6Bo8v4ROpYDzOOxA) and [Telegram](https://t.me/lokale_ki). diff --git a/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/benchmark.py b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/benchmark.py new file mode 100644 index 00000000000..33f823beee8 --- /dev/null +++ b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/benchmark.py @@ -0,0 +1,198 @@ +#!/usr/bin/env python3 +"""Serial, fresh-prompt Strata benchmark (Windows-capable). + +Derived from bench/results/2026-09-30-community-rtx-5090/benchmark.py: same synthetic prompts, nonces, +token targets, greedy decoding, reasoning off and 256-token output cap, so the numbers are comparable. +Changes: runs with Strata's own Windows venv; a failed or cancelled run is recorded with its error and +excluded from the statistics instead of stopping the suite; peak host RAM and GPU memory are sampled +every 0.5 s during each request. + + .venv\\Scripts\\python.exe benchmark.py --root . --pack \\packs\\q2_0 --url http://127.0.0.1:18096 --out results +""" +import argparse +import hashlib +import json +import statistics +import subprocess +import sys +import threading +import time +import urllib.request +from pathlib import Path + +import psutil + + +def get(url): + with urllib.request.urlopen(url, timeout=30) as response: + return json.load(response) + + +class Peak: + """Samples system-wide host RAM in use (total - available) and GPU memory used while a request runs.""" + + def __init__(self, interval=0.5): + self.interval, self.ram, self.vram, self._stop = interval, 0, 0, threading.Event() + + def _gpu_mib(self): + try: + out = subprocess.check_output(["nvidia-smi", "--query-gpu=memory.used", "--format=csv,noheader,nounits"], + text=True, timeout=5) + return int(out.strip().splitlines()[0]) + except Exception: + return 0 + + def _run(self): + while not self._stop.is_set(): + vm = psutil.virtual_memory() + self.ram = max(self.ram, vm.total - vm.available) + self.vram = max(self.vram, self._gpu_mib()) + self._stop.wait(self.interval) + + def __enter__(self): + self._thread = threading.Thread(target=self._run, daemon=True) + self._thread.start() + return self + + def __exit__(self, *exc): + self._stop.set() + self._thread.join() + + def as_dict(self): + return {"peak_host_ram_used_gib": round(self.ram / 2**30, 2), "peak_gpu_memory_used_mib": self.vram} + + +def main(): + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--root", type=Path, default=Path(".")) + parser.add_argument("--pack", type=Path, required=True) + parser.add_argument("--url", default="http://127.0.0.1:18096") + parser.add_argument("--out", type=Path, default=Path("results")) + parser.add_argument("--targets", default="4096,32768,128000") + parser.add_argument("--runs", type=int, default=3) + args = parser.parse_args() + args.out.mkdir(parents=True, exist_ok=True) + sys.path[:0] = [str(args.root), str(args.root / "tools")] + from strata_tokenizer import Tokenizer + from serve.frontend import ChatTemplate, openai_to_messages + directory = args.pack / "tokenizer" + vocab = json.loads((directory / "vocab.json").read_text(encoding="utf-8")) + tokens = [None] * len(vocab) + for token, number in vocab.items(): + tokens[number] = token + tokenizer = Tokenizer(tokens, (directory / "merges.txt").read_text(encoding="utf-8").splitlines(), + json.loads((directory / "token_type.json").read_text(encoding="utf-8"))) + template = ChatTemplate(directory / "chat_template.jinja") + initial = get(args.url + "/v1/status") + (args.out / "initial-status.json").write_text(json.dumps(initial, indent=2) + "\n", encoding="utf-8") + rows = [] + + def request(content, maximum=256): + return {"model": "strata", "messages": [{"role": "user", "content": content}], + "temperature": 0, "reasoning_effort": "none", "max_tokens": maximum, + "stream": True, "stream_options": {"include_usage": True}} + + def count(req): + messages, tools, kwargs = openai_to_messages(req) + return len(tokenizer.encode(template.render(messages, tools, **kwargs), parse_special=True)) + + def save(): + (args.out / "results.json").write_text(json.dumps(rows, indent=2) + "\n", encoding="utf-8") + + def perform(label, req, expected=None): + body = json.dumps(req).encode() + (args.out / (label + "-request.json")).write_bytes(body + b"\n") + print("start", label, "tokens", expected, flush=True) + chunks, texts, first, finish = [], [], None, None + started = time.perf_counter() + try: + with Peak() as peak: + wire = urllib.request.Request(args.url + "/v1/chat/completions", data=body, + headers={"Content-Type": "application/json"}) + with urllib.request.urlopen(wire, timeout=1800) as response: + for line in response: + if not line.startswith(b"data: "): + continue + payload = line[6:].strip() + if payload == b"[DONE]": + break + chunk = json.loads(payload) + chunks.append(chunk) + for choice in chunk.get("choices", []): + delta = choice.get("delta", {}) + text = (delta.get("content") or "") + (delta.get("reasoning_content") or "") + if text: + first = first if first is not None else time.perf_counter() - started + texts.append(text) + finish = choice.get("finish_reason") or finish + elapsed = time.perf_counter() - started + engine = get(args.url + "/metrics")["requests"][0] + usage = next((chunk["usage"] for chunk in reversed(chunks) if "usage" in chunk), {}) + row = {"label": label, "status": "ok", "expected_prompt_tokens": expected, + "request_sha256": hashlib.sha256(body).hexdigest(), + "client_ttft_s": first, "client_elapsed_s": elapsed, "finish_reason": finish, + "usage": usage, "engine": engine, "memory": peak.as_dict(), "text": "".join(texts)} + if expected is not None and engine["prompt_tokens"] != expected: + raise RuntimeError(f"token count mismatch: {expected} vs {engine['prompt_tokens']}") + if not texts: + raise RuntimeError(f"no text received for {label}") + except Exception as error: # recorded, excluded from the statistics + row = {"label": label, "status": "failed", "error": repr(error), "expected_prompt_tokens": expected, + "request_sha256": hashlib.sha256(body).hexdigest(), + "client_elapsed_s": time.perf_counter() - started, "text": "".join(texts)} + (args.out / (label + "-raw.json")).write_text(json.dumps({"chunks": chunks, "row": row}, indent=2) + "\n", + encoding="utf-8") + rows.append(row) + save() + print("done", label, row["status"], json.dumps({k: row.get(k) for k in ("client_ttft_s", "client_elapsed_s", "memory")}), + flush=True) + return row + + perform("warmup", request("Reply with exactly the word READY.", 16)) + # Each nonce occurs before the filler so previous long prompts cannot be reused. + filler = "\n".join(f"def task_{i:05d}(value: int) -> int: return (value * {(i % 97) + 1} + {i}) % 100003" + for i in range(12000)) + ending = ("\n\nWrite a detailed explanation of the code above. Discuss deterministic integer transforms, " + "modulo arithmetic, testing, naming, complexity, and maintainability. Write at least 600 words.") + for target in map(int, args.targets.split(",")): + for run in range(1, args.runs + 1): + prefix = f"Benchmark nonce: series-{target}-trial-{run}.\nReview this synthetic Python module:\n" + lo, hi = 0, len(filler) + while lo < hi: + middle = (lo + hi + 1) // 2 + if count(request(prefix + filler[:middle] + ending)) <= target: + lo = middle + else: + hi = middle - 1 + req = request(prefix + filler[:lo] + ending) + actual = count(req) + if actual < target - 20 or actual > target: + raise RuntimeError(f"could not construct target {target}: {actual}") + perform(f"tokens-{target}-run-{run}", req, actual) + measured = [row for row in rows if row["label"] != "warmup" and row["status"] == "ok"] + groups = {} + for target in map(int, args.targets.split(",")): + subset = [r for r in measured if r["label"].startswith(f"tokens-{target}-")] + failed = [r["label"] for r in rows if r["label"].startswith(f"tokens-{target}-") and r["status"] != "ok"] + if not subset: + groups[str(target)] = {"runs_ok": 0, "failed": failed} + continue + values = { + "prompt_tokens": [r["engine"]["prompt_tokens"] for r in subset], + "cached_tokens": [r["engine"].get("reused", 0) for r in subset], + "output_tokens": [r["engine"]["engine_generated"] for r in subset], + "client_ttft_s": [r["client_ttft_s"] for r in subset], + "client_elapsed_s": [r["client_elapsed_s"] for r in subset], + "prefill_tok_s": [(r["engine"]["prompt_tokens"] - (r["engine"].get("reused") or 0)) / (r["engine"]["prompt_ms"] / 1000) for r in subset], + "decode_tok_s": [r["engine"]["engine_generated"] / (r["engine"]["decode_ms"] / 1000) for r in subset], + "peak_host_ram_used_gib": [r["memory"]["peak_host_ram_used_gib"] for r in subset], + "peak_gpu_memory_used_mib": [r["memory"]["peak_gpu_memory_used_mib"] for r in subset], + } + groups[str(target)] = {"runs_ok": len(subset), "failed": failed, + **{key: {"median": statistics.median(v), "min": min(v), "max": max(v)} for key, v in values.items()}} + (args.out / "summary.json").write_text(json.dumps(groups, indent=2) + "\n", encoding="utf-8") + print("summary", json.dumps(groups), flush=True) + + +if __name__ == "__main__": + main() diff --git a/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/coding-check.json b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/coding-check.json new file mode 100644 index 00000000000..4b5480f8f1d --- /dev/null +++ b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/coding-check.json @@ -0,0 +1,78 @@ +{ + "task": "Write a Python function `roman_to_int(s: str) -> int` that converts a Roman numeral (I, V, X, L, C, D, M, including subtractive forms like IV, IX, XL, XC, CD, CM) to an integer, and raises ValueError for an empty string or any character that is not a Roman numeral. Reply with one Python code block only.", + "reasoning_effort": "low", + "elapsed_s": 7.9, + "usage": { + "prompt_tokens": 121, + "completion_tokens": 823, + "total_tokens": 944, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "passed": 10, + "total": 10, + "results": [ + { + "input": "III", + "expected": 3, + "got": 3, + "ok": true + }, + { + "input": "IV", + "expected": 4, + "got": 4, + "ok": true + }, + { + "input": "IX", + "expected": 9, + "got": 9, + "ok": true + }, + { + "input": "LVIII", + "expected": 58, + "got": 58, + "ok": true + }, + { + "input": "MCMXCIV", + "expected": 1994, + "got": 1994, + "ok": true + }, + { + "input": "MMXXVI", + "expected": 2026, + "got": 2026, + "ok": true + }, + { + "input": "CDXLIV", + "expected": 444, + "got": 444, + "ok": true + }, + { + "input": "", + "expected": "ValueError", + "got": "ValueError", + "ok": true + }, + { + "input": "ABC", + "expected": "ValueError", + "got": "ValueError", + "ok": true + }, + { + "input": "MMX1", + "expected": "ValueError", + "got": "ValueError", + "ok": true + } + ], + "code": "def roman_to_int(s: str) -> int:\n if not s:\n raise ValueError(\"Empty string\")\n\n values = {\n \"I\": 1,\n \"V\": 5,\n \"X\": 10,\n \"L\": 50,\n \"C\": 100,\n \"D\": 500,\n \"M\": 1000,\n }\n\n if any(ch not in values for ch in s):\n raise ValueError(\"Invalid Roman numeral character\")\n\n total = 0\n i = 0\n\n while i < len(s):\n if i + 1 < len(s) and s[i:i + 2] in {\n \"IV\": 4,\n \"IX\": 9,\n \"XL\": 40,\n \"XC\": 90,\n \"CD\": 400,\n \"CM\": 900,\n }:\n total += values[s[i + 1]] - values[s[i]]\n i += 2\n else:\n total += values[s[i]]\n i += 1\n\n return total\n" +} \ No newline at end of file diff --git a/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/coding_check.py b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/coding_check.py new file mode 100644 index 00000000000..c370816f29a --- /dev/null +++ b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/coding_check.py @@ -0,0 +1,64 @@ +#!/usr/bin/env python3 +"""Small verifiable coding check: the model writes a function, a fixed test suite runs it locally. + + python coding_check.py --url http://127.0.0.1:18096 --out coding-check.json +""" +import argparse +import json +import re +import time +import urllib.request + +TASK = ("Write a Python function `roman_to_int(s: str) -> int` that converts a Roman numeral (I, V, X, L, C, D, M, " + "including subtractive forms like IV, IX, XL, XC, CD, CM) to an integer, and raises ValueError for an empty " + "string or any character that is not a Roman numeral. Reply with one Python code block only.") +TESTS = [("III", 3), ("IV", 4), ("IX", 9), ("LVIII", 58), ("MCMXCIV", 1994), ("MMXXVI", 2026), ("CDXLIV", 444)] +ERRORS = ["", "ABC", "MMX1"] + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--url", default="http://127.0.0.1:18096") + ap.add_argument("--out", default="coding-check.json") + ap.add_argument("--effort", default="low") + a = ap.parse_args() + body = {"model": "strata", "messages": [{"role": "user", "content": TASK}], "temperature": 0, + "reasoning_effort": a.effort, "max_tokens": 4096} + t0 = time.perf_counter() + req = urllib.request.Request(a.url + "/v1/chat/completions", data=json.dumps(body).encode(), + headers={"Content-Type": "application/json"}) + with urllib.request.urlopen(req, timeout=900) as r: + answer = json.loads(r.read()) + elapsed = time.perf_counter() - t0 + text = answer["choices"][0]["message"].get("content") or "" + m = re.search(r"```(?:python)?\n(.*?)```", text, re.S) + code = m.group(1) if m else text + results, passed = [], 0 + namespace = {} + try: + exec(compile(code, "", "exec"), namespace) + f = namespace["roman_to_int"] + for s, want in TESTS: + got = f(s) + ok = got == want + passed += ok + results.append({"input": s, "expected": want, "got": got, "ok": ok}) + for s in ERRORS: + try: + f(s) + results.append({"input": s, "expected": "ValueError", "got": "no error", "ok": False}) + except ValueError: + passed += 1 + results.append({"input": s, "expected": "ValueError", "got": "ValueError", "ok": True}) + except Exception as e: + results.append({"error": repr(e)}) + total = len(TESTS) + len(ERRORS) + out = {"task": TASK, "reasoning_effort": a.effort, "elapsed_s": round(elapsed, 2), "usage": answer.get("usage"), + "passed": passed, "total": total, "results": results, "code": code} + with open(a.out, "w", encoding="utf-8") as fh: + json.dump(out, fh, indent=2) + print(f"coding check: {passed}/{total} tests passed in {elapsed:.1f} s") + + +if __name__ == "__main__": + main() diff --git a/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/final-status.json b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/final-status.json new file mode 100644 index 00000000000..ac269834552 --- /dev/null +++ b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/final-status.json @@ -0,0 +1,74 @@ +{ + "service": "strata", + "model": "qwen3.8-flash-next-q2_0", + "loaded": true, + "auto_load": true, + "structured_output": { + "formats": [ + "json_object", + "json_schema" + ], + "method": "prompt_and_validate", + "constrained_decoding": false, + "stream_buffered": true + }, + "engine": "0.1.40.3", + "started": 1791413434, + "uptime_s": 383, + "cache_max_tokens": 131072, + "context": { + "native": 131072, + "max_positions": 131072 + }, + "concurrency": { + "serving": 1, + "requested": 1 + }, + "dialects": [ + "/v1/chat/completions", + "/v1/messages", + "/v1/responses" + ], + "vision": { + "enabled": true, + "available": true, + "error": null + }, + "activity": { + "requests": 17, + "in_flight": 0, + "last_request_at": 1791413818 + }, + "last_timings": { + "cache_n": 0, + "prompt_n": 121, + "prompt_ms": 1174.8, + "prompt_per_token_ms": 9.709, + "prompt_per_second": 103.0, + "predicted_n": 823, + "predicted_ms": 6700.2, + "predicted_per_token_ms": 8.141, + "predicted_per_second": 122.8, + "draft_n": 516, + "draft_n_accepted": 444, + "at": 1791413818 + }, + "vram": { + "elastic": false + }, + "machine": { + "at": 1791413818, + "gpu": { + "name": "NVIDIA GeForce RTX 5090 Laptop GPU", + "used_mib": 23546.0, + "total_mib": 24463.0, + "util_pct": 94, + "temp_c": 77, + "power_w": 140.214 + }, + "ram": { + "used_gib": 54.0, + "total_gib": 61.7 + } + } +} \ No newline at end of file diff --git a/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/initial-status.json b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/initial-status.json new file mode 100644 index 00000000000..88cdde15f96 --- /dev/null +++ b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/initial-status.json @@ -0,0 +1,61 @@ +{ + "service": "strata", + "model": "qwen3.8-flash-next-q2_0", + "loaded": true, + "auto_load": true, + "structured_output": { + "formats": [ + "json_object", + "json_schema" + ], + "method": "prompt_and_validate", + "constrained_decoding": false, + "stream_buffered": true + }, + "engine": "0.1.40.3", + "started": 1791413434, + "uptime_s": 1, + "cache_max_tokens": 131072, + "context": { + "native": 131072, + "max_positions": 131072 + }, + "concurrency": { + "serving": 1, + "requested": 1 + }, + "dialects": [ + "/v1/chat/completions", + "/v1/messages", + "/v1/responses" + ], + "vision": { + "enabled": true, + "available": true, + "error": null + }, + "activity": { + "requests": 0, + "in_flight": 0, + "last_request_at": null + }, + "last_timings": null, + "vram": { + "elastic": false + }, + "machine": { + "at": 1791413435, + "gpu": { + "name": "NVIDIA GeForce RTX 5090 Laptop GPU", + "used_mib": 23462.0, + "total_mib": 24463.0, + "util_pct": 0, + "temp_c": 55, + "power_w": 31.012 + }, + "ram": { + "used_gib": 32.3, + "total_gib": 61.7 + } + } +} \ No newline at end of file diff --git a/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/memory-summary.json b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/memory-summary.json new file mode 100644 index 00000000000..d4eac42bc1e --- /dev/null +++ b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/memory-summary.json @@ -0,0 +1,25 @@ +{ + "samples": 298, + "duration_s": 393.0, + "host_total_gib": 61.677, + "host_used_gib": { + "first": 12.738, + "peak": 54.883 + }, + "swap_used_gib": { + "first": 0.497, + "peak": 0.497 + }, + "gpu_memory_used_mib_peak": 23221, + "gpu_memory_total_mib": 24463, + "gpu_power_draw_w": { + "median": 163.88, + "peak": 179.36 + }, + "gpu_temperature_c_peak": 84, + "pcie_link_seen": [ + "Gen1 x8", + "Gen2 x8", + "Gen4 x8" + ] +} \ No newline at end of file diff --git a/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/monitor.py b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/monitor.py new file mode 100644 index 00000000000..a8831f7ef59 --- /dev/null +++ b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/monitor.py @@ -0,0 +1,11 @@ +"""Sample host RAM in use (total - available), swap and GPU state once per second (Windows, psutil + nvidia-smi).""" +import json, subprocess, time, psutil +while True: + vm, sw = psutil.virtual_memory(), psutil.swap_memory() + try: + g = subprocess.check_output(["nvidia-smi", "--query-gpu=memory.used,memory.total,utilization.gpu,power.draw,temperature.gpu,pcie.link.gen.current,pcie.link.width.current", "--format=csv,noheader,nounits"], text=True, timeout=5).strip() + except Exception as e: + g = repr(e) + print(json.dumps({"epoch_s": time.time(), "host_used_gib": round((vm.total - vm.available) / 2**30, 3), + "host_total_gib": round(vm.total / 2**30, 3), "swap_used_gib": round(sw.used / 2**30, 3), "gpu": g}), flush=True) + time.sleep(1) diff --git a/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/needles.json b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/needles.json new file mode 100644 index 00000000000..c38177b0f62 --- /dev/null +++ b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/needles.json @@ -0,0 +1,56 @@ +[ + { + "length": "32k", + "depth": 10, + "prompt_tokens": 32895, + "seconds": 13.0, + "found": true, + "word": "meadow-copper-504", + "answer": "meadow-copper-504" + }, + { + "length": "32k", + "depth": 50, + "prompt_tokens": 32895, + "seconds": 12.4, + "found": true, + "word": "falcon-quartz-940", + "answer": "falcon-quartz-940" + }, + { + "length": "32k", + "depth": 90, + "prompt_tokens": 32896, + "seconds": 7.1, + "found": true, + "word": "willow-cobalt-696", + "answer": "willow-cobalt-696" + }, + { + "length": "128k", + "depth": 10, + "prompt_tokens": 125870, + "seconds": 45.9, + "found": true, + "word": "falcon-saffron-138", + "answer": "falcon-saffron-138" + }, + { + "length": "128k", + "depth": 50, + "prompt_tokens": 125868, + "seconds": 41.1, + "found": true, + "word": "quartz-tundra-528", + "answer": "quartz-tundra-528" + }, + { + "length": "128k", + "depth": 90, + "prompt_tokens": 125868, + "seconds": 40.9, + "found": true, + "word": "quartz-glacier-192", + "answer": "quartz-glacier-192" + } +] \ No newline at end of file diff --git a/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/results.json b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/results.json new file mode 100644 index 00000000000..7657d35213c --- /dev/null +++ b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/results.json @@ -0,0 +1,462 @@ +[ + { + "label": "warmup", + "status": "ok", + "expected_prompt_tokens": null, + "request_sha256": "b63e57873441a996a529b246a66cf12de70a7c671777f99329354d5c195f788f", + "client_ttft_s": 0.2916134000115562, + "client_elapsed_s": 0.37406460000784136, + "finish_reason": "stop", + "usage": { + "prompt_tokens": 19, + "completion_tokens": 2, + "total_tokens": 21, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "engine": { + "projection": null, + "time": 1791413435.9922154, + "duration_s": 0.4, + "finish": "stop", + "prompt_tokens": 19, + "reused": 0, + "output_tokens": 2, + "prompt_total": 19, + "prompt_read": 19, + "reasoning_recoveries": 0, + "reasoning_repeat_coverage": 0.0, + "engine_generated": 2, + "prompt_ms": 263.7, + "decode_ms": 106.4, + "decode_tok_s": 18.8, + "hit_rate": 0.596, + "pcie_share": 0.0, + "ram_blobs": 0, + "file_blobs": 0, + "file_mb": 1054.8, + "drafts_offered": 3, + "drafts_accepted": 3 + }, + "memory": { + "peak_host_ram_used_gib": 32.49, + "peak_gpu_memory_used_mib": 23143 + }, + "text": "READY" + }, + { + "label": "tokens-4096-run-1", + "status": "ok", + "expected_prompt_tokens": 4096, + "request_sha256": "5d94c652067354bbe72930d2bdad3886564f08cf2546b32411af02d33ad3ce05", + "client_ttft_s": 2.301761800015811, + "client_elapsed_s": 4.3289522000122815, + "finish_reason": "length", + "usage": { + "prompt_tokens": 4096, + "completion_tokens": 256, + "total_tokens": 4352, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "engine": { + "projection": null, + "time": 1791413436.9593554, + "duration_s": 4.3, + "finish": "length", + "prompt_tokens": 4096, + "reused": 0, + "output_tokens": 256, + "prompt_total": 4096, + "prompt_read": 4096, + "reasoning_recoveries": 0, + "reasoning_repeat_coverage": 0.0, + "engine_generated": 256, + "prompt_ms": 2268.5, + "decode_ms": 2039.3, + "decode_tok_s": 125.5, + "hit_rate": 0.902, + "pcie_share": 0.0, + "ram_blobs": 0, + "file_blobs": 0, + "file_mb": 18065.2, + "drafts_offered": 186, + "drafts_accepted": 148 + }, + "memory": { + "peak_host_ram_used_gib": 49.68, + "peak_gpu_memory_used_mib": 23215 + }, + "text": "This synthetic Python module presents a fascinating case study in algorithmic redundancy, modular arithmetic, and the pitfalls of manual code generation. At first glance, the code appears to be a collection of 113 distinct functions, each named `task_XXXXX`, where `XXXXX` is a zero-padded integer index. However, upon closer inspection, these functions are not unique algorithms but rather instances of a single, generalized linear transformation applied to an input value.\n\n### The Core Mathematical Pattern\n\nEvery function in the module follows the exact same structural template:\n```python\ndef task_N(value: int) -> int:\n return (value * A + B) % M\n```\nWhere:\n- `M` is the constant modulus `100003`.\n- `A` is the multiplier.\n- `B` is the additive constant.\n\nFor the first 97 functions (`task_00000` to `task_00096`), the pattern is strictly linear with respect to the index `N`:\n- Multiplier `A = N + 1`\n- Additive constant `B = N`\n\nFor example:\n- `task_00000" + }, + { + "label": "tokens-4096-run-2", + "status": "ok", + "expected_prompt_tokens": 4096, + "request_sha256": "8da8a04e70e4d916f6630729082bdfb9d6ebbb47117aad60bc0541bce198a213", + "client_ttft_s": 2.1244121000054292, + "client_elapsed_s": 4.082541699986905, + "finish_reason": "length", + "usage": { + "prompt_tokens": 4096, + "completion_tokens": 256, + "total_tokens": 4352, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "engine": { + "projection": null, + "time": 1791413441.9287343, + "duration_s": 3.9, + "finish": "length", + "prompt_tokens": 4096, + "reused": 0, + "output_tokens": 256, + "prompt_total": 4096, + "prompt_read": 4096, + "reasoning_recoveries": 0, + "reasoning_repeat_coverage": 0.0, + "engine_generated": 256, + "prompt_ms": 2091.3, + "decode_ms": 1853.9, + "decode_tok_s": 138.1, + "hit_rate": 0.964, + "pcie_share": 0.0, + "ram_blobs": 0, + "file_blobs": 0, + "file_mb": 7397.2, + "drafts_offered": 170, + "drafts_accepted": 135 + }, + "memory": { + "peak_host_ram_used_gib": 49.97, + "peak_gpu_memory_used_mib": 23215 + }, + "text": "This synthetic Python module presents a fascinating case study in algorithmic redundancy, modular arithmetic, and the pitfalls of manual code generation. At first glance, the code appears to be a collection of 113 distinct functions, each named `task_XXXXX`, where `XXXXX` is a zero-padded integer index. However, a deeper analysis reveals that these functions are not unique algorithms but rather instances of a single, generalized linear transformation applied to different parameters. This review will dissect the mathematical structure, discuss the implications for software engineering, and propose a more maintainable approach.\n\n### Mathematical Structure and Deterministic Transforms\n\nThe core logic of every function in the module follows the pattern:\n`return (value * multiplier + offset) % 100003`\n\nThis is a classic **affine transformation** over the integers, specifically within the ring of integers modulo $M$, where $M = 100003$. The number 100003 is a prime number, which is significant in cryptography and number theory because it ensures that the multiplicative group modulo $M$ is well-behaved. In this context, the transformation maps an input integer `value` to a new integer within the range $[" + }, + { + "label": "tokens-4096-run-3", + "status": "ok", + "expected_prompt_tokens": 4096, + "request_sha256": "946079edb58e764050805886156dc8f877cc29e1ca8d01c84e9a1e7917d49492", + "client_ttft_s": 2.133382400003029, + "client_elapsed_s": 3.8481876000005286, + "finish_reason": "length", + "usage": { + "prompt_tokens": 4096, + "completion_tokens": 256, + "total_tokens": 4352, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "engine": { + "projection": null, + "time": 1791413446.6427827, + "duration_s": 3.8, + "finish": "length", + "prompt_tokens": 4096, + "reused": 0, + "output_tokens": 256, + "prompt_total": 4096, + "prompt_read": 4096, + "reasoning_recoveries": 0, + "reasoning_repeat_coverage": 0.0, + "engine_generated": 256, + "prompt_ms": 2100.8, + "decode_ms": 1723.9, + "decode_tok_s": 148.5, + "hit_rate": 0.97, + "pcie_share": 0.0, + "ram_blobs": 0, + "file_blobs": 0, + "file_mb": 6200.1, + "drafts_offered": 181, + "drafts_accepted": 144 + }, + "memory": { + "peak_host_ram_used_gib": 50.01, + "peak_gpu_memory_used_mib": 23215 + }, + "text": "This synthetic Python module presents a fascinating case study in algorithmic redundancy, mathematical patterns, and software maintainability. At first glance, the code appears to be a collection of 113 distinct functions, each named `task_XXXXX`, where `XXXXX` is a zero-padded integer index. However, a deeper analysis reveals that these functions are not unique algorithms but rather instances of a single, generalized mathematical formula instantiated with specific parameters. This observation forms the core of the critique regarding the module's design, efficiency, and purpose.\n\n### The Mathematical Core: Deterministic Linear Transforms\n\nEvery function in the module follows the exact same structural pattern:\n```python\ndef task_N(value: int) -> int:\n return (value * A + B) % M\n```\nWhere:\n- `value` is the input integer.\n- `A` is the multiplier.\n- `B` is the additive constant.\n- `M` is the modulus, which is consistently `100003` across all functions.\n\nThe modulus `100003` is significant. It is a prime number. In modular arithmetic, working with a prime modulus ensures that the field of integers modulo `M`" + }, + { + "label": "tokens-32768-run-1", + "status": "ok", + "expected_prompt_tokens": 32768, + "request_sha256": "e37cfc2ab9867d91130927440df24a57255a736b8d0d829d07d828e22e3de572", + "client_ttft_s": 11.329635700007202, + "client_elapsed_s": 13.145479600003455, + "finish_reason": "length", + "usage": { + "prompt_tokens": 32768, + "completion_tokens": 256, + "total_tokens": 33024, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "engine": { + "projection": null, + "time": 1791413451.689636, + "duration_s": 13.1, + "finish": "length", + "prompt_tokens": 32768, + "reused": 0, + "output_tokens": 256, + "prompt_total": 32768, + "prompt_read": 32768, + "reasoning_recoveries": 0, + "reasoning_repeat_coverage": 0.0, + "engine_generated": 256, + "prompt_ms": 11272.9, + "decode_ms": 1827.0, + "decode_tok_s": 140.1, + "hit_rate": 0.968, + "pcie_share": 0.0, + "ram_blobs": 0, + "file_blobs": 0, + "file_mb": 6148.9, + "drafts_offered": 169, + "drafts_accepted": 143 + }, + "memory": { + "peak_host_ram_used_gib": 50.13, + "peak_gpu_memory_used_mib": 23215 + }, + "text": "This synthetic Python module presents a highly repetitive, yet structurally coherent, collection of functions designed to perform deterministic integer transformations. At its core, the module defines a series of functions named `task_00000` through `task_00888` (and potentially beyond, as the snippet cuts off), each adhering to a strict mathematical pattern. The code serves as a benchmark for evaluating how automated systems handle large-scale, repetitive code generation, as well as a case study in the trade-offs between explicit definition and algorithmic abstraction.\n\n### Mathematical Structure and Deterministic Transforms\n\nThe fundamental operation in every function is a linear transformation followed by a modulo operation. The general form is:\n\n$$ f_i(x) = (x \\cdot m_i + c_i) \\pmod{100003} $$\n\nWhere:\n* $x$ is the input integer (`value`).\n* $m_i$ is the multiplier for task $i$.\n* $c_i$ is the additive constant for task $i$.\n* $100003$ is the modulus, which is a prime number.\n\nThe choice of $100003$ as the modulus is" + }, + { + "label": "tokens-32768-run-2", + "status": "ok", + "expected_prompt_tokens": 32768, + "request_sha256": "ed2dfaa11aecad9f4563f33bb5f5d4b8704486508700b8fa572722cc1225ee26", + "client_ttft_s": 11.426643400016474, + "client_elapsed_s": 13.176572700001998, + "finish_reason": "length", + "usage": { + "prompt_tokens": 32768, + "completion_tokens": 256, + "total_tokens": 33024, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "engine": { + "projection": null, + "time": 1791413466.0680475, + "duration_s": 13.1, + "finish": "length", + "prompt_tokens": 32768, + "reused": 0, + "output_tokens": 256, + "prompt_total": 32768, + "prompt_read": 32768, + "reasoning_recoveries": 0, + "reasoning_repeat_coverage": 0.0, + "engine_generated": 256, + "prompt_ms": 11342.6, + "decode_ms": 1761.6, + "decode_tok_s": 145.3, + "hit_rate": 0.981, + "pcie_share": 0.0, + "ram_blobs": 0, + "file_blobs": 0, + "file_mb": 3753.2, + "drafts_offered": 170, + "drafts_accepted": 145 + }, + "memory": { + "peak_host_ram_used_gib": 50.64, + "peak_gpu_memory_used_mib": 23215 + }, + "text": "This synthetic Python module presents a highly repetitive, yet structurally coherent, collection of functions designed to perform deterministic integer transformations. At its core, the module defines a series of functions named `task_00000` through `task_00888` (and potentially beyond, as the snippet cuts off), each adhering to a strict mathematical pattern. The code serves as a benchmark for evaluating how automated systems handle large-scale, repetitive code generation, static analysis, and execution performance.\n\n### Mathematical Structure and Deterministic Transforms\n\nThe fundamental operation in every function is a linear affine transformation followed by a modulo operation. The general form is:\n\n$$ f(x) = (x \\cdot a + b) \\pmod{100003} $$\n\nWhere:\n- $x$ is the input integer (`value`).\n- $a$ is the multiplier coefficient.\n- $b$ is the additive constant.\n- $100003$ is the modulus, which is a prime number.\n\nThe choice of $100003$ as the modulus is significant. In modular arithmetic, using a prime modulus ensures that the ring $\\mathbb{Z}_{100003" + }, + { + "label": "tokens-32768-run-3", + "status": "ok", + "expected_prompt_tokens": 32768, + "request_sha256": "2a3c7342622e80c3cd8b37e062f0a2f64640c119ece8cccbf168639041eac079", + "client_ttft_s": 11.384466100018471, + "client_elapsed_s": 13.145414400001755, + "finish_reason": "length", + "usage": { + "prompt_tokens": 32768, + "completion_tokens": 256, + "total_tokens": 33024, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "engine": { + "projection": null, + "time": 1791413480.5209217, + "duration_s": 13.1, + "finish": "length", + "prompt_tokens": 32768, + "reused": 0, + "output_tokens": 256, + "prompt_total": 32768, + "prompt_read": 32768, + "reasoning_recoveries": 0, + "reasoning_repeat_coverage": 0.0, + "engine_generated": 256, + "prompt_ms": 11325.6, + "decode_ms": 1772.5, + "decode_tok_s": 144.4, + "hit_rate": 0.98, + "pcie_share": 0.0, + "ram_blobs": 0, + "file_blobs": 0, + "file_mb": 3970.3, + "drafts_offered": 168, + "drafts_accepted": 140 + }, + "memory": { + "peak_host_ram_used_gib": 51.11, + "peak_gpu_memory_used_mib": 23215 + }, + "text": "This synthetic Python module presents a highly repetitive, yet structurally coherent, collection of functions designed to perform deterministic integer transformations. At its core, the module defines a series of functions named `task_00000` through `task_00888` (and potentially beyond, as the snippet cuts off), each accepting a single integer argument `value` and returning an integer result. The underlying logic is uniform across all functions: it applies a linear transformation to the input and then reduces the result modulo a fixed prime number, 100003.\n\n### The Mathematical Core: Linear Transformation and Modulo Arithmetic\n\nThe fundamental operation in every function is `(value * multiplier + offset) % 100003`. This is a classic example of an affine transformation in modular arithmetic. In mathematics, an affine transformation is a linear transformation followed by a translation. Here, the \"linear\" part is the multiplication by a specific coefficient, and the \"translation\" is the addition of a constant offset.\n\nThe choice of the modulus, 100003, is significant. 100003 is a prime number. In modular arithmetic, working with a prime modulus ensures that the ring of integers modulo $" + }, + { + "label": "tokens-128000-run-1", + "status": "ok", + "expected_prompt_tokens": 128000, + "request_sha256": "3235aac3f934cf1d889c25973ca172b355509dc2522fdfb463fbefe979a9075d", + "client_ttft_s": 45.88402260001749, + "client_elapsed_s": 47.82689720002236, + "finish_reason": "length", + "usage": { + "prompt_tokens": 128000, + "completion_tokens": 256, + "total_tokens": 128256, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "engine": { + "projection": null, + "time": 1791413497.4171362, + "duration_s": 47.7, + "finish": "length", + "prompt_tokens": 128000, + "reused": 0, + "output_tokens": 256, + "prompt_total": 128000, + "prompt_read": 128000, + "reasoning_recoveries": 0, + "reasoning_repeat_coverage": 0.0, + "engine_generated": 256, + "prompt_ms": 45696.8, + "decode_ms": 1954.5, + "decode_tok_s": 131.0, + "hit_rate": 0.984, + "pcie_share": 0.0, + "ram_blobs": 0, + "file_blobs": 0, + "file_mb": 3149.1, + "drafts_offered": 165, + "drafts_accepted": 128 + }, + "memory": { + "peak_host_ram_used_gib": 51.54, + "peak_gpu_memory_used_mib": 23215 + }, + "text": "This synthetic Python module presents a highly repetitive, yet structurally coherent, collection of functions designed to perform deterministic integer transformations. At its core, the module defines a series of functions named `task_XXXXX`, where `XXXXX` is a sequential integer identifier. Each function accepts a single integer argument `value` and returns an integer result derived from a linear transformation followed by a modular reduction.\n\n### The Mathematical Core: Linear Transformation and Modulo Arithmetic\n\nThe fundamental operation in every function is identical in structure: `(value * multiplier + offset) % modulus`. Here, the modulus is consistently set to `100003`. This specific number is significant because it is a prime number. In modular arithmetic, using a prime modulus ensures that the ring of integers modulo `p` is a field, which has desirable properties for cryptographic applications and error-correcting codes. However, in this synthetic context, the choice of `100003` likely serves as a benchmark to test the performance of modulo operations on large integers or to ensure results stay within a specific range.\n\nThe transformation is an affine map. For any given function `task_N`, the multiplier and offset are determined by the function's index. For the first 97 functions (`" + }, + { + "label": "tokens-128000-run-2", + "status": "ok", + "expected_prompt_tokens": 128000, + "request_sha256": "5f4b8bc9f0afc7765320abd89960c0938e0d7cf5974e962fdbd0613fef31c8d4", + "client_ttft_s": 46.246270400006324, + "client_elapsed_s": 48.53999679998378, + "finish_reason": "length", + "usage": { + "prompt_tokens": 128000, + "completion_tokens": 256, + "total_tokens": 128256, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "engine": { + "projection": null, + "time": 1791413549.0017734, + "duration_s": 48.1, + "finish": "length", + "prompt_tokens": 128000, + "reused": 0, + "output_tokens": 256, + "prompt_total": 128000, + "prompt_read": 128000, + "reasoning_recoveries": 0, + "reasoning_repeat_coverage": 0.0, + "engine_generated": 256, + "prompt_ms": 46041.8, + "decode_ms": 2078.2, + "decode_tok_s": 123.2, + "hit_rate": 0.987, + "pcie_share": 0.0, + "ram_blobs": 0, + "file_blobs": 0, + "file_mb": 2583.7, + "drafts_offered": 163, + "drafts_accepted": 118 + }, + "memory": { + "peak_host_ram_used_gib": 53.28, + "peak_gpu_memory_used_mib": 23215 + }, + "text": "This synthetic Python module presents a highly repetitive, yet structurally coherent, collection of functions designed to perform deterministic integer transformations. At its core, the module defines a series of functions named `task_XXXXX`, where `XXXXX` is a sequential integer identifier. Each function accepts a single integer argument `value` and returns an integer result derived from a linear transformation followed by a modular reduction.\n\n### The Mathematical Core: Linear Transformation and Modulo Arithmetic\n\nThe fundamental operation in every function is identical in structure: `(value * multiplier + offset) % modulus`. Here, the modulus is consistently set to `100003`. This specific number is significant because it is a prime number. In modular arithmetic, using a prime modulus ensures that the ring of integers modulo `p` is a field, which has desirable properties for cryptographic applications and error-correcting codes. However, in this synthetic context, the choice of `100003` likely serves as a fixed \"hash space\" or a constraint to keep the output within a specific range, preventing integer overflow in languages with fixed-width integers (though Python handles arbitrary precision integers, the modulo operation still bounds the result).\n\nThe transformation is an affine map: $f(x) = ax +" + }, + { + "label": "tokens-128000-run-3", + "status": "ok", + "expected_prompt_tokens": 128000, + "request_sha256": "5c3fb71db9f9f44765f28389d1032f2c8b6687d1bf5ca6f33aeb924072cfa6e4", + "client_ttft_s": 46.70187610000721, + "client_elapsed_s": 48.513164699979825, + "finish_reason": "length", + "usage": { + "prompt_tokens": 128000, + "completion_tokens": 256, + "total_tokens": 128256, + "prompt_tokens_details": { + "cached_tokens": 0 + } + }, + "engine": { + "projection": null, + "time": 1791413601.2886786, + "duration_s": 48.3, + "finish": "length", + "prompt_tokens": 128000, + "reused": 0, + "output_tokens": 256, + "prompt_total": 128000, + "prompt_read": 128000, + "reasoning_recoveries": 0, + "reasoning_repeat_coverage": 0.0, + "engine_generated": 256, + "prompt_ms": 46498.9, + "decode_ms": 1823.0, + "decode_tok_s": 140.4, + "hit_rate": 0.977, + "pcie_share": 0.0, + "ram_blobs": 0, + "file_blobs": 0, + "file_mb": 4380.8, + "drafts_offered": 167, + "drafts_accepted": 144 + }, + "memory": { + "peak_host_ram_used_gib": 54.97, + "peak_gpu_memory_used_mib": 23215 + }, + "text": "This synthetic Python module presents a highly repetitive, yet structurally coherent, collection of functions designed to perform deterministic integer transformations. At its core, the module defines a series of functions named `task_XXXXX`, where `XXXXX` is a sequential integer identifier. Each function accepts a single integer argument `value` and returns an integer result derived from a linear transformation followed by a modulo operation.\n\n### 1. The Mathematical Core: Linear Transformation and Modulo Arithmetic\n\nThe fundamental operation in every function is identical in structure:\n```python\nreturn (value * multiplier + offset) % 100003\n```\n\nThis is a classic **affine transformation** (or linear congruential generator step) applied to an integer. The transformation consists of two components:\n1. **Multiplication:** The input `value` is scaled by a `multiplier`.\n2. **Addition:** A constant `offset` is added to the scaled value.\n3. **Modulo:** The result is constrained to the range `[0, 100002]` by taking the remainder when divided by `100003`.\n\nThe modulus `100003` is a prime" + } +] \ No newline at end of file diff --git a/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/strata-q2_0.json b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/strata-q2_0.json new file mode 100644 index 00000000000..a36d76557c5 --- /dev/null +++ b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/strata-q2_0.json @@ -0,0 +1,60 @@ +{ + "exe": "\\engine\\strata.exe", + "args": [ + "--pack", + "\\packs\\q2_0", + "--native", + "\\Q2_0\\Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00001-of-00002.gguf", + "--ple-gguf", + "\\Q2_0\\Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00002-of-00002.gguf", + "--expert-profile", + "\\data\\expert-profile.bin", + "--expert-cache", + "auto", + "--prefill", + "auto", + "--spec", + "4", + "--spec-min-p", + "0.7", + "--mtp", + "\\mtp\\rt", + "--max-context", + "131072", + "--kv", + "int8", + "--kv-resident", + "32768", + "--vision", + "--vram-reserve-mib", + "1000", + "--mmap-experts", + "--pool-workers", + "10", + "--pcie-frac", + "0", + "--conversation-cache-mib", + "6144", + "--conversation-cache-slots", + "2" + ], + "cwd": "", + "tokenizer": "\\packs\\q2_0\\tokenizer", + "model_name": "qwen3.8-flash-next-q2_0", + "log": "\\strata-community.log", + "lib_dirs": [ + "\\bin" + ], + "port": 8095, + "gpu": 0, + "gpus_asked": true, + "vision": { + "exe": "\\engine\\strata-vision.exe", + "mmproj": "\\Q2_0\\mmproj-Qwen3.8-Flash-Next-BF16.gguf", + "model": "\\Q2_0\\Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00001-of-00002.gguf", + "gpu": true, + "max_tokens": 1024 + }, + "reasoning_budget_tokens": 12000, + "reasoning_loop_recovery": "stop" +} \ No newline at end of file diff --git a/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/summary.json b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/summary.json new file mode 100644 index 00000000000..1bea6d3a23e --- /dev/null +++ b/bench/results/2026-10-08-community-rtx-5090-laptop-engine-0.1.40.3/summary.json @@ -0,0 +1,149 @@ +{ + "4096": { + "runs_ok": 3, + "failed": [], + "prompt_tokens": { + "median": 4096, + "min": 4096, + "max": 4096 + }, + "cached_tokens": { + "median": 0, + "min": 0, + "max": 0 + }, + "output_tokens": { + "median": 256, + "min": 256, + "max": 256 + }, + "client_ttft_s": { + "median": 2.133382400003029, + "min": 2.1244121000054292, + "max": 2.301761800015811 + }, + "client_elapsed_s": { + "median": 4.082541699986905, + "min": 3.8481876000005286, + "max": 4.3289522000122815 + }, + "prefill_tok_s": { + "median": 1949.7334348819497, + "min": 1805.5984130482698, + "max": 1958.5903504996888 + }, + "decode_tok_s": { + "median": 138.08727547332649, + "min": 125.53327122051685, + "max": 148.5004930680434 + }, + "peak_host_ram_used_gib": { + "median": 49.97, + "min": 49.68, + "max": 50.01 + }, + "peak_gpu_memory_used_mib": { + "median": 23215, + "min": 23215, + "max": 23215 + } + }, + "32768": { + "runs_ok": 3, + "failed": [], + "prompt_tokens": { + "median": 32768, + "min": 32768, + "max": 32768 + }, + "cached_tokens": { + "median": 0, + "min": 0, + "max": 0 + }, + "output_tokens": { + "median": 256, + "min": 256, + "max": 256 + }, + "client_ttft_s": { + "median": 11.384466100018471, + "min": 11.329635700007202, + "max": 11.426643400016474 + }, + "client_elapsed_s": { + "median": 13.145479600003455, + "min": 13.145414400001755, + "max": 13.176572700001998 + }, + "prefill_tok_s": { + "median": 2893.2683478138024, + "min": 2888.9319909015567, + "max": 2906.7941700893293 + }, + "decode_tok_s": { + "median": 144.42877291960508, + "min": 140.12041598248496, + "max": 145.32243415077204 + }, + "peak_host_ram_used_gib": { + "median": 50.64, + "min": 50.13, + "max": 51.11 + }, + "peak_gpu_memory_used_mib": { + "median": 23215, + "min": 23215, + "max": 23215 + } + }, + "128000": { + "runs_ok": 3, + "failed": [], + "prompt_tokens": { + "median": 128000, + "min": 128000, + "max": 128000 + }, + "cached_tokens": { + "median": 0, + "min": 0, + "max": 0 + }, + "output_tokens": { + "median": 256, + "min": 256, + "max": 256 + }, + "client_ttft_s": { + "median": 46.246270400006324, + "min": 45.88402260001749, + "max": 46.70187610000721 + }, + "client_elapsed_s": { + "median": 48.513164699979825, + "min": 47.82689720002236, + "max": 48.53999679998378 + }, + "prefill_tok_s": { + "median": 2780.0824468200635, + "min": 2752.753290938065, + "max": 2801.071409814254 + }, + "decode_tok_s": { + "median": 130.97979022767973, + "min": 123.18352420363777, + "max": 140.42786615469007 + }, + "peak_host_ram_used_gib": { + "median": 53.28, + "min": 51.54, + "max": 54.97 + }, + "peak_gpu_memory_used_mib": { + "median": 23215, + "min": 23215, + "max": 23215 + } + } +} \ No newline at end of file