From 864bf374403bbd35adfd20a2c930e316bebd6ed3 Mon Sep 17 00:00:00 2001 From: beanz Date: Mon, 12 Oct 2026 01:37:50 -0700 Subject: [PATCH 1/2] bench: community report, RTX 5060 Ti 16 GB + RTX 3060 12 GB unequal layer split (PCIe gen5 x8 + gen3 x4), IQ3_S, engine 0.1.43: auto planner K=3 costs 25% of 32K prompt; the printed --layer-split suggestion is in the reordered frame (literal 6: -76%, as-given 42: a wash, worse hit rate); hit rate tracks per-layer coverage, not held count (8,842 vs 5,935 experts, p=0.93); ADAPT_OVERLAP null on the split; ROUTE_TAIL_SKIP +5.2% not +10-20%, decaying with context, and hit_rate's denominator excludes skipped misses; needles 9/9 at 32K/128K/256K; #2062 confirmed and extended --- .../A/A-after-full.json | 1 + .../A/A-after.json | 1 + .../A/A-before.json | 1 + .../A/README.md | 150 +++ .../A/results.json | 1016 ++++++++++++++++ .../B1/B1-after-full.json | 1 + .../B1/B1-after.json | 1 + .../B1/B1-before.json | 1 + .../B1/README.md | 129 +++ .../B1/config.json | 44 + .../B1/results.json | 999 ++++++++++++++++ .../B1c/B1c-after-full.json | 1 + .../B1c/B1c-after.json | 1 + .../B1c/B1c-before.json | 1 + .../B1c/README.md | 157 +++ .../B1c/config.json | 44 + .../B1c/results.json | 999 ++++++++++++++++ .../B1d/B1d-after-full.json | 1 + .../B1d/B1d-after.json | 1 + .../B1d/B1d-before.json | 1 + .../B1d/README.md | 145 +++ .../B1d/config.json | 64 ++ .../B1d/results.json | 1018 +++++++++++++++++ .../C/C-after-full.json | 1 + .../C/C-after.json | 1 + .../C/C-before.json | 1 + .../C/README.md | 112 ++ .../C/config.json | 64 ++ .../C/results.json | 1018 +++++++++++++++++ .../D/D-after-full.json | 1 + .../D/D-after.json | 1 + .../D/D-before.json | 1 + .../D/README.md | 144 +++ .../D/config.json | 64 ++ .../D/results.json | 1018 +++++++++++++++++ .../NOTES.md | 19 + .../README.md | 145 +++ .../needles/README.md | 78 ++ .../needles/needles-final.json | 83 ++ .../sha256sums.txt | 5 + 40 files changed, 7533 insertions(+) create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/A/A-after-full.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/A/A-after.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/A/A-before.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/A/README.md create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/A/results.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/B1-after-full.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/B1-after.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/B1-before.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/README.md create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/config.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/results.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/B1c-after-full.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/B1c-after.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/B1c-before.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/README.md create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/config.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/results.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/B1d-after-full.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/B1d-after.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/B1d-before.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/README.md create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/config.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/results.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/C-after-full.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/C-after.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/C-before.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/README.md create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/config.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/results.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/D-after-full.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/D-after.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/D-before.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/README.md create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/config.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/results.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/NOTES.md create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/README.md create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/needles/README.md create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/needles/needles-final.json create mode 100644 bench/results/2026-10-11-community-rtx-5060ti-3060-split/sha256sums.txt diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/A/A-after-full.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/A/A-after-full.json new file mode 100644 index 00000000000..92fb511ff69 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/A/A-after-full.json @@ -0,0 +1 @@ +{"engine": {"model": "qwen3.8-flash-next-iq3_s", "max_context": 524288, "images": true, "version": "0.1.43", "logit_bias": 1, "context": 524288, "kv": "int8", "kv_resident": 20480, "expert_slots": 8842, "expert_cache_mib": 17910, "expert_slots_primary": 4984, "expert_cache_primary_mib": 9474, "spec": 6, "mtp_max": 4, "lookup": 3, "vram_free_mib": 324, "cvec": 0, "arena_mib": 47962, "pool_workers": 11, "pcie_frac": "0.00", "spec_min_p": "0.70", "conversation_cache_mib": 0, "conversation_cache_slots": 4, "conversation_cache_min_free_mib": 2560, "conversation_cache_min_tokens": 0, "tail_role_token": -1, "vram_elastic": 0, "engine": "0.1.43"}, "live": {"state": "idle", "queued": 0, "phase": null, "prompt_tokens": null, "prompt_read": null, "prompt_total": null, "generated": null, "max_tokens": null, "elapsed_s": null, "tok_s": null, "tok_s_mean": null, "prefill_tok_s_mean": null, "tok_s_window_s": null}, "requests": [{"projection": null, "time": 1791782114.5695024, "duration_s": 19.1, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 18627.7, "decode_ms": 421.7, "decode_tok_s": 56.9, "hit_rate": 0.958, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 22, "drafts_accepted": 12}, {"projection": null, "time": 1791782095.4778943, "duration_s": 19.1, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 18615.1, "decode_ms": 451.2, "decode_tok_s": 53.2, "hit_rate": 0.926, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 23, "drafts_accepted": 11}, {"projection": null, "time": 1791782076.331768, "duration_s": 19.1, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 18624.4, "decode_ms": 497.4, "decode_tok_s": 48.3, "hit_rate": 0.806, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 24, "drafts_accepted": 10}, {"projection": null, "time": 1791782071.402105, "duration_s": 4.9, "finish": "stop", "prompt_tokens": 14418, "reused": 11572, "output_tokens": 51, "prompt_total": 14418, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 51, "prompt_ms": 4195.5, "decode_ms": 710.0, "decode_tok_s": 71.8, "hit_rate": 0.908, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 42, "drafts_accepted": 26}, {"projection": null, "time": 1791782067.1535203, "duration_s": 4.2, "finish": "stop", "prompt_tokens": 11570, "reused": 8710, "output_tokens": 3, "prompt_total": 11570, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 3, "prompt_ms": 4183.3, "decode_ms": 47.0, "decode_tok_s": 63.8, "hit_rate": 0.969, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 2, "drafts_accepted": 1}, {"projection": null, "time": 1791782062.3347528, "duration_s": 4.8, "finish": "stop", "prompt_tokens": 8692, "reused": 5311, "output_tokens": 19, "prompt_total": 8692, "prompt_read": 3381, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 19, "prompt_ms": 4502.8, "decode_ms": 304.7, "decode_tok_s": 62.4, "hit_rate": 0.92, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 19, "drafts_accepted": 8}, {"projection": null, "time": 1791782058.1376424, "duration_s": 4.2, "finish": "length", "prompt_tokens": 5248, "reused": 3708, "output_tokens": 64, "prompt_total": 5248, "prompt_read": 1540, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 3344.1, "decode_ms": 843.2, "decode_tok_s": 75.9, "hit_rate": 0.88, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 55, "drafts_accepted": 35}, {"projection": null, "time": 1791782053.1116652, "duration_s": 5.0, "finish": "length", "prompt_tokens": 3715, "reused": 633, "output_tokens": 64, "prompt_total": 3715, "prompt_read": 3082, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 4271.3, "decode_ms": 745.8, "decode_tok_s": 85.8, "hit_rate": 0.907, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 51, "drafts_accepted": 41}, {"projection": null, "time": 1791782050.7852345, "duration_s": 2.3, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 24, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 2008.9, "decode_ms": 306.3, "decode_tok_s": 78.4, "hit_rate": 0.812, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 25, "drafts_accepted": 15}, {"projection": null, "time": 1791782046.2375796, "duration_s": 4.5, "finish": "stop", "prompt_tokens": 14421, "reused": 11575, "output_tokens": 18, "prompt_total": 14421, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 18, "prompt_ms": 4216.4, "decode_ms": 319.4, "decode_tok_s": 56.4, "hit_rate": 0.912, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 21, "drafts_accepted": 7}, {"projection": null, "time": 1791782041.9197686, "duration_s": 4.3, "finish": "stop", "prompt_tokens": 11569, "reused": 8709, "output_tokens": 7, "prompt_total": 11569, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 7, "prompt_ms": 4219.4, "decode_ms": 86.5, "decode_tok_s": 80.9, "hit_rate": 0.962, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 6, "drafts_accepted": 4}, {"projection": null, "time": 1791782037.1317878, "duration_s": 4.8, "finish": "stop", "prompt_tokens": 8691, "reused": 5310, "output_tokens": 19, "prompt_total": 8691, "prompt_read": 3381, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 19, "prompt_ms": 4544.4, "decode_ms": 230.9, "decode_tok_s": 82.3, "hit_rate": 0.925, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 15, "drafts_accepted": 11}, {"projection": null, "time": 1791782033.0626135, "duration_s": 4.1, "finish": "length", "prompt_tokens": 5247, "reused": 3777, "output_tokens": 64, "prompt_total": 5247, "prompt_read": 1470, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 3284.5, "decode_ms": 774.6, "decode_tok_s": 82.6, "hit_rate": 0.887, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 56, "drafts_accepted": 38}, {"projection": null, "time": 1791782028.0836315, "duration_s": 5.0, "finish": "length", "prompt_tokens": 3714, "reused": 634, "output_tokens": 64, "prompt_total": 3714, "prompt_read": 3080, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 4286.5, "decode_ms": 683.7, "decode_tok_s": 93.6, "hit_rate": 0.894, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 49, "drafts_accepted": 41}, {"projection": null, "time": 1791782025.7439706, "duration_s": 2.3, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 23, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 23, "prompt_ms": 2011.3, "decode_ms": 321.1, "decode_tok_s": 71.6, "hit_rate": 0.842, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 24, "drafts_accepted": 15}, {"projection": null, "time": 1791782021.2054064, "duration_s": 4.5, "finish": "stop", "prompt_tokens": 14425, "reused": 11579, "output_tokens": 15, "prompt_total": 14425, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 15, "prompt_ms": 4245.8, "decode_ms": 282.3, "decode_tok_s": 53.1, "hit_rate": 0.914, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 17, "drafts_accepted": 5}, {"projection": null, "time": 1791782016.7466705, "duration_s": 4.4, "finish": "stop", "prompt_tokens": 11568, "reused": 8708, "output_tokens": 12, "prompt_total": 11568, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 12, "prompt_ms": 4304.3, "decode_ms": 140.5, "decode_tok_s": 85.4, "hit_rate": 0.935, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 9, "drafts_accepted": 7}, {"projection": null, "time": 1791782011.8003876, "duration_s": 4.9, "finish": "stop", "prompt_tokens": 8689, "reused": 5308, "output_tokens": 20, "prompt_total": 8689, "prompt_read": 3381, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 20, "prompt_ms": 4633.0, "decode_ms": 297.8, "decode_tok_s": 67.2, "hit_rate": 0.869, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 16, "drafts_accepted": 9}, {"projection": null, "time": 1791782007.6404397, "duration_s": 4.1, "finish": "length", "prompt_tokens": 5245, "reused": 3775, "output_tokens": 64, "prompt_total": 5245, "prompt_read": 1470, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 3353.0, "decode_ms": 793.4, "decode_tok_s": 80.7, "hit_rate": 0.83, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 55, "drafts_accepted": 38}, {"projection": null, "time": 1791782002.466468, "duration_s": 5.2, "finish": "length", "prompt_tokens": 3712, "reused": 631, "output_tokens": 64, "prompt_total": 3712, "prompt_read": 3081, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 4438.3, "decode_ms": 723.4, "decode_tok_s": 88.5, "hit_rate": 0.721, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 49, "drafts_accepted": 42}, {"projection": null, "time": 1791782000.1430116, "duration_s": 2.3, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 21, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 21, "prompt_ms": 2000.0, "decode_ms": 305.8, "decode_tok_s": 68.7, "hit_rate": 0.752, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 22, "drafts_accepted": 13}, {"projection": null, "time": 1791781988.3488822, "duration_s": 11.8, "finish": "length", "prompt_tokens": 16058, "reused": 0, "output_tokens": 24, "prompt_total": 16058, "prompt_read": 16058, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 11367.1, "decode_ms": 411.9, "decode_tok_s": 58.3, "hit_rate": 0.969, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 19, "drafts_accepted": 10}, {"projection": null, "time": 1791781976.5627692, "duration_s": 11.8, "finish": "length", "prompt_tokens": 16058, "reused": 0, "output_tokens": 24, "prompt_total": 16058, "prompt_read": 16058, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 11371.7, "decode_ms": 398.2, "decode_tok_s": 60.3, "hit_rate": 0.944, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 19, "drafts_accepted": 12}, {"projection": null, "time": 1791781964.7714674, "duration_s": 11.8, "finish": "length", "prompt_tokens": 16058, "reused": 0, "output_tokens": 24, "prompt_total": 16058, "prompt_read": 16058, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 11350.7, "decode_ms": 425.3, "decode_tok_s": 56.4, "hit_rate": 0.907, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 20, "drafts_accepted": 10}, {"projection": null, "time": 1791781959.7870252, "duration_s": 5.0, "finish": "length", "prompt_tokens": 4019, "reused": 0, "output_tokens": 24, "prompt_total": 4019, "prompt_read": 4019, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 4629.8, "decode_ms": 339.9, "decode_tok_s": 70.6, "hit_rate": 0.922, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 12, "drafts_accepted": 10}, {"projection": null, "time": 1791781954.8552415, "duration_s": 4.9, "finish": "length", "prompt_tokens": 4019, "reused": 0, "output_tokens": 24, "prompt_total": 4019, "prompt_read": 4019, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 4595.1, "decode_ms": 328.2, "decode_tok_s": 73.1, "hit_rate": 0.866, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 13, "drafts_accepted": 11}, {"projection": null, "time": 1791781949.8687265, "duration_s": 5.0, "finish": "length", "prompt_tokens": 4019, "reused": 0, "output_tokens": 24, "prompt_total": 4019, "prompt_read": 4019, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 4638.3, "decode_ms": 339.4, "decode_tok_s": 70.7, "hit_rate": 0.704, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 15, "drafts_accepted": 12}, {"projection": null, "time": 1791781948.1314867, "duration_s": 1.7, "finish": "length", "prompt_tokens": 46, "reused": 0, "output_tokens": 100, "prompt_total": 46, "prompt_read": 46, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 370.1, "decode_ms": 1358.6, "decode_tok_s": 73.6, "hit_rate": 0.833, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 74, "drafts_accepted": 50}, {"projection": null, "time": 1791781946.4257963, "duration_s": 1.7, "finish": "length", "prompt_tokens": 43, "reused": 0, "output_tokens": 100, "prompt_total": 43, "prompt_read": 43, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 343.9, "decode_ms": 1355.6, "decode_tok_s": 73.8, "hit_rate": 0.841, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 86, "drafts_accepted": 51}, {"projection": null, "time": 1791781944.875469, "duration_s": 1.5, "finish": "length", "prompt_tokens": 51, "reused": 0, "output_tokens": 100, "prompt_total": 51, "prompt_read": 51, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 418.7, "decode_ms": 1125.2, "decode_tok_s": 88.9, "hit_rate": 0.697, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 84, "drafts_accepted": 70}, {"projection": null, "time": 1791781943.1284473, "duration_s": 1.7, "finish": "length", "prompt_tokens": 46, "reused": 0, "output_tokens": 100, "prompt_total": 46, "prompt_read": 46, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 374.8, "decode_ms": 1366.0, "decode_tok_s": 73.2, "hit_rate": 0.833, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 76, "drafts_accepted": 50}, {"projection": null, "time": 1791781941.4607239, "duration_s": 1.7, "finish": "length", "prompt_tokens": 43, "reused": 0, "output_tokens": 100, "prompt_total": 43, "prompt_read": 43, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 348.6, "decode_ms": 1312.8, "decode_tok_s": 76.2, "hit_rate": 0.851, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 76, "drafts_accepted": 52}, {"projection": null, "time": 1791781939.9606142, "duration_s": 1.5, "finish": "length", "prompt_tokens": 51, "reused": 0, "output_tokens": 100, "prompt_total": 51, "prompt_read": 51, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 416.9, "decode_ms": 1076.8, "decode_tok_s": 92.9, "hit_rate": 0.728, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 80, "drafts_accepted": 70}, {"projection": null, "time": 1791781938.1940742, "duration_s": 1.8, "finish": "length", "prompt_tokens": 46, "reused": 0, "output_tokens": 100, "prompt_total": 46, "prompt_read": 46, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 394.4, "decode_ms": 1365.7, "decode_tok_s": 73.2, "hit_rate": 0.759, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 76, "drafts_accepted": 53}, {"projection": null, "time": 1791781936.4698443, "duration_s": 1.7, "finish": "length", "prompt_tokens": 43, "reused": 0, "output_tokens": 100, "prompt_total": 43, "prompt_read": 43, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 373.4, "decode_ms": 1343.9, "decode_tok_s": 74.4, "hit_rate": 0.775, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 76, "drafts_accepted": 52}, {"projection": null, "time": 1791781934.9259958, "duration_s": 1.5, "finish": "length", "prompt_tokens": 51, "reused": 0, "output_tokens": 100, "prompt_total": 51, "prompt_read": 51, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 425.3, "decode_ms": 1111.4, "decode_tok_s": 90.0, "hit_rate": 0.712, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 86, "drafts_accepted": 70}, {"projection": null, "time": 1791781932.7963, "duration_s": 2.1, "finish": "length", "prompt_tokens": 1028, "reused": 0, "output_tokens": 8, "prompt_total": 1028, "prompt_read": 1028, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 8, "prompt_ms": 1981.8, "decode_ms": 141.0, "decode_tok_s": 56.7, "hit_rate": 0.7, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 7, "drafts_accepted": 3}, {"projection": null, "time": 1791781930.8468044, "duration_s": 1.9, "finish": "length", "prompt_tokens": 906, "reused": 0, "output_tokens": 1, "prompt_total": 906, "prompt_read": 906, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 1, "prompt_ms": 1924.1, "decode_ms": 18.4, "decode_tok_s": 54.3, "hit_rate": 0.774, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 0, "drafts_accepted": 0}, {"projection": null, "time": 1791781929.3537009, "duration_s": 1.5, "finish": "length", "prompt_tokens": 260, "reused": 0, "output_tokens": 1, "prompt_total": 260, "prompt_read": 260, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 1, "prompt_ms": 1466.3, "decode_ms": 19.0, "decode_tok_s": 52.6, "hit_rate": 0.773, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 0, "drafts_accepted": 0}, {"projection": null, "time": 1791781928.606237, "duration_s": 0.7, "finish": "length", "prompt_tokens": 34, "reused": 0, "output_tokens": 24, "prompt_total": 34, "prompt_read": 34, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 379.7, "decode_ms": 357.4, "decode_tok_s": 67.2, "hit_rate": 0.55, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 18, "drafts_accepted": 17}], "conversation_cache": {"enabled": false, "budget_mib": 0, "slots": 4, "parked": 0, "bytes": 0, "evictions": 0, "parks": 0, "restores": 0, "last_event": null, "last_tokens": null, "last_at": null, "requests": 40, "requests_reused": 15, "reused_tokens": 89940, "prompt_tokens": 291777, "last_reused": 0, "last_prompt": 32049}, "requests_kept": 40, "totals": {"since": 1791781196.593416, "requests": 40, "prompt_tokens": 291777, "reused": 89940, "output_tokens": 1766, "prompt_ms": 181080.7, "decode_ms": 23481.4, "drafts_offered": 1439, "drafts_accepted": 992}, "hardware": {"gpus": [{"index": 0, "util": 0, "mem_used": 16836001792, "mem_total": 17103323136, "temp": 38, "power": 4.978}, {"index": 1, "util": 0, "mem_used": 12494372864, "mem_total": 12884901888, "temp": 48, "power": 7.95}], "gpu_util": 0.0, "gpu_mem_used": 29330374656, "gpu_mem_total": 29988225024, "gpu_temp": 48, "gpu_power": 12.928, "gpu_power_limit": 350.0, "gpu_pcie_gen": 1, "gpu_pcie_gen_max": 5, "gpu_pcie_width": 8, "gpu_pcie_rx_mb": 0.9453125, "gpu_pcie_tx_mb": 1.376953125, "cpu": 0.1, "ram_used": 65355415552, "ram_total": 66472062976, "disk_read_mb": 0.15788155029198064, "disk_write_mb": 0.0, "tok_s": 0.0, "tok_s_mean": 0, "prefill_tok_s_mean": 0.0}, "hardware_static": {"gpu_name": "NVIDIA GeForce RTX 5060 Ti + NVIDIA GeForce RTX 3060", "gpu_count": 2, "gpu_note": null, "cpu_name": "AMD Ryzen 9 7900X 12-Core Processor", "cores": 12, "threads": 24, "psutil": true}, "history": {"gpu_util": [100.0, 100.0, 50.0, 100.0, 99.0, 50.0, 50.0, 50.0, 55.0, 50.0, 19.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "gpu_mem_used": [29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656], "gpu_temp": [69, 70, 71, 70, 68, 70, 59, 59, 59, 56, 58, 55, 55, 54, 54, 53, 53, 52, 52, 52, 52, 51, 51, 51, 51, 51, 49, 49, 49, 49, 48, 49, 49, 49, 49, 49, 49, 49, 49, 49, 49, 49, 49, 49, 49, 49, 48, 49, 49, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48], "gpu_power": [261.36, 264.96, 230.88, 225.87, 265.02, 248.61, 165.01, 159.69, 138.49, 73.56, 112.27, 83.5, 66.97, 59.95, 56.92, 56.76, 56.48, 56.45, 56.37, 56.34, 56.25, 56.15, 56.05, 55.98, 55.85, 55.64, 42.57, 15.6, 15.11, 14.05, 13.62, 13.25, 13.25, 13.25, 13.23, 13.19, 13.21, 13.22, 13.17, 13.2, 13.15, 13.19, 13.24, 13.22, 13.16, 13.16, 13.2, 13.14, 13.17, 13.12, 13.13, 13.07, 13.07, 13.07, 13.07, 13.05, 13.04, 12.98, 12.91, 12.93], "gpu_pcie_rx_mb": [27620.7, 29526.81, 3362.14, 17883.41, 13661.52, 23955.87, 3291.81, 226.95, 3291.48, 3309.58, 0.82, 0.78, 0.79, 0.79, 0.78, 0.83, 0.88, 0.79, 0.79, 0.78, 0.79, 0.82, 0.85, 0.78, 0.78, 1.11, 0.99, 1.03, 1.2, 1.01, 0.94, 0.94, 0.93, 1.02, 0.98, 0.96, 1.01, 1.01, 1.0, 0.93, 1.01, 0.94, 0.99, 0.93, 1.0, 1.04, 1.0, 1.0, 0.99, 0.93, 0.99, 1.0, 0.94, 1.04, 0.95, 0.97, 0.93, 1.0, 0.99, 0.95], "cpu": [4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 22.5, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 0.1, 0.1, 0.0, 0.1, 0.0, 0.1, 0.0, 0.1, 0.0, 0.1, 0.0, 0.1, 0.0, 0.2, 0.0, 0.1, 0.1, 0.0, 0.1, 0.1, 0.1, 0.0, 0.0, 0.1, 0.1, 0.0, 0.1, 0.4, 0.0, 0.1, 0.0, 0.2, 0.0, 0.1, 0.0, 0.1, 0.1, 0.0, 0.1, 0.0, 0.1, 0.0, 0.1], "ram_used": [65238073344, 65241382912, 65241686016, 65241686016, 65241686016, 65241686016, 65241686016, 65241686016, 65339961344, 65339961344, 65340436480, 65340665856, 65340665856, 65340665856, 65340665856, 65340657664, 65340657664, 65340915712, 65340915712, 65340915712, 65340915712, 65341173760, 65341173760, 65341173760, 65340960768, 65340919808, 65340919808, 65340919808, 65340919808, 65340919808, 65340919808, 65340919808, 65340919808, 65340919808, 65340919808, 65340919808, 65340919808, 65340919808, 65340919808, 65340919808, 65340919808, 65340919808, 65340919808, 65340919808, 65354366976, 65354924032, 65354924032, 65354924032, 65355169792, 65355169792, 65355169792, 65355169792, 65355165696, 65355157504, 65355157504, 65355157504, 65355157504, 65355157504, 65355157504, 65355415552], "disk_read_mb": [0.16, 0.16, 0.16, 0.16, 0.16, 0.14, 0.16, 0.16, 0.16, 0.16, 2.65, 0.16, 0.16, 0.16, 0.16, 0.14, 0.16, 0.16, 0.18, 0.16, 0.16, 0.14, 0.16, 0.43, 3.16, 0.16, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.14, 0.16, 0.16, 23.86, 0.16, 0.16, 0.16, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.15, 0.16, 0.16, 0.29, 0.16], "tok_s": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "prefill_tok_s_mean": [1394.4, 1672.2, 1672.2, 1672.2, 1672.2, 1823.2, 1823.2, 1823.2, 1896.9, 1896.9, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}, "time": 1791782187.574169} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/A/A-after.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/A/A-after.json new file mode 100644 index 00000000000..f5ebff69c46 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/A/A-after.json @@ -0,0 +1 @@ +{"engine": {"model": "qwen3.8-flash-next-iq3_s", "max_context": 524288, "images": true, "version": "0.1.43", "logit_bias": 1, "context": 524288, "kv": "int8", "kv_resident": 20480, "expert_slots": 8842, "expert_cache_mib": 17910, "expert_slots_primary": 4984, "expert_cache_primary_mib": 9474, "spec": 6, "mtp_max": 4, "lookup": 3, "vram_free_mib": 324, "cvec": 0, "arena_mib": 47962, "pool_workers": 11, "pcie_frac": "0.00", "spec_min_p": "0.70", "conversation_cache_mib": 0, "conversation_cache_slots": 4, "conversation_cache_min_free_mib": 2560, "conversation_cache_min_tokens": 0, "tail_role_token": -1, "vram_elastic": 0, "engine": "0.1.43"}, "live": {"state": "idle", "queued": 0, "phase": null, "prompt_tokens": null, "prompt_read": null, "prompt_total": null, "generated": null, "max_tokens": null, "elapsed_s": null, "tok_s": null, "tok_s_mean": null, "prefill_tok_s_mean": null, "tok_s_window_s": null}, "requests": [{"projection": null, "time": 1791782114.5695024, "duration_s": 19.1, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 18627.7, "decode_ms": 421.7, "decode_tok_s": 56.9, "hit_rate": 0.958, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 22, "drafts_accepted": 12}, {"projection": null, "time": 1791782095.4778943, "duration_s": 19.1, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 18615.1, "decode_ms": 451.2, "decode_tok_s": 53.2, "hit_rate": 0.926, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 23, "drafts_accepted": 11}, {"projection": null, "time": 1791782076.331768, "duration_s": 19.1, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 18624.4, "decode_ms": 497.4, "decode_tok_s": 48.3, "hit_rate": 0.806, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 24, "drafts_accepted": 10}, {"projection": null, "time": 1791782071.402105, "duration_s": 4.9, "finish": "stop", "prompt_tokens": 14418, "reused": 11572, "output_tokens": 51, "prompt_total": 14418, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 51, "prompt_ms": 4195.5, "decode_ms": 710.0, "decode_tok_s": 71.8, "hit_rate": 0.908, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 42, "drafts_accepted": 26}, {"projection": null, "time": 1791782067.1535203, "duration_s": 4.2, "finish": "stop", "prompt_tokens": 11570, "reused": 8710, "output_tokens": 3, "prompt_total": 11570, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 3, "prompt_ms": 4183.3, "decode_ms": 47.0, "decode_tok_s": 63.8, "hit_rate": 0.969, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 2, "drafts_accepted": 1}, {"projection": null, "time": 1791782062.3347528, "duration_s": 4.8, "finish": "stop", "prompt_tokens": 8692, "reused": 5311, "output_tokens": 19, "prompt_total": 8692, "prompt_read": 3381, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 19, "prompt_ms": 4502.8, "decode_ms": 304.7, "decode_tok_s": 62.4, "hit_rate": 0.92, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 19, "drafts_accepted": 8}, {"projection": null, "time": 1791782058.1376424, "duration_s": 4.2, "finish": "length", "prompt_tokens": 5248, "reused": 3708, "output_tokens": 64, "prompt_total": 5248, "prompt_read": 1540, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 3344.1, "decode_ms": 843.2, "decode_tok_s": 75.9, "hit_rate": 0.88, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 55, "drafts_accepted": 35}, {"projection": null, "time": 1791782053.1116652, "duration_s": 5.0, "finish": "length", "prompt_tokens": 3715, "reused": 633, "output_tokens": 64, "prompt_total": 3715, "prompt_read": 3082, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 4271.3, "decode_ms": 745.8, "decode_tok_s": 85.8, "hit_rate": 0.907, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 51, "drafts_accepted": 41}, {"projection": null, "time": 1791782050.7852345, "duration_s": 2.3, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 24, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 2008.9, "decode_ms": 306.3, "decode_tok_s": 78.4, "hit_rate": 0.812, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 25, "drafts_accepted": 15}, {"projection": null, "time": 1791782046.2375796, "duration_s": 4.5, "finish": "stop", "prompt_tokens": 14421, "reused": 11575, "output_tokens": 18, "prompt_total": 14421, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 18, "prompt_ms": 4216.4, "decode_ms": 319.4, "decode_tok_s": 56.4, "hit_rate": 0.912, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 21, "drafts_accepted": 7}, {"projection": null, "time": 1791782041.9197686, "duration_s": 4.3, "finish": "stop", "prompt_tokens": 11569, "reused": 8709, "output_tokens": 7, "prompt_total": 11569, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 7, "prompt_ms": 4219.4, "decode_ms": 86.5, "decode_tok_s": 80.9, "hit_rate": 0.962, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 6, "drafts_accepted": 4}, {"projection": null, "time": 1791782037.1317878, "duration_s": 4.8, "finish": "stop", "prompt_tokens": 8691, "reused": 5310, "output_tokens": 19, "prompt_total": 8691, "prompt_read": 3381, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 19, "prompt_ms": 4544.4, "decode_ms": 230.9, "decode_tok_s": 82.3, "hit_rate": 0.925, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 15, "drafts_accepted": 11}], "conversation_cache": {"enabled": false, "budget_mib": 0, "slots": 4, "parked": 0, "bytes": 0, "evictions": 0, "parks": 0, "restores": 0, "last_event": null, "last_tokens": null, "last_at": null, "requests": 40, "requests_reused": 15, "reused_tokens": 89940, "prompt_tokens": 291777, "last_reused": 0, "last_prompt": 32049}, "requests_kept": 40, "totals": {"since": 1791781196.593416, "requests": 40, "prompt_tokens": 291777, "reused": 89940, "output_tokens": 1766, "prompt_ms": 181080.7, "decode_ms": 23481.4, "drafts_offered": 1439, "drafts_accepted": 992}, "hardware": {"gpus": [{"index": 0, "util": 0, "mem_used": 16836001792, "mem_total": 17103323136, "temp": 43, "power": 5.326}, {"index": 1, "util": 0, "mem_used": 12494372864, "mem_total": 12884901888, "temp": 49, "power": 7.926}], "gpu_util": 0.0, "gpu_mem_used": 29330374656, "gpu_mem_total": 29988225024, "gpu_temp": 49, "gpu_power": 13.251999999999999, "gpu_power_limit": 350.0, "gpu_pcie_gen": 1, "gpu_pcie_gen_max": 5, "gpu_pcie_width": 8, "gpu_pcie_rx_mb": 0.9375, "gpu_pcie_tx_mb": 3.361328125, "cpu": 0.0, "ram_used": 65340919808, "ram_total": 66472062976, "disk_read_mb": 0.1578600807545618, "disk_write_mb": 0.0, "tok_s": 0.0, "tok_s_mean": 0, "prefill_tok_s_mean": 0.0}, "hardware_static": {"gpu_name": "NVIDIA GeForce RTX 5060 Ti + NVIDIA GeForce RTX 3060", "gpu_count": 2, "gpu_note": null, "cpu_name": "AMD Ryzen 9 7900X 12-Core Processor", "cores": 12, "threads": 24, "psutil": true}, "history": {"gpu_util": [50.0, 50.0, 83.0, 50.0, 50.0, 50.0, 100.0, 99.5, 100.0, 50.0, 100.0, 100.0, 50.0, 100.0, 99.5, 50.0, 50.0, 50.0, 50.0, 50.0, 65.5, 50.0, 50.0, 50.0, 100.0, 99.5, 50.0, 100.0, 100.0, 100.0, 50.0, 100.0, 99.0, 50.0, 50.0, 50.0, 55.0, 50.0, 19.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "gpu_mem_used": [29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656], "gpu_temp": [58, 58, 56, 64, 68, 68, 68, 67, 69, 66, 67, 70, 69, 63, 69, 70, 58, 61, 62, 57, 59, 66, 68, 68, 66, 69, 69, 68, 69, 70, 71, 70, 68, 70, 59, 59, 59, 56, 58, 55, 55, 54, 54, 53, 53, 52, 52, 52, 52, 51, 51, 51, 51, 51, 49, 49, 49, 49, 48, 49], "gpu_power": [148.14, 129.07, 78.64, 171.18, 197.97, 185.1, 182.17, 256.9, 274.0, 208.5, 223.2, 254.39, 253.19, 196.86, 259.59, 259.75, 179.28, 157.19, 142.62, 107.69, 104.02, 182.95, 185.51, 184.26, 203.8, 272.94, 272.71, 197.66, 261.36, 264.96, 230.88, 225.87, 265.02, 248.61, 165.01, 159.69, 138.49, 73.56, 112.27, 83.5, 66.97, 59.95, 56.92, 56.76, 56.48, 56.45, 56.37, 56.34, 56.25, 56.15, 56.05, 55.98, 55.85, 55.64, 42.57, 15.6, 15.11, 14.05, 13.62, 13.25], "gpu_pcie_rx_mb": [62.23, 3293.39, 331.0, 10.41, 1.27, 18271.07, 3390.28, 29786.76, 12825.33, 23239.11, 3344.74, 13830.49, 8185.34, 3356.42, 16687.96, 30589.62, 3294.48, 3333.71, 30342.31, 3300.11, 29881.13, 19573.26, 1.24, 30811.72, 3319.51, 107.16, 7601.33, 33183.73, 27620.7, 29526.81, 3362.14, 17883.41, 13661.52, 23955.87, 3291.81, 226.95, 3291.48, 3309.58, 0.82, 0.78, 0.79, 0.79, 0.78, 0.83, 0.88, 0.79, 0.79, 0.78, 0.79, 0.82, 0.85, 0.78, 0.78, 1.11, 0.99, 1.03, 1.2, 1.01, 0.94, 0.94], "cpu": [4.3, 4.2, 14.6, 17.6, 4.2, 4.2, 4.3, 4.3, 4.1, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 25.8, 4.2, 4.2, 4.1, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 22.5, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 0.1, 0.1, 0.0, 0.1, 0.0, 0.1, 0.0, 0.1, 0.0, 0.1, 0.0, 0.1, 0.0, 0.2, 0.0], "ram_used": [65166311424, 65263038464, 65263038464, 65083301888, 65083301888, 65083301888, 65087741952, 65095163904, 65095163904, 65193435136, 65193435136, 65215959040, 65216364544, 65216364544, 65217216512, 65217216512, 65217216512, 65217216512, 65313685504, 65313943552, 65139728384, 65140584448, 65141100544, 65141100544, 65141346304, 65141346304, 65141346304, 65237815296, 65238073344, 65241382912, 65241686016, 65241686016, 65241686016, 65241686016, 65241686016, 65241686016, 65339961344, 65339961344, 65340436480, 65340665856, 65340665856, 65340665856, 65340665856, 65340657664, 65340657664, 65340915712, 65340915712, 65340915712, 65340915712, 65341173760, 65341173760, 65341173760, 65340960768, 65340919808, 65340919808, 65340919808, 65340919808, 65340919808, 65340919808, 65340919808], "disk_read_mb": [0.16, 0.16, 0.36, 0.62, 0.16, 0.16, 0.16, 47.55, 0.16, 0.14, 0.16, 0.17, 0.16, 0.16, 4.87, 0.16, 0.14, 0.16, 0.16, 0.16, 0.88, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.14, 0.16, 0.16, 0.16, 0.16, 2.65, 0.16, 0.16, 0.16, 0.16, 0.14, 0.16, 0.16, 0.18, 0.16, 0.16, 0.14, 0.16, 0.43, 3.16, 0.16, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16], "tok_s": [0.0, 0.0, 36.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "prefill_tok_s_mean": [1825.6, 1897.4, 0.0, null, null, null, null, null, 1394.5, 1394.5, 1394.5, 1394.5, 1676.1, 1676.1, 1676.1, 1826.6, 1826.6, 1826.6, 1898.2, 1898.2, null, null, null, null, null, null, 1394.4, 1394.4, 1394.4, 1672.2, 1672.2, 1672.2, 1672.2, 1823.2, 1823.2, 1823.2, 1896.9, 1896.9, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}, "time": 1791782157.3806937} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/A/A-before.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/A/A-before.json new file mode 100644 index 00000000000..167e49aa81f --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/A/A-before.json @@ -0,0 +1 @@ +{"engine": {"model": "qwen3.8-flash-next-iq3_s", "max_context": 524288, "images": true, "version": "0.1.43", "logit_bias": 1, "context": 524288, "kv": "int8", "kv_resident": 20480, "expert_slots": 8842, "expert_cache_mib": 17910, "expert_slots_primary": 4984, "expert_cache_primary_mib": 9474, "spec": 6, "mtp_max": 4, "lookup": 3, "vram_free_mib": 324, "cvec": 0, "arena_mib": 47962, "pool_workers": 11, "pcie_frac": "0.00", "spec_min_p": "0.70", "conversation_cache_mib": 0, "conversation_cache_slots": 4, "conversation_cache_min_free_mib": 2560, "conversation_cache_min_tokens": 0, "tail_role_token": -1, "vram_elastic": 0, "engine": "0.1.43"}, "live": {"state": "idle", "queued": 0, "phase": null, "prompt_tokens": null, "prompt_read": null, "prompt_total": null, "generated": null, "max_tokens": null, "elapsed_s": null, "tok_s": null, "tok_s_mean": null, "prefill_tok_s_mean": null, "tok_s_window_s": null}, "requests": [], "conversation_cache": {"enabled": false, "budget_mib": 0, "slots": 4, "parked": 0, "bytes": 0, "evictions": 0, "parks": 0, "restores": 0, "last_event": null, "last_tokens": null, "last_at": null, "requests": 0, "requests_reused": 0, "reused_tokens": 0, "prompt_tokens": 0, "last_reused": null, "last_prompt": null}, "requests_kept": 0, "totals": {"since": 1791781196.593416, "requests": 0, "prompt_tokens": 0, "reused": 0, "output_tokens": 0, "prompt_ms": 0.0, "decode_ms": 0.0, "drafts_offered": 0, "drafts_accepted": 0}, "hardware": {"gpus": [{"index": 0, "util": 0, "mem_used": 16762601472, "mem_total": 17103323136, "temp": 34, "power": 4.415}, {"index": 1, "util": 0, "mem_used": 12427264000, "mem_total": 12884901888, "temp": 36, "power": 7.309}], "gpu_util": 0.0, "gpu_mem_used": 29189865472, "gpu_mem_total": 29988225024, "gpu_temp": 36, "gpu_power": 11.724, "gpu_power_limit": 350.0, "gpu_pcie_gen": 1, "gpu_pcie_gen_max": 5, "gpu_pcie_width": 8, "gpu_pcie_rx_mb": 0.9716796875, "gpu_pcie_tx_mb": 1.24609375, "cpu": 0.0, "ram_used": 66173644800, "ram_total": 66472062976, "disk_read_mb": 0.0, "disk_write_mb": 0.0, "tok_s": 0.0, "tok_s_mean": 0, "prefill_tok_s_mean": 0.0}, "hardware_static": {"gpu_name": "NVIDIA GeForce RTX 5060 Ti + NVIDIA GeForce RTX 3060", "gpu_count": 2, "gpu_note": null, "cpu_name": "AMD Ryzen 9 7900X 12-Core Processor", "cores": 12, "threads": 24, "psutil": true}, "history": {"gpu_util": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "gpu_mem_used": [29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472], "gpu_temp": [36, 37, 36, 36, 37, 36, 36, 37, 37, 36, 36, 37, 36, 37, 36, 36, 36, 36, 36, 36, 37, 36, 36, 37, 36, 36, 36, 36, 36, 36, 36, 36, 37, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36], "gpu_power": [11.78, 11.93, 11.9, 11.84, 11.84, 11.81, 11.79, 11.75, 11.8, 11.75, 11.69, 11.72, 11.8, 11.81, 11.84, 11.95, 12.0, 11.78, 11.81, 11.76, 11.77, 11.73, 11.55, 11.6, 11.71, 11.75, 11.72, 11.75, 11.77, 11.78, 11.89, 11.86, 11.79, 11.79, 11.8, 11.82, 11.87, 11.8, 11.77, 11.79, 11.79, 11.79, 11.7, 11.65, 11.67, 11.69, 11.81, 11.84, 11.82, 11.77, 11.71, 11.77, 11.8, 11.82, 11.79, 11.63, 11.6, 11.57, 11.74, 11.72], "gpu_pcie_rx_mb": [1.03, 0.92, 1.0, 0.98, 0.94, 0.98, 1.01, 1.0, 0.99, 1.0, 0.91, 0.92, 1.08, 0.92, 0.92, 1.0, 0.96, 0.97, 0.93, 0.98, 1.03, 1.0, 0.99, 0.97, 0.95, 0.99, 0.99, 0.99, 0.99, 1.08, 0.99, 0.96, 0.99, 0.95, 0.99, 0.94, 0.98, 1.0, 1.0, 1.0, 0.94, 0.93, 0.98, 0.94, 0.98, 1.03, 1.07, 0.99, 0.93, 0.98, 0.93, 0.93, 0.98, 1.0, 0.98, 0.95, 0.92, 0.94, 0.99, 0.97], "cpu": [0.0, 0.1, 0.0, 0.1, 0.1, 0.1, 0.0, 0.0, 0.1, 0.0, 0.1, 0.1, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.1, 0.1, 0.1, 0.1, 0.0, 0.1, 0.0, 0.1, 0.0, 0.1, 0.0, 0.0, 0.1, 0.1, 0.1, 0.1, 0.1, 0.0, 0.0, 0.1, 0.0, 0.2, 0.1, 0.0, 0.1, 0.1, 0.1, 0.0, 0.1, 0.1, 0.1, 0.0, 0.0, 0.1, 0.1, 0.1, 0.0, 0.0], "ram_used": [66170105856, 66170105856, 66171854848, 66171854848, 66171854848, 66172268544, 66172370944, 66172370944, 66172370944, 66172190720, 66172190720, 66172190720, 66172190720, 66172190720, 66172190720, 66172190720, 66172694528, 66172694528, 66172694528, 66172694528, 66172694528, 66162712576, 66162745344, 66162745344, 66162745344, 66163408896, 66164027392, 66164273152, 66164273152, 66164686848, 66163236864, 66163699712, 66163699712, 66163699712, 66163957760, 66163957760, 66164203520, 66165243904, 66165243904, 66165243904, 66165243904, 66165243904, 66165489664, 66166317056, 66167283712, 66167283712, 66167541760, 66167541760, 66167541760, 66167541760, 66167541760, 66167541760, 66167758848, 66167758848, 66167758848, 66167758848, 66171068416, 66173284352, 66173644800, 66173644800], "disk_read_mb": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.1, 0.34, 0.0, 0.01, 0.0, 0.0, 0.0, 0.0, 0.02, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.01, 0.0, 0.0, 0.0, 0.0, 0.05, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.04, 0.0, 0.0, 0.0, 0.01, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "tok_s": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "prefill_tok_s_mean": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}, "time": 1791781923.7022662} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/A/README.md b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/A/README.md new file mode 100644 index 00000000000..05de4d1ea79 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/A/README.md @@ -0,0 +1,150 @@ +# Community benchmark: RTX 5060 Ti 16 GB + RTX 3060 12 GB layer split, IQ3_S + +Measured on 2026-10-11 by @mmbeanz. Speed suite plus expert-cache +telemetry; no correctness check in this run. **Headline:** on an unequal 16+12 GB pair with +unequal weak links, the tuned `--layer-split 40` holds 8,842 of 24,576 experts and measures +**85.9% median expert-cache hit rate** — between the planner's old power-law coverage curve +(97.0% predicted) and its new `H(f)` fit (73.8%). Details in *Expert-cache coverage* below. + +## Hardware and software + +Hand-written from `nvidia-smi`, `lspci -vv` **under load**, `dmidecode` and `nvcc` on the +host — do not use the auto-generated `machine` block (see *Tool findings*). + +- GPUs: **RTX 5060 Ti 16,311 MiB** on PCIe **5.0 x8** (`LnkSta` under load; measured H2D + **28.2-28.7 GB/s**) + **RTX 3060 12,288 MiB** on PCIe **3.0 x4** (`LnkSta`; measured H2D + **3.15-3.2 GB/s**), the 3060 behind the 600-series chipset. Layer split across both, + explicit `--layer-split 40`, `--trim-stage-weights`. The board cannot negotiate better + widths; idle `lspci` shows gen1 (ASPM), so only loaded readings are meaningful. +- CPU: AMD Ryzen 9 7900X (12 cores / 24 threads). RAM: **64 GB DDR5-6400** dual-channel + (61.9 GiB visible to the container). Storage: Samsung 990 EVO Plus 4 TB NVMe + (**DRAM-less**, HMB); serial redacted. +- OS: Linux 7.2.9-1-cachyos, bare metal. The engine runs in a **systemd-nspawn** container + (shared host kernel: real PCIe links, real CPU topology, no virtio artifacts). +- GPU driver 615.71.09; CUDA 13.4 (nvcc V13.4.92). +- Strata 0.1.43, engine built from upstream `7695859a` (tag v0.1.43); the install carries + one local commit changing **only `setup.py` (pip -> uv)** — no engine code changes. +- Background workloads: none (dedicated inference box); no power-limit changes. + +## Model and configuration + +- Model: Qwen3.8-Flash-Next **IQ3_S**, ISTA-DASLab GSQ-RCO pack (sha256 in + `sha256sums.txt` in the report root), 512K context via yarn factor 2.0. +- Expert profile: the install's `data/expert-profile.bin`, **no `--expert-profile-save`** + (pinned boot profile, no mutation between runs). Engine log confirms + `profile ...: 24576 ranked pairs, built for 24576 slots`. +- Engine facts at boot (`GET /metrics`): `expert_slots 8842` (17,910 MiB) across both + cards, `expert_slots_primary 4984` (9,474 MiB), `arena_mib 47962`, `pool_workers 11`, + `kv int8`, `kv_resident 20480`, `pcie_frac 0.00`, `spec_min_p 0.70`, `vram_free_mib 324`. + +```text +strata --serve --pack /home/llm/Strata-data/packs/iq3_s \ + --native Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf \ + --ple-gguf Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf \ + --expert-profile /home/llm/Strata/data/expert-profile.bin \ + --expert-cache auto --prefill auto --spec 4 --mtp rt \ + --max-context 524288 --rope-scaling yarn --rope-scale 2.0 \ + --kv int8 --kv-resident 20480 --vision --remote-expert-opt --trim-stage-weights \ + --pcie-frac 0.00 --spec-min-p 0.70 --layer-split 40 +``` + +(`--layer-split 40` is appended by the server from the config's `layer_split` key. +`--remote-expert-opt` is present but **inert** with a layer split — it feeds the +helper-cache path only, and setup adds it automatically, #578.) + +## Method + +`python3 tools/strata_bench.py --url --config --long --runs 3` +against a **server started by the operator** (cold boot, 0 requests before the suite; +`GET /metrics` snapshots in `A-before.json` / `A-after.json` / `A-after-full.json`). +Model loading is not timed. One warm-up chat and a 1K prompt are discarded first; every +fresh prompt carries a unique tag so the conversation cache cannot help it. Thinking off, +temperature 0. + +The bench client ran on a **different machine** on the same LAN: RTT 0.25 ms, HTTPS TTFB +~7 ms, so TTFT inflation is negligible (<0.2% of the smallest TTFT, 0.40 s) — but see +*Tool findings* for what the tool records in that mode. + +Suite contents and sizes are the tool's standard ones (short chat, 4K/16K/32K fresh +prompts with a 24-token cap, 6-turn agent sessions x3). Total suite time 205 s. + +## Results + +| Configuration | Prompt tokens | Reused | Generated | Runs | Prompt tok/s (median, range) | Decode tok/s (median, range) | TTFT s (median, range) | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 46 (43-51) | 0 | 100 | 9 | n/a | 74.4 (73.2-92.9) | 0.40 (0.37-0.45) | +| 4K prompt, fresh | 4019 | 0 | 24 cap | 3 | 868.1 (866.5-874.6) | 70.7 (70.6-73.1)* | 4.66 (4.62-4.67) | +| 16K prompt, fresh | 16058 | 0 | 24 cap | 3 | 1412.7 (1412.1-1414.7) | 58.3 (56.4-60.3)* | 11.40 (11.38-11.40) | +| 32K prompt, fresh | 32049 | 0 | 24 cap | 3 | 1720.8 (1720.5-1721.7) | 53.2 (48.3-56.9)* | 18.66 (18.65-18.67) | +| agent follow-up turns | 14421 at last turn | see below | up to 64 | 3 | see below | 80.7 (53.1-93.6) | 4.28 (3.31-4.67) | + +\* decode for the capped prompt tests is not in the tool's table (only 24 tokens); these +come from the engine's per-request `decode_tok_s` in `GET /metrics?requests=all`. A 24-token +window is short, so treat as indicative — the long-context tax (70.7 -> 53.2 tok/s from 4K +to 32K) is the point, not the decimals. + +Draft acceptance over the whole suite: **992 / 1,439 = 69.0%**. + +### Expert-cache coverage (the interesting part) + +Per-request `hit_rate` from `/metrics?requests=all`, all 40 requests: +**median 0.859, mean 0.842, range 0.550-0.969.** By prompt size (median): + +| prompt read | hit_rate median (range) | decode tok/s median | +| ---: | --- | ---: | +| 32,049 | 0.926 (0.806-0.958) | 53.2 | +| 16,058 | 0.944 (0.907-0.969) | 58.3 | +| 4,019 | 0.866 (0.704-0.922) | 70.7 | +| 609 | 0.812 (0.752-0.842) | 71.6 | + +This config holds **8,842 of 24,576 experts (f = 0.360)**. The layer-split planner's +coverage curve predicts the routed mass those 8,842 capture: + +| curve | predicted | +| --- | ---: | +| old `(r+1)^-1.2` (still the default at <=3 GPUs) | **97.0%** | +| new `H(f) = 1-(1-f)^b`, b=3 (0.1.43, fit on a 4-way rig) | **73.8%** | +| **measured hit_rate (median)** | **85.9%** | + +Neither curve is right on this box; **b = 4.4** would fit the measurement. For scale, the +same comparison for the `auto` K=3 placement (6,135 pairs, f=0.25) predicts 95.7% (old) vs +57.8% (new) — that placement's measured hit_rate is the next data point (run B1). + +Also confirmed by telemetry, all 40 requests: `pcie_share = 0`, `ram_blobs = 0`, +`file_blobs = 0`, `file_mb = 0` — with `--pcie-frac 0.00` no expert is copied over PCIe and +nothing is read from the SSD in steady use; every miss is computed by the CPU pool from the +pinned arena. Hit rate rises with prompt length (0.81 at 609 tokens -> 0.93 at 32K), +consistent with the long prompt warming the cache before its decode phase. + +### Reproducibility against the earlier on-box run + +The same suite run the day before, executed **on** bertha, gave: 32K prompt 1721.8 vs **1720.8** here (0.06%), short-chat +decode 75.1 vs **74.4**, agent cache hit 68.8% vs **68.7%**. Remote measurement from a LAN +client is equivalent for these numbers. + +## Tool findings (bugs, with the above as evidence) + +1. **`gpu_slug()` names multi-GPU folders after the first card only**: `f"{len(names)}x-{s}"` + with `s` from `names[0]` — on this box it produced `2x-rtx-5060-ti` and silently dropped + the 3060. (Related to #2001 / #1985.) The remote run named it correctly, + because with no local `nvidia-smi` it fell back to the server's combined `gpu_name` + string. +2. **`collect_machine()` blends client and server fields when `--url` points at a remote + server**: `gpu` name, `cpu`, `strata_commit` come from the server (correct), but + `ram_gib` (recorded **3.8 GiB** — the client!), `os`, `gpu_driver` and `cuda` + (null, no local `nvidia-smi`) come from the client. A remote-run report is silently + wrong unless hand-patched. +3. The generated README references `engine-log-excerpt.txt` even when the tool did not + start the server (no such file is written in `--url` mode). +4. `/metrics` returns only the last **12** requests by default; the full history needs + `?requests=all`. Worth a line in `docs/COMMUNITY_BENCHMARKS.md`, which asks for + per-request data but does not mention the flag. + +## Correctness and limitations + +- Speed only: no needle test, no task-quality check in this run (the needle suite: `../needles/`). +- Decode for the capped prompt tests is a 24-token window (see footnote). +- No concurrency row (`--clients` not used), no vision latency, no mmap/low-RAM tier, + Windows/HIP not tested, single server config (the `auto` split variants are separate runs). +- `vram_free_mib 324` at boot is tight; the engine's stall line is 256 MiB — worth noting + for anyone copying this config on a 16+12 GB pair. diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/A/results.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/A/results.json new file mode 100644 index 00000000000..04372e3726e --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/A/results.json @@ -0,0 +1,1016 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-11", + "params": { + "runs": 3, + "think": false, + "clients": 0, + "long": true, + "started_server": false, + "command": "python tools/strata_bench.py --long" + }, + "elapsed_s": 205.0940580368042, + "machine": { + "gpu": [ + "NVIDIA GeForce RTX 5060 Ti", + "NVIDIA GeForce RTX 3060" + ], + "gpu_vram_mib": 28599.0, + "gpu_driver": "615.71.09", + "cuda": "13.4 (nvcc V13.4.92)", + "cpu": "AMD Ryzen 9 7900X 12-Core Processor", + "cpu_cores": 12, + "cpu_threads": 24, + "ram_gib": 61.9, + "os": "Linux 7.2.9-1-cachyos", + "python": "3.14.7", + "strata_version": "0.1.43", + "strata_commit": "7695859a", + "engine_version": "0.1.43", + "model": "qwen3.8-flash-next-iq3_s", + "quant": "IQ3_S", + "context": 524288, + "parallel_slots": 1, + "config": { + "exe": "strata", + "args": [ + "--pack", + "iq3_s", + "--native", + "Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf", + "--ple-gguf", + "Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf", + "--expert-profile", + "expert-profile.bin", + "--expert-cache", + "auto", + "--prefill", + "auto", + "--spec", + "4", + "--mtp", + "rt", + "--max-context", + "524288", + "--rope-scaling", + "yarn", + "--rope-scale", + "2.0", + "--kv", + "int8", + "--kv-resident", + "20480", + "--vision", + "--remote-expert-opt", + "--trim-stage-weights", + "--pcie-frac", + "0.00", + "--spec-min-p", + "0.70" + ], + "model_name": "qwen3.8-flash-next-iq3_s", + "lib_dirs": [ + "/opt/cuda/bin", + "/opt/cuda/lib64" + ], + "port": 8080, + "gpu": [ + 0, + 1 + ], + "gpus_asked": true, + "layer_split": "40", + "vision": { + "exe": "/home//Strata/engine/strata-vision", + "mmproj": "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/mmproj-Qwen3.8-Flash-Next-BF16.gguf", + "model": "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf", + "gpu": true, + "max_tokens": 1024 + }, + "allowed_hosts": [ + "bertha" + ], + "draft_vocab": "en" + }, + "launch": "strata --pack iq3_s --native Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf --ple-gguf Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf --expert-profile expert-profile.bin --expert-cache auto --prefill auto --spec 4 --mtp rt --max-context 524288 --rope-scaling yarn --rope-scale 2.0 --kv int8 --kv-resident 20480 --vision --remote-expert-opt --trim-stage-weights --pcie-frac 0.00 --spec-min-p 0.70", + "engine_info": { + "model": "qwen3.8-flash-next-iq3_s", + "max_context": 524288, + "images": true, + "version": "0.1.43", + "logit_bias": 1, + "context": 524288, + "kv": "int8", + "kv_resident": 20480, + "expert_slots": 8842, + "expert_cache_mib": 17910, + "expert_slots_primary": 4984, + "expert_cache_primary_mib": 9474, + "spec": 6, + "mtp_max": 4, + "lookup": 3, + "vram_free_mib": 324, + "cvec": 0, + "arena_mib": 47962, + "pool_workers": 11, + "pcie_frac": "0.00", + "spec_min_p": "0.70", + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "conversation_cache_min_tokens": 0, + "tail_role_token": -1, + "vram_elastic": 0, + "engine": "0.1.43" + }, + "vram_after": { + "elastic": false + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.449, + "total_s": 1.543, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 119.9, + "prompt_tps_source": "engine", + "decode_tps": 90.0, + "decode_tps_source": "engine", + "prompt_ms": 425.3, + "decode_ms": 1111.4, + "drafts_offered": 86, + "drafts_accepted": 70, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.398, + "total_s": 1.724, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 115.2, + "prompt_tps_source": "engine", + "decode_tps": 74.4, + "decode_tps_source": "engine", + "prompt_ms": 373.4, + "decode_ms": 1343.9, + "drafts_offered": 76, + "drafts_accepted": 52, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.419, + "total_s": 1.767, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 116.6, + "prompt_tps_source": "engine", + "decode_tps": 73.2, + "decode_tps_source": "engine", + "prompt_ms": 394.4, + "decode_ms": 1365.7, + "drafts_offered": 76, + "drafts_accepted": 53, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.441, + "total_s": 1.5, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 122.3, + "prompt_tps_source": "engine", + "decode_tps": 92.9, + "decode_tps_source": "engine", + "prompt_ms": 416.9, + "decode_ms": 1076.8, + "drafts_offered": 80, + "drafts_accepted": 70, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.372, + "total_s": 1.668, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 123.4, + "prompt_tps_source": "engine", + "decode_tps": 76.2, + "decode_tps_source": "engine", + "prompt_ms": 348.6, + "decode_ms": 1312.8, + "drafts_offered": 76, + "drafts_accepted": 52, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.399, + "total_s": 1.747, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 122.7, + "prompt_tps_source": "engine", + "decode_tps": 73.2, + "decode_tps_source": "engine", + "prompt_ms": 374.8, + "decode_ms": 1366.0, + "drafts_offered": 76, + "drafts_accepted": 50, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.443, + "total_s": 1.55, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 121.8, + "prompt_tps_source": "engine", + "decode_tps": 88.9, + "decode_tps_source": "engine", + "prompt_ms": 418.7, + "decode_ms": 1125.2, + "drafts_offered": 84, + "drafts_accepted": 70, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.368, + "total_s": 1.706, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 125.0, + "prompt_tps_source": "engine", + "decode_tps": 73.8, + "decode_tps_source": "engine", + "prompt_ms": 343.9, + "decode_ms": 1355.6, + "drafts_offered": 86, + "drafts_accepted": 51, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.394, + "total_s": 1.735, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 124.3, + "prompt_tps_source": "engine", + "decode_tps": 73.6, + "decode_tps_source": "engine", + "prompt_ms": 370.1, + "decode_ms": 1358.6, + "drafts_offered": 74, + "drafts_accepted": 50, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 74.4, + "min": 73.2, + "max": 92.9, + "n": 9 + }, + "ttft_s": { + "median": 0.399, + "min": 0.368, + "max": 0.449, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 46, + "min": 43, + "max": 51, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 4.665, + "total_s": 4.986, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 866.5, + "prompt_tps_source": "engine", + "decode_tps": 70.7, + "decode_tps_source": "engine", + "prompt_ms": 4638.3, + "decode_ms": 339.4, + "drafts_offered": 15, + "drafts_accepted": 12, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.621, + "total_s": 4.931, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 874.6, + "prompt_tps_source": "engine", + "decode_tps": 73.1, + "decode_tps_source": "engine", + "prompt_ms": 4595.1, + "decode_ms": 328.2, + "drafts_offered": 13, + "drafts_accepted": 11, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.656, + "total_s": 4.977, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 868.1, + "prompt_tps_source": "engine", + "decode_tps": 70.6, + "decode_tps_source": "engine", + "prompt_ms": 4629.8, + "decode_ms": 339.9, + "drafts_offered": 12, + "drafts_accepted": 10, + "run": 2 + } + ], + "prompt_tps": { + "median": 868.1, + "min": 866.5, + "max": 874.6, + "n": 3 + }, + "ttft_s": { + "median": 4.656, + "min": 4.621, + "max": 4.665, + "n": 3 + }, + "prompt_tokens": { + "median": 4019, + "min": 4019, + "max": 4019, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 11.383, + "total_s": 11.79, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1414.7, + "prompt_tps_source": "engine", + "decode_tps": 56.4, + "decode_tps_source": "engine", + "prompt_ms": 11350.7, + "decode_ms": 425.3, + "drafts_offered": 20, + "drafts_accepted": 10, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 11.402, + "total_s": 11.782, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1412.1, + "prompt_tps_source": "engine", + "decode_tps": 60.3, + "decode_tps_source": "engine", + "prompt_ms": 11371.7, + "decode_ms": 398.2, + "drafts_offered": 19, + "drafts_accepted": 12, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 11.399, + "total_s": 11.792, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1412.7, + "prompt_tps_source": "engine", + "decode_tps": 58.3, + "decode_tps_source": "engine", + "prompt_ms": 11367.1, + "decode_ms": 411.9, + "drafts_offered": 19, + "drafts_accepted": 10, + "run": 2 + } + ], + "prompt_tps": { + "median": 1412.7, + "min": 1412.1, + "max": 1414.7, + "n": 3 + }, + "ttft_s": { + "median": 11.399, + "min": 11.383, + "max": 11.402, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "32K": { + "target_tokens": 32768, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 18.662, + "total_s": 19.141, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 1720.8, + "prompt_tps_source": "engine", + "decode_tps": 48.3, + "decode_tps_source": "engine", + "prompt_ms": 18624.4, + "decode_ms": 497.4, + "drafts_offered": 24, + "drafts_accepted": 10, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 18.653, + "total_s": 19.085, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 1721.7, + "prompt_tps_source": "engine", + "decode_tps": 53.2, + "decode_tps_source": "engine", + "prompt_ms": 18615.1, + "decode_ms": 451.2, + "drafts_offered": 23, + "drafts_accepted": 11, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 18.666, + "total_s": 19.069, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 1720.5, + "prompt_tps_source": "engine", + "decode_tps": 56.9, + "decode_tps_source": "engine", + "prompt_ms": 18627.7, + "decode_ms": 421.7, + "drafts_offered": 22, + "drafts_accepted": 12, + "run": 2 + } + ], + "prompt_tps": { + "median": 1720.8, + "min": 1720.5, + "max": 1721.7, + "n": 3 + }, + "ttft_s": { + "median": 18.662, + "min": 18.653, + "max": 18.666, + "n": 3 + }, + "prompt_tokens": { + "median": 32049, + "min": 32049, + "max": 32049, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 2.029, + "total_s": 2.316, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 21, + "prompt_tps": 304.5, + "prompt_tps_source": "engine", + "decode_tps": 68.7, + "decode_tps_source": "engine", + "prompt_ms": 2000.0, + "decode_ms": 305.8, + "drafts_offered": 22, + "drafts_accepted": 13, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.475, + "total_s": 5.18, + "prompt_tokens": 3712, + "reused_tokens": 631, + "read_tokens": 3081, + "generated_tokens": 64, + "prompt_tps": 694.2, + "prompt_tps_source": "engine", + "decode_tps": 88.5, + "decode_tps_source": "engine", + "prompt_ms": 4438.3, + "decode_ms": 723.4, + "drafts_offered": 49, + "drafts_accepted": 42, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.383, + "total_s": 4.159, + "prompt_tokens": 5245, + "reused_tokens": 3775, + "read_tokens": 1470, + "generated_tokens": 64, + "prompt_tps": 438.4, + "prompt_tps_source": "engine", + "decode_tps": 80.7, + "decode_tps_source": "engine", + "prompt_ms": 3353.0, + "decode_ms": 793.4, + "drafts_offered": 55, + "drafts_accepted": 38, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.665, + "total_s": 4.945, + "prompt_tokens": 8689, + "reused_tokens": 5308, + "read_tokens": 3381, + "generated_tokens": 20, + "prompt_tps": 729.8, + "prompt_tps_source": "engine", + "decode_tps": 67.2, + "decode_tps_source": "engine", + "prompt_ms": 4633.0, + "decode_ms": 297.8, + "drafts_offered": 16, + "drafts_accepted": 9, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.338, + "total_s": 4.461, + "prompt_tokens": 11568, + "reused_tokens": 8708, + "read_tokens": 2860, + "generated_tokens": 12, + "prompt_tps": 664.5, + "prompt_tps_source": "engine", + "decode_tps": 85.4, + "decode_tps_source": "engine", + "prompt_ms": 4304.3, + "decode_ms": 140.5, + "drafts_offered": 9, + "drafts_accepted": 7, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.278, + "total_s": 4.542, + "prompt_tokens": 14425, + "reused_tokens": 11579, + "read_tokens": 2846, + "generated_tokens": 15, + "prompt_tps": 670.3, + "prompt_tps_source": "engine", + "decode_tps": 53.1, + "decode_tps_source": "engine", + "prompt_ms": 4245.8, + "decode_ms": 282.3, + "drafts_offered": 17, + "drafts_accepted": 5, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 2.035, + "total_s": 2.339, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 23, + "prompt_tps": 302.8, + "prompt_tps_source": "engine", + "decode_tps": 71.6, + "decode_tps_source": "engine", + "prompt_ms": 2011.3, + "decode_ms": 321.1, + "drafts_offered": 24, + "drafts_accepted": 15, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.312, + "total_s": 4.978, + "prompt_tokens": 3714, + "reused_tokens": 634, + "read_tokens": 3080, + "generated_tokens": 64, + "prompt_tps": 718.5, + "prompt_tps_source": "engine", + "decode_tps": 93.6, + "decode_tps_source": "engine", + "prompt_ms": 4286.5, + "decode_ms": 683.7, + "drafts_offered": 49, + "drafts_accepted": 41, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.312, + "total_s": 4.068, + "prompt_tokens": 5247, + "reused_tokens": 3777, + "read_tokens": 1470, + "generated_tokens": 64, + "prompt_tps": 447.6, + "prompt_tps_source": "engine", + "decode_tps": 82.6, + "decode_tps_source": "engine", + "prompt_ms": 3284.5, + "decode_ms": 774.6, + "drafts_offered": 56, + "drafts_accepted": 38, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.573, + "total_s": 4.786, + "prompt_tokens": 8691, + "reused_tokens": 5310, + "read_tokens": 3381, + "generated_tokens": 19, + "prompt_tps": 744.0, + "prompt_tps_source": "engine", + "decode_tps": 82.3, + "decode_tps_source": "engine", + "prompt_ms": 4544.4, + "decode_ms": 230.9, + "drafts_offered": 15, + "drafts_accepted": 11, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.25, + "total_s": 4.319, + "prompt_tokens": 11569, + "reused_tokens": 8709, + "read_tokens": 2860, + "generated_tokens": 7, + "prompt_tps": 677.8, + "prompt_tps_source": "engine", + "decode_tps": 80.9, + "decode_tps_source": "engine", + "prompt_ms": 4219.4, + "decode_ms": 86.5, + "drafts_offered": 6, + "drafts_accepted": 4, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.246, + "total_s": 4.548, + "prompt_tokens": 14421, + "reused_tokens": 11575, + "read_tokens": 2846, + "generated_tokens": 18, + "prompt_tps": 675.0, + "prompt_tps_source": "engine", + "decode_tps": 56.4, + "decode_tps_source": "engine", + "prompt_ms": 4216.4, + "decode_ms": 319.4, + "drafts_offered": 21, + "drafts_accepted": 7, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 2.034, + "total_s": 2.323, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 24, + "prompt_tps": 303.2, + "prompt_tps_source": "engine", + "decode_tps": 78.4, + "decode_tps_source": "engine", + "prompt_ms": 2008.9, + "decode_ms": 306.3, + "drafts_offered": 25, + "drafts_accepted": 15, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.301, + "total_s": 5.029, + "prompt_tokens": 3715, + "reused_tokens": 633, + "read_tokens": 3082, + "generated_tokens": 64, + "prompt_tps": 721.6, + "prompt_tps_source": "engine", + "decode_tps": 85.8, + "decode_tps_source": "engine", + "prompt_ms": 4271.3, + "decode_ms": 745.8, + "drafts_offered": 51, + "drafts_accepted": 41, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.371, + "total_s": 4.196, + "prompt_tokens": 5248, + "reused_tokens": 3708, + "read_tokens": 1540, + "generated_tokens": 64, + "prompt_tps": 460.5, + "prompt_tps_source": "engine", + "decode_tps": 75.9, + "decode_tps_source": "engine", + "prompt_ms": 3344.1, + "decode_ms": 843.2, + "drafts_offered": 55, + "drafts_accepted": 35, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.531, + "total_s": 4.818, + "prompt_tokens": 8692, + "reused_tokens": 5311, + "read_tokens": 3381, + "generated_tokens": 19, + "prompt_tps": 750.9, + "prompt_tps_source": "engine", + "decode_tps": 62.4, + "decode_tps_source": "engine", + "prompt_ms": 4502.8, + "decode_ms": 304.7, + "drafts_offered": 19, + "drafts_accepted": 8, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.216, + "total_s": 4.245, + "prompt_tokens": 11570, + "reused_tokens": 8710, + "read_tokens": 2860, + "generated_tokens": 3, + "prompt_tps": 683.7, + "prompt_tps_source": "engine", + "decode_tps": 63.8, + "decode_tps_source": "engine", + "prompt_ms": 4183.3, + "decode_ms": 47.0, + "drafts_offered": 2, + "drafts_accepted": 1, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.228, + "total_s": 4.92, + "prompt_tokens": 14418, + "reused_tokens": 11572, + "read_tokens": 2846, + "generated_tokens": 51, + "prompt_tps": 678.3, + "prompt_tps_source": "engine", + "decode_tps": 71.8, + "decode_tps_source": "engine", + "prompt_ms": 4195.5, + "decode_ms": 710.0, + "drafts_offered": 42, + "drafts_accepted": 26, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 68.748, + "min": 68.588, + "max": 68.753, + "n": 3 + }, + "reread_tokens": { + "median": 4, + "min": 4, + "max": 74, + "n": 3 + }, + "new_tokens": { + "median": 13634, + "min": 13633, + "max": 13635, + "n": 3 + }, + "turn_ttft_s": { + "median": 4.278, + "min": 3.312, + "max": 4.665, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 14421, + "min": 14418, + "max": 14425, + "n": 3 + }, + "turn_decode_tps": { + "median": 80.7, + "min": 53.1, + "max": 93.6, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 2.034, + "min": 2.029, + "max": 2.035, + "n": 3 + } + }, + "clients": {}, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/B1-after-full.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/B1-after-full.json new file mode 100644 index 00000000000..f156f54eb59 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/B1-after-full.json @@ -0,0 +1 @@ +{"engine": {"model": "qwen3.8-flash-next-iq3_s", "max_context": 524288, "images": true, "version": "0.1.43", "logit_bias": 1, "context": 524288, "kv": "int8", "kv_resident": 20480, "expert_slots": 5935, "expert_cache_mib": 11394, "expert_slots_primary": 1536, "expert_cache_primary_mib": 2762, "spec": 6, "mtp_max": 4, "lookup": 3, "vram_free_mib": 3548, "cvec": 0, "arena_mib": 47962, "pool_workers": 11, "pcie_frac": "0.09", "spec_min_p": "0.70", "conversation_cache_mib": 0, "conversation_cache_slots": 4, "conversation_cache_min_free_mib": 2560, "conversation_cache_min_tokens": 0, "tail_role_token": -1, "vram_elastic": 0, "engine": "0.1.43"}, "live": {"state": "idle", "queued": 0, "phase": null, "prompt_tokens": null, "prompt_read": null, "prompt_total": null, "generated": null, "max_tokens": null, "elapsed_s": null, "tok_s": null, "tok_s_mean": null, "prefill_tok_s_mean": null, "tok_s_window_s": null}, "requests": [{"projection": null, "time": 1791783981.3060672, "duration_s": 25.1, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 24794.0, "decode_ms": 307.9, "decode_tok_s": 77.9, "hit_rate": 0.952, "pcie_share": 0.025, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 12, "drafts_accepted": 12}, {"projection": null, "time": 1791783955.9602377, "duration_s": 25.3, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 24881.3, "decode_ms": 437.0, "decode_tok_s": 54.9, "hit_rate": 0.915, "pcie_share": 0.05, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 18, "drafts_accepted": 9}, {"projection": null, "time": 1791783930.550667, "duration_s": 25.4, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 24826.3, "decode_ms": 557.0, "decode_tok_s": 43.1, "hit_rate": 0.795, "pcie_share": 0.12, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 24, "drafts_accepted": 10}, {"projection": null, "time": 1791783926.3476424, "duration_s": 4.2, "finish": "stop", "prompt_tokens": 14475, "reused": 11629, "output_tokens": 15, "prompt_total": 14475, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 15, "prompt_ms": 3885.5, "decode_ms": 294.9, "decode_tok_s": 50.9, "hit_rate": 0.912, "pcie_share": 0.052, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 16, "drafts_accepted": 5}, {"projection": null, "time": 1791783922.2710514, "duration_s": 4.1, "finish": "stop", "prompt_tokens": 11613, "reused": 8753, "output_tokens": 17, "prompt_total": 11613, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 17, "prompt_ms": 3865.3, "decode_ms": 198.8, "decode_tok_s": 85.5, "hit_rate": 0.935, "pcie_share": 0.041, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 15, "drafts_accepted": 11}, {"projection": null, "time": 1791783916.45339, "duration_s": 5.8, "finish": "length", "prompt_tokens": 8689, "reused": 5238, "output_tokens": 64, "prompt_total": 8689, "prompt_read": 3451, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 4899.1, "decode_ms": 904.1, "decode_tok_s": 70.8, "hit_rate": 0.921, "pcie_share": 0.046, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 52, "drafts_accepted": 34}, {"projection": null, "time": 1791783912.2975943, "duration_s": 4.1, "finish": "length", "prompt_tokens": 5245, "reused": 3775, "output_tokens": 64, "prompt_total": 5245, "prompt_read": 1470, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 3316.7, "decode_ms": 828.6, "decode_tok_s": 77.2, "hit_rate": 0.888, "pcie_share": 0.075, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 55, "drafts_accepted": 42}, {"projection": null, "time": 1791783907.4979033, "duration_s": 4.8, "finish": "length", "prompt_tokens": 3712, "reused": 631, "output_tokens": 64, "prompt_total": 3712, "prompt_read": 3081, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 4014.2, "decode_ms": 776.9, "decode_tok_s": 82.4, "hit_rate": 0.881, "pcie_share": 0.075, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 48, "drafts_accepted": 40}, {"projection": null, "time": 1791783905.0281515, "duration_s": 2.5, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 21, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 21, "prompt_ms": 2098.6, "decode_ms": 363.7, "decode_tok_s": 57.7, "hit_rate": 0.786, "pcie_share": 0.128, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 24, "drafts_accepted": 13}, {"projection": null, "time": 1791783900.411642, "duration_s": 4.6, "finish": "stop", "prompt_tokens": 14456, "reused": 11610, "output_tokens": 45, "prompt_total": 14456, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 45, "prompt_ms": 3858.3, "decode_ms": 747.8, "decode_tok_s": 60.2, "hit_rate": 0.915, "pcie_share": 0.054, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 40, "drafts_accepted": 20}, {"projection": null, "time": 1791783896.2668543, "duration_s": 4.1, "finish": "stop", "prompt_tokens": 11589, "reused": 8729, "output_tokens": 22, "prompt_total": 11589, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 22, "prompt_ms": 3865.2, "decode_ms": 266.5, "decode_tok_s": 82.6, "hit_rate": 0.92, "pcie_share": 0.051, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 19, "drafts_accepted": 14}, {"projection": null, "time": 1791783891.0316339, "duration_s": 5.2, "finish": "stop", "prompt_tokens": 8687, "reused": 5306, "output_tokens": 43, "prompt_total": 8687, "prompt_read": 3381, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 43, "prompt_ms": 4658.6, "decode_ms": 565.1, "decode_tok_s": 76.1, "hit_rate": 0.915, "pcie_share": 0.051, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 32, "drafts_accepted": 24}, {"projection": null, "time": 1791783886.73802, "duration_s": 4.3, "finish": "length", "prompt_tokens": 5243, "reused": 3773, "output_tokens": 64, "prompt_total": 5243, "prompt_read": 1470, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 3325.2, "decode_ms": 958.5, "decode_tok_s": 66.8, "hit_rate": 0.885, "pcie_share": 0.073, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 58, "drafts_accepted": 35}, {"projection": null, "time": 1791783881.9870005, "duration_s": 4.7, "finish": "length", "prompt_tokens": 3710, "reused": 629, "output_tokens": 64, "prompt_total": 3710, "prompt_read": 3081, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 3978.4, "decode_ms": 762.1, "decode_tok_s": 84.0, "hit_rate": 0.889, "pcie_share": 0.075, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 53, "drafts_accepted": 42}, {"projection": null, "time": 1791783879.5604925, "duration_s": 2.4, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 19, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 19, "prompt_ms": 2098.4, "decode_ms": 320.3, "decode_tok_s": 59.3, "hit_rate": 0.818, "pcie_share": 0.107, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 22, "drafts_accepted": 12}, {"projection": null, "time": 1791783875.1460893, "duration_s": 4.4, "finish": "stop", "prompt_tokens": 14435, "reused": 11589, "output_tokens": 31, "prompt_total": 14435, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 31, "prompt_ms": 3874.6, "decode_ms": 529.1, "decode_tok_s": 58.6, "hit_rate": 0.882, "pcie_share": 0.077, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 27, "drafts_accepted": 14}, {"projection": null, "time": 1791783870.9816136, "duration_s": 4.2, "finish": "stop", "prompt_tokens": 11571, "reused": 8711, "output_tokens": 19, "prompt_total": 11571, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 19, "prompt_ms": 3891.4, "decode_ms": 258.3, "decode_tok_s": 73.6, "hit_rate": 0.918, "pcie_share": 0.055, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 20, "drafts_accepted": 12}, {"projection": null, "time": 1791783866.0308068, "duration_s": 4.9, "finish": "stop", "prompt_tokens": 8692, "reused": 5311, "output_tokens": 20, "prompt_total": 8692, "prompt_read": 3381, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 20, "prompt_ms": 4661.8, "decode_ms": 273.5, "decode_tok_s": 73.1, "hit_rate": 0.877, "pcie_share": 0.077, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 14, "drafts_accepted": 11}, {"projection": null, "time": 1791783861.8295135, "duration_s": 4.2, "finish": "length", "prompt_tokens": 5248, "reused": 3779, "output_tokens": 64, "prompt_total": 5248, "prompt_read": 1469, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 3315.1, "decode_ms": 872.6, "decode_tok_s": 73.3, "hit_rate": 0.84, "pcie_share": 0.108, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 52, "drafts_accepted": 40}, {"projection": null, "time": 1791783856.8711545, "duration_s": 4.9, "finish": "length", "prompt_tokens": 3715, "reused": 633, "output_tokens": 64, "prompt_total": 3715, "prompt_read": 3082, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 4015.1, "decode_ms": 929.2, "decode_tok_s": 68.9, "hit_rate": 0.734, "pcie_share": 0.16, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 49, "drafts_accepted": 40}, {"projection": null, "time": 1791783854.3570595, "duration_s": 2.5, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 24, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 2079.5, "decode_ms": 417.2, "decode_tok_s": 57.5, "hit_rate": 0.75, "pcie_share": 0.149, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 22, "drafts_accepted": 13}, {"projection": null, "time": 1791783840.7863905, "duration_s": 13.6, "finish": "length", "prompt_tokens": 16058, "reused": 0, "output_tokens": 24, "prompt_total": 16058, "prompt_read": 16058, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 13139.8, "decode_ms": 414.7, "decode_tok_s": 57.9, "hit_rate": 0.966, "pcie_share": 0.017, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 23, "drafts_accepted": 11}, {"projection": null, "time": 1791783827.2929657, "duration_s": 13.5, "finish": "length", "prompt_tokens": 16058, "reused": 0, "output_tokens": 24, "prompt_total": 16058, "prompt_read": 16058, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 13083.1, "decode_ms": 394.1, "decode_tok_s": 60.9, "hit_rate": 0.952, "pcie_share": 0.025, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 19, "drafts_accepted": 10}, {"projection": null, "time": 1791783813.8672, "duration_s": 13.4, "finish": "length", "prompt_tokens": 16058, "reused": 0, "output_tokens": 24, "prompt_total": 16058, "prompt_read": 16058, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 12926.9, "decode_ms": 482.9, "decode_tok_s": 49.7, "hit_rate": 0.907, "pcie_share": 0.054, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 23, "drafts_accepted": 10}, {"projection": null, "time": 1791783808.6983736, "duration_s": 5.2, "finish": "length", "prompt_tokens": 4019, "reused": 0, "output_tokens": 24, "prompt_total": 4019, "prompt_read": 4019, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 4792.3, "decode_ms": 361.8, "decode_tok_s": 66.3, "hit_rate": 0.912, "pcie_share": 0.052, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 13, "drafts_accepted": 10}, {"projection": null, "time": 1791783803.4931953, "duration_s": 5.2, "finish": "length", "prompt_tokens": 4019, "reused": 0, "output_tokens": 24, "prompt_total": 4019, "prompt_read": 4019, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 4809.9, "decode_ms": 387.1, "decode_tok_s": 62.0, "hit_rate": 0.857, "pcie_share": 0.09, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 13, "drafts_accepted": 10}, {"projection": null, "time": 1791783798.225861, "duration_s": 5.3, "finish": "length", "prompt_tokens": 4019, "reused": 0, "output_tokens": 24, "prompt_total": 4019, "prompt_read": 4019, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 4859.2, "decode_ms": 399.6, "decode_tok_s": 60.1, "hit_rate": 0.7, "pcie_share": 0.189, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 15, "drafts_accepted": 12}, {"projection": null, "time": 1791783796.2714171, "duration_s": 1.9, "finish": "length", "prompt_tokens": 46, "reused": 0, "output_tokens": 100, "prompt_total": 46, "prompt_read": 46, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 445.2, "decode_ms": 1500.8, "decode_tok_s": 66.6, "hit_rate": 0.835, "pcie_share": 0.099, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 67, "drafts_accepted": 50}, {"projection": null, "time": 1791783794.3746936, "duration_s": 1.9, "finish": "length", "prompt_tokens": 43, "reused": 0, "output_tokens": 100, "prompt_total": 43, "prompt_read": 43, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 405.7, "decode_ms": 1484.7, "decode_tok_s": 67.4, "hit_rate": 0.87, "pcie_share": 0.078, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 79, "drafts_accepted": 50}, {"projection": null, "time": 1791783792.5568578, "duration_s": 1.8, "finish": "length", "prompt_tokens": 51, "reused": 0, "output_tokens": 100, "prompt_total": 51, "prompt_read": 51, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 494.2, "decode_ms": 1317.3, "decode_tok_s": 75.9, "hit_rate": 0.731, "pcie_share": 0.169, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 79, "drafts_accepted": 70}, {"projection": null, "time": 1791783790.5589814, "duration_s": 2.0, "finish": "length", "prompt_tokens": 46, "reused": 0, "output_tokens": 100, "prompt_total": 46, "prompt_read": 46, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 456.2, "decode_ms": 1535.4, "decode_tok_s": 65.1, "hit_rate": 0.827, "pcie_share": 0.102, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 72, "drafts_accepted": 50}, {"projection": null, "time": 1791783788.6335347, "duration_s": 1.9, "finish": "length", "prompt_tokens": 43, "reused": 0, "output_tokens": 100, "prompt_total": 43, "prompt_read": 43, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 412.9, "decode_ms": 1506.4, "decode_tok_s": 66.4, "hit_rate": 0.861, "pcie_share": 0.082, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 74, "drafts_accepted": 48}, {"projection": null, "time": 1791783786.8134844, "duration_s": 1.8, "finish": "length", "prompt_tokens": 51, "reused": 0, "output_tokens": 100, "prompt_total": 51, "prompt_read": 51, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 494.0, "decode_ms": 1319.8, "decode_tok_s": 75.8, "hit_rate": 0.725, "pcie_share": 0.169, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 80, "drafts_accepted": 70}, {"projection": null, "time": 1791783784.7311506, "duration_s": 2.1, "finish": "length", "prompt_tokens": 46, "reused": 0, "output_tokens": 100, "prompt_total": 46, "prompt_read": 46, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 476.0, "decode_ms": 1599.9, "decode_tok_s": 62.5, "hit_rate": 0.783, "pcie_share": 0.13, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 70, "drafts_accepted": 50}, {"projection": null, "time": 1791783782.7535534, "duration_s": 2.0, "finish": "length", "prompt_tokens": 43, "reused": 0, "output_tokens": 100, "prompt_total": 43, "prompt_read": 43, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 441.1, "decode_ms": 1529.4, "decode_tok_s": 65.4, "hit_rate": 0.809, "pcie_share": 0.114, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 76, "drafts_accepted": 52}, {"projection": null, "time": 1791783780.8935204, "duration_s": 1.9, "finish": "length", "prompt_tokens": 51, "reused": 0, "output_tokens": 100, "prompt_total": 51, "prompt_read": 51, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 502.5, "decode_ms": 1350.2, "decode_tok_s": 74.1, "hit_rate": 0.737, "pcie_share": 0.166, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 85, "drafts_accepted": 70}, {"projection": null, "time": 1791783778.542239, "duration_s": 2.3, "finish": "length", "prompt_tokens": 1028, "reused": 0, "output_tokens": 8, "prompt_total": 1028, "prompt_read": 1028, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 8, "prompt_ms": 2176.5, "decode_ms": 167.8, "decode_tok_s": 47.7, "hit_rate": 0.696, "pcie_share": 0.178, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 7, "drafts_accepted": 3}, {"projection": null, "time": 1791783776.409469, "duration_s": 2.1, "finish": "length", "prompt_tokens": 906, "reused": 0, "output_tokens": 1, "prompt_total": 906, "prompt_read": 906, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 1, "prompt_ms": 2104.3, "decode_ms": 20.8, "decode_tok_s": 48.1, "hit_rate": 0.777, "pcie_share": 0.128, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 0, "drafts_accepted": 0}, {"projection": null, "time": 1791783774.352323, "duration_s": 2.1, "finish": "length", "prompt_tokens": 260, "reused": 0, "output_tokens": 1, "prompt_total": 260, "prompt_read": 260, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 1, "prompt_ms": 2029.4, "decode_ms": 21.0, "decode_tok_s": 47.6, "hit_rate": 0.782, "pcie_share": 0.129, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 0, "drafts_accepted": 0}, {"projection": null, "time": 1791783773.5319479, "duration_s": 0.8, "finish": "length", "prompt_tokens": 34, "reused": 0, "output_tokens": 24, "prompt_total": 34, "prompt_read": 34, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 410.6, "decode_ms": 399.6, "decode_tok_s": 60.1, "hit_rate": 0.543, "pcie_share": 0.247, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 18, "drafts_accepted": 17}], "conversation_cache": {"enabled": false, "budget_mib": 0, "slots": 4, "parked": 0, "bytes": 0, "evictions": 0, "parks": 0, "restores": 0, "last_event": null, "last_tokens": null, "last_at": null, "requests": 40, "requests_reused": 15, "reused_tokens": 90096, "prompt_tokens": 291933, "last_reused": 0, "last_prompt": 32049}, "requests_kept": 40, "totals": {"since": 1791782893.273165, "requests": 40, "prompt_tokens": 291933, "reused": 90096, "output_tokens": 1874, "prompt_ms": 204662.4, "decode_ms": 27762.399999999994, "drafts_offered": 1485, "drafts_accepted": 1046}, "hardware": {"gpus": [{"index": 0, "util": 0, "mem_used": 16829710336, "mem_total": 17103323136, "temp": 47, "power": 19.552}, {"index": 1, "util": 0, "mem_used": 9210232832, "mem_total": 12884901888, "temp": 50, "power": 35.779}], "gpu_util": 0.0, "gpu_mem_used": 26039943168, "gpu_mem_total": 29988225024, "gpu_temp": 50, "gpu_power": 55.331, "gpu_power_limit": 350.0, "gpu_pcie_gen": 5, "gpu_pcie_gen_max": 5, "gpu_pcie_width": 8, "gpu_pcie_rx_mb": 0.7705078125, "gpu_pcie_tx_mb": 1.06640625, "cpu": 0.1, "ram_used": 65101131776, "ram_total": 66472062976, "disk_read_mb": 0.15855517581379022, "disk_write_mb": 0.0, "tok_s": 0.0, "tok_s_mean": 0, "prefill_tok_s_mean": 0.0}, "hardware_static": {"gpu_name": "NVIDIA GeForce RTX 5060 Ti + NVIDIA GeForce RTX 3060", "gpu_count": 2, "gpu_note": null, "cpu_name": "AMD Ryzen 9 7900X 12-Core Processor", "cores": 12, "threads": 24, "psutil": true}, "history": {"gpu_util": [50.0, 50.0, 100.0, 49.5, 49.5, 50.0, 50.0, 100.0, 50.0, 99.5, 50.0, 99.0, 50.0, 99.0, 50.0, 50.0, 79.5, 100.0, 48.5, 50.0, 50.0, 50.0, 95.0, 50.0, 99.0, 50.0, 98.5, 50.0, 100.0, 50.0, 99.5, 50.0, 99.5, 50.0, 48.0, 50.0, 100.0, 57.0, 50.0, 99.5, 50.0, 100.0, 50.0, 50.0, 50.0, 50.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "gpu_mem_used": [26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168], "gpu_temp": [64, 63, 63, 64, 63, 64, 64, 65, 64, 66, 64, 66, 63, 63, 63, 63, 63, 61, 63, 66, 65, 62, 59, 60, 64, 65, 66, 65, 67, 66, 66, 63, 67, 67, 67, 67, 66, 65, 68, 65, 68, 69, 66, 67, 66, 56, 54, 54, 53, 53, 53, 52, 52, 52, 51, 51, 51, 51, 51, 50], "gpu_power": [102.79, 187.62, 181.46, 191.63, 170.7, 187.09, 173.73, 184.77, 176.84, 182.63, 184.6, 185.27, 179.01, 176.73, 188.87, 176.4, 188.25, 175.22, 185.07, 175.35, 164.06, 159.26, 89.41, 132.43, 161.38, 178.31, 191.77, 179.44, 177.84, 173.46, 182.48, 184.5, 175.67, 189.64, 170.92, 190.71, 173.07, 193.46, 175.43, 186.41, 181.01, 180.64, 177.16, 169.18, 165.34, 136.24, 104.16, 80.74, 66.21, 59.21, 57.1, 56.75, 56.69, 56.54, 56.37, 56.3, 56.12, 56.12, 55.87, 55.33], "gpu_pcie_rx_mb": [19410.51, 28805.05, 11766.07, 6653.37, 26719.96, 28930.68, 3412.48, 19490.46, 29913.5, 19801.28, 28481.76, 10575.58, 19666.48, 9247.35, 28229.04, 28366.13, 28209.13, 13091.06, 19879.08, 17349.33, 28516.76, 3298.29, 2229.7, 3296.18, 20270.85, 20367.29, 29951.48, 27669.25, 11028.2, 20498.94, 25198.52, 19636.86, 17627.96, 19220.35, 10731.5, 19216.57, 18157.73, 26875.87, 95.29, 23064.05, 12962.66, 23006.42, 24627.01, 29656.46, 19582.95, 785.91, 0.8, 0.78, 1.0, 0.79, 0.78, 0.79, 0.78, 0.79, 0.8, 0.8, 0.78, 0.79, 0.8, 0.77], "cpu": [4.2, 4.3, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 19.7, 10.0, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 5.5, 15.8, 0.0, 0.1, 0.0, 0.1, 0.1, 0.0, 0.1, 0.1, 0.0, 0.0, 0.1, 0.0, 0.1], "ram_used": [64880349184, 64880594944, 64878682112, 64880144384, 64881881088, 64881876992, 64881876992, 64883093504, 64883093504, 64884899840, 64884899840, 64994467840, 64994467840, 64994467840, 64994467840, 64994455552, 64994455552, 64994455552, 64987488256, 64987488256, 64987488256, 65099939840, 65100345344, 64892776448, 64893022208, 64886468608, 64886984704, 64886984704, 64886984704, 64886984704, 64885022720, 64885022720, 64885022720, 64885022720, 64990654464, 64990654464, 64990654464, 64990912512, 64990912512, 64990912512, 64990912512, 64990912512, 64990912512, 64990912512, 64990912512, 65100931072, 65101139968, 65101139968, 65101139968, 65101139968, 65101139968, 65101139968, 65101139968, 65101139968, 65101139968, 65101139968, 65101139968, 65101139968, 65101139968, 65101131776], "disk_read_mb": [0.16, 0.16, 0.16, 13.39, 0.16, 0.16, 0.16, 13.21, 0.16, 0.14, 0.16, 0.17, 0.22, 0.16, 0.15, 0.16, 0.16, 0.16, 0.16, 0.16, 0.14, 0.16, 0.16, 0.29, 0.16, 0.14, 0.16, 0.16, 0.17, 0.16, 0.14, 0.16, 0.16, 0.16, 0.17, 0.16, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.14, 0.16, 0.16, 0.16, 2.48, 0.16, 0.16, 0.14, 0.16, 0.16, 0.16, 0.16, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16], "tok_s": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 46.95, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "prefill_tok_s_mean": [null, null, 1047.1, 1047.1, 1214.1, 1214.1, 1276.6, 1276.6, 1276.6, 1310.3, 1310.3, 1328.5, 1328.5, 1340.1, 1340.1, 1349.4, 1349.4, 1354.1, 1354.1, 1354.1, 1358.1, 1335.0, 0.0, null, null, null, 1049.1, 1049.1, 1212.9, 1212.9, 1277.3, 1277.3, 1310.2, 1310.2, 1330.8, 1330.8, 1342.8, 1342.8, 1342.8, 1352.0, 1352.0, 1358.0, 1358.0, 1362.3, 1362.3, 1339.8, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}, "time": 1791784021.4510696} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/B1-after.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/B1-after.json new file mode 100644 index 00000000000..548c5ac3201 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/B1-after.json @@ -0,0 +1 @@ +{"engine": {"model": "qwen3.8-flash-next-iq3_s", "max_context": 524288, "images": true, "version": "0.1.43", "logit_bias": 1, "context": 524288, "kv": "int8", "kv_resident": 20480, "expert_slots": 5935, "expert_cache_mib": 11394, "expert_slots_primary": 1536, "expert_cache_primary_mib": 2762, "spec": 6, "mtp_max": 4, "lookup": 3, "vram_free_mib": 3548, "cvec": 0, "arena_mib": 47962, "pool_workers": 11, "pcie_frac": "0.09", "spec_min_p": "0.70", "conversation_cache_mib": 0, "conversation_cache_slots": 4, "conversation_cache_min_free_mib": 2560, "conversation_cache_min_tokens": 0, "tail_role_token": -1, "vram_elastic": 0, "engine": "0.1.43"}, "live": {"state": "idle", "queued": 0, "phase": null, "prompt_tokens": null, "prompt_read": null, "prompt_total": null, "generated": null, "max_tokens": null, "elapsed_s": null, "tok_s": null, "tok_s_mean": null, "prefill_tok_s_mean": null, "tok_s_window_s": null}, "requests": [{"projection": null, "time": 1791783981.3060672, "duration_s": 25.1, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 24794.0, "decode_ms": 307.9, "decode_tok_s": 77.9, "hit_rate": 0.952, "pcie_share": 0.025, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 12, "drafts_accepted": 12}, {"projection": null, "time": 1791783955.9602377, "duration_s": 25.3, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 24881.3, "decode_ms": 437.0, "decode_tok_s": 54.9, "hit_rate": 0.915, "pcie_share": 0.05, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 18, "drafts_accepted": 9}, {"projection": null, "time": 1791783930.550667, "duration_s": 25.4, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 24826.3, "decode_ms": 557.0, "decode_tok_s": 43.1, "hit_rate": 0.795, "pcie_share": 0.12, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 24, "drafts_accepted": 10}, {"projection": null, "time": 1791783926.3476424, "duration_s": 4.2, "finish": "stop", "prompt_tokens": 14475, "reused": 11629, "output_tokens": 15, "prompt_total": 14475, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 15, "prompt_ms": 3885.5, "decode_ms": 294.9, "decode_tok_s": 50.9, "hit_rate": 0.912, "pcie_share": 0.052, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 16, "drafts_accepted": 5}, {"projection": null, "time": 1791783922.2710514, "duration_s": 4.1, "finish": "stop", "prompt_tokens": 11613, "reused": 8753, "output_tokens": 17, "prompt_total": 11613, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 17, "prompt_ms": 3865.3, "decode_ms": 198.8, "decode_tok_s": 85.5, "hit_rate": 0.935, "pcie_share": 0.041, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 15, "drafts_accepted": 11}, {"projection": null, "time": 1791783916.45339, "duration_s": 5.8, "finish": "length", "prompt_tokens": 8689, "reused": 5238, "output_tokens": 64, "prompt_total": 8689, "prompt_read": 3451, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 4899.1, "decode_ms": 904.1, "decode_tok_s": 70.8, "hit_rate": 0.921, "pcie_share": 0.046, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 52, "drafts_accepted": 34}, {"projection": null, "time": 1791783912.2975943, "duration_s": 4.1, "finish": "length", "prompt_tokens": 5245, "reused": 3775, "output_tokens": 64, "prompt_total": 5245, "prompt_read": 1470, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 3316.7, "decode_ms": 828.6, "decode_tok_s": 77.2, "hit_rate": 0.888, "pcie_share": 0.075, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 55, "drafts_accepted": 42}, {"projection": null, "time": 1791783907.4979033, "duration_s": 4.8, "finish": "length", "prompt_tokens": 3712, "reused": 631, "output_tokens": 64, "prompt_total": 3712, "prompt_read": 3081, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 4014.2, "decode_ms": 776.9, "decode_tok_s": 82.4, "hit_rate": 0.881, "pcie_share": 0.075, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 48, "drafts_accepted": 40}, {"projection": null, "time": 1791783905.0281515, "duration_s": 2.5, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 21, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 21, "prompt_ms": 2098.6, "decode_ms": 363.7, "decode_tok_s": 57.7, "hit_rate": 0.786, "pcie_share": 0.128, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 24, "drafts_accepted": 13}, {"projection": null, "time": 1791783900.411642, "duration_s": 4.6, "finish": "stop", "prompt_tokens": 14456, "reused": 11610, "output_tokens": 45, "prompt_total": 14456, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 45, "prompt_ms": 3858.3, "decode_ms": 747.8, "decode_tok_s": 60.2, "hit_rate": 0.915, "pcie_share": 0.054, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 40, "drafts_accepted": 20}, {"projection": null, "time": 1791783896.2668543, "duration_s": 4.1, "finish": "stop", "prompt_tokens": 11589, "reused": 8729, "output_tokens": 22, "prompt_total": 11589, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 22, "prompt_ms": 3865.2, "decode_ms": 266.5, "decode_tok_s": 82.6, "hit_rate": 0.92, "pcie_share": 0.051, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 19, "drafts_accepted": 14}, {"projection": null, "time": 1791783891.0316339, "duration_s": 5.2, "finish": "stop", "prompt_tokens": 8687, "reused": 5306, "output_tokens": 43, "prompt_total": 8687, "prompt_read": 3381, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 43, "prompt_ms": 4658.6, "decode_ms": 565.1, "decode_tok_s": 76.1, "hit_rate": 0.915, "pcie_share": 0.051, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 32, "drafts_accepted": 24}], "conversation_cache": {"enabled": false, "budget_mib": 0, "slots": 4, "parked": 0, "bytes": 0, "evictions": 0, "parks": 0, "restores": 0, "last_event": null, "last_tokens": null, "last_at": null, "requests": 40, "requests_reused": 15, "reused_tokens": 90096, "prompt_tokens": 291933, "last_reused": 0, "last_prompt": 32049}, "requests_kept": 40, "totals": {"since": 1791782893.273165, "requests": 40, "prompt_tokens": 291933, "reused": 90096, "output_tokens": 1874, "prompt_ms": 204662.4, "decode_ms": 27762.399999999994, "drafts_offered": 1485, "drafts_accepted": 1046}, "hardware": {"gpus": [{"index": 0, "util": 0, "mem_used": 16829710336, "mem_total": 17103323136, "temp": 47, "power": 19.552}, {"index": 1, "util": 0, "mem_used": 9210232832, "mem_total": 12884901888, "temp": 50, "power": 35.779}], "gpu_util": 0.0, "gpu_mem_used": 26039943168, "gpu_mem_total": 29988225024, "gpu_temp": 50, "gpu_power": 55.331, "gpu_power_limit": 350.0, "gpu_pcie_gen": 5, "gpu_pcie_gen_max": 5, "gpu_pcie_width": 8, "gpu_pcie_rx_mb": 0.7705078125, "gpu_pcie_tx_mb": 1.06640625, "cpu": 0.1, "ram_used": 65101131776, "ram_total": 66472062976, "disk_read_mb": 0.15855517581379022, "disk_write_mb": 0.0, "tok_s": 0.0, "tok_s_mean": 0, "prefill_tok_s_mean": 0.0}, "hardware_static": {"gpu_name": "NVIDIA GeForce RTX 5060 Ti + NVIDIA GeForce RTX 3060", "gpu_count": 2, "gpu_note": null, "cpu_name": "AMD Ryzen 9 7900X 12-Core Processor", "cores": 12, "threads": 24, "psutil": true}, "history": {"gpu_util": [50.0, 50.0, 100.0, 49.5, 49.5, 50.0, 50.0, 100.0, 50.0, 99.5, 50.0, 99.0, 50.0, 99.0, 50.0, 50.0, 79.5, 100.0, 48.5, 50.0, 50.0, 50.0, 95.0, 50.0, 99.0, 50.0, 98.5, 50.0, 100.0, 50.0, 99.5, 50.0, 99.5, 50.0, 48.0, 50.0, 100.0, 57.0, 50.0, 99.5, 50.0, 100.0, 50.0, 50.0, 50.0, 50.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "gpu_mem_used": [26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168, 26039943168], "gpu_temp": [64, 63, 63, 64, 63, 64, 64, 65, 64, 66, 64, 66, 63, 63, 63, 63, 63, 61, 63, 66, 65, 62, 59, 60, 64, 65, 66, 65, 67, 66, 66, 63, 67, 67, 67, 67, 66, 65, 68, 65, 68, 69, 66, 67, 66, 56, 54, 54, 53, 53, 53, 52, 52, 52, 51, 51, 51, 51, 51, 50], "gpu_power": [102.79, 187.62, 181.46, 191.63, 170.7, 187.09, 173.73, 184.77, 176.84, 182.63, 184.6, 185.27, 179.01, 176.73, 188.87, 176.4, 188.25, 175.22, 185.07, 175.35, 164.06, 159.26, 89.41, 132.43, 161.38, 178.31, 191.77, 179.44, 177.84, 173.46, 182.48, 184.5, 175.67, 189.64, 170.92, 190.71, 173.07, 193.46, 175.43, 186.41, 181.01, 180.64, 177.16, 169.18, 165.34, 136.24, 104.16, 80.74, 66.21, 59.21, 57.1, 56.75, 56.69, 56.54, 56.37, 56.3, 56.12, 56.12, 55.87, 55.33], "gpu_pcie_rx_mb": [19410.51, 28805.05, 11766.07, 6653.37, 26719.96, 28930.68, 3412.48, 19490.46, 29913.5, 19801.28, 28481.76, 10575.58, 19666.48, 9247.35, 28229.04, 28366.13, 28209.13, 13091.06, 19879.08, 17349.33, 28516.76, 3298.29, 2229.7, 3296.18, 20270.85, 20367.29, 29951.48, 27669.25, 11028.2, 20498.94, 25198.52, 19636.86, 17627.96, 19220.35, 10731.5, 19216.57, 18157.73, 26875.87, 95.29, 23064.05, 12962.66, 23006.42, 24627.01, 29656.46, 19582.95, 785.91, 0.8, 0.78, 1.0, 0.79, 0.78, 0.79, 0.78, 0.79, 0.8, 0.8, 0.78, 0.79, 0.8, 0.77], "cpu": [4.2, 4.3, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 19.7, 10.0, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 5.5, 15.8, 0.0, 0.1, 0.0, 0.1, 0.1, 0.0, 0.1, 0.1, 0.0, 0.0, 0.1, 0.0, 0.1], "ram_used": [64880349184, 64880594944, 64878682112, 64880144384, 64881881088, 64881876992, 64881876992, 64883093504, 64883093504, 64884899840, 64884899840, 64994467840, 64994467840, 64994467840, 64994467840, 64994455552, 64994455552, 64994455552, 64987488256, 64987488256, 64987488256, 65099939840, 65100345344, 64892776448, 64893022208, 64886468608, 64886984704, 64886984704, 64886984704, 64886984704, 64885022720, 64885022720, 64885022720, 64885022720, 64990654464, 64990654464, 64990654464, 64990912512, 64990912512, 64990912512, 64990912512, 64990912512, 64990912512, 64990912512, 64990912512, 65100931072, 65101139968, 65101139968, 65101139968, 65101139968, 65101139968, 65101139968, 65101139968, 65101139968, 65101139968, 65101139968, 65101139968, 65101139968, 65101139968, 65101131776], "disk_read_mb": [0.16, 0.16, 0.16, 13.39, 0.16, 0.16, 0.16, 13.21, 0.16, 0.14, 0.16, 0.17, 0.22, 0.16, 0.15, 0.16, 0.16, 0.16, 0.16, 0.16, 0.14, 0.16, 0.16, 0.29, 0.16, 0.14, 0.16, 0.16, 0.17, 0.16, 0.14, 0.16, 0.16, 0.16, 0.17, 0.16, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.14, 0.16, 0.16, 0.16, 2.48, 0.16, 0.16, 0.14, 0.16, 0.16, 0.16, 0.16, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16], "tok_s": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 46.95, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "prefill_tok_s_mean": [null, null, 1047.1, 1047.1, 1214.1, 1214.1, 1276.6, 1276.6, 1276.6, 1310.3, 1310.3, 1328.5, 1328.5, 1340.1, 1340.1, 1349.4, 1349.4, 1354.1, 1354.1, 1354.1, 1358.1, 1335.0, 0.0, null, null, null, 1049.1, 1049.1, 1212.9, 1212.9, 1277.3, 1277.3, 1310.2, 1310.2, 1330.8, 1330.8, 1342.8, 1342.8, 1342.8, 1352.0, 1352.0, 1358.0, 1358.0, 1362.3, 1362.3, 1339.8, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}, "time": 1791784021.4632456} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/B1-before.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/B1-before.json new file mode 100644 index 00000000000..d723a367b61 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/B1-before.json @@ -0,0 +1 @@ +{"engine": {"model": "qwen3.8-flash-next-iq3_s", "max_context": 524288, "images": true, "version": "0.1.43", "logit_bias": 1, "context": 524288, "kv": "int8", "kv_resident": 20480, "expert_slots": 5935, "expert_cache_mib": 11394, "expert_slots_primary": 1536, "expert_cache_primary_mib": 2762, "spec": 6, "mtp_max": 4, "lookup": 3, "vram_free_mib": 3548, "cvec": 0, "arena_mib": 47962, "pool_workers": 11, "pcie_frac": "0.09", "spec_min_p": "0.70", "conversation_cache_mib": 0, "conversation_cache_slots": 4, "conversation_cache_min_free_mib": 2560, "conversation_cache_min_tokens": 0, "tail_role_token": -1, "vram_elastic": 0, "engine": "0.1.43"}, "live": {"state": "idle", "queued": 0, "phase": null, "prompt_tokens": null, "prompt_read": null, "prompt_total": null, "generated": null, "max_tokens": null, "elapsed_s": null, "tok_s": null, "tok_s_mean": null, "prefill_tok_s_mean": null, "tok_s_window_s": null}, "requests": [], "conversation_cache": {"enabled": false, "budget_mib": 0, "slots": 4, "parked": 0, "bytes": 0, "evictions": 0, "parks": 0, "restores": 0, "last_event": null, "last_tokens": null, "last_at": null, "requests": 0, "requests_reused": 0, "reused_tokens": 0, "prompt_tokens": 0, "last_reused": null, "last_prompt": null}, "requests_kept": 0, "totals": {"since": 1791782893.273165, "requests": 0, "prompt_tokens": 0, "reused": 0, "output_tokens": 0, "prompt_ms": 0.0, "decode_ms": 0.0, "drafts_offered": 0, "drafts_accepted": 0}, "hardware": {"gpus": [{"index": 0, "util": 0, "mem_used": 16756310016, "mem_total": 17103323136, "temp": 33, "power": 4.367}, {"index": 1, "util": 0, "mem_used": 9164095488, "mem_total": 12884901888, "temp": 36, "power": 7.411}], "gpu_util": 0.0, "gpu_mem_used": 25920405504, "gpu_mem_total": 29988225024, "gpu_temp": 36, "gpu_power": 11.777999999999999, "gpu_power_limit": 350.0, "gpu_pcie_gen": 1, "gpu_pcie_gen_max": 5, "gpu_pcie_width": 8, "gpu_pcie_rx_mb": 0.9326171875, "gpu_pcie_tx_mb": 3.21484375, "cpu": 0.1, "ram_used": 66198978560, "ram_total": 66472062976, "disk_read_mb": 0.0, "disk_write_mb": 0.0, "tok_s": 0.0, "tok_s_mean": 0, "prefill_tok_s_mean": 0.0}, "hardware_static": {"gpu_name": "NVIDIA GeForce RTX 5060 Ti + NVIDIA GeForce RTX 3060", "gpu_count": 2, "gpu_note": null, "cpu_name": "AMD Ryzen 9 7900X 12-Core Processor", "cores": 12, "threads": 24, "psutil": true}, "history": {"gpu_util": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "gpu_mem_used": [25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504, 25920405504], "gpu_temp": [36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36], "gpu_power": [11.86, 11.83, 11.74, 11.69, 11.74, 11.81, 11.9, 11.91, 11.86, 11.74, 11.83, 11.93, 11.86, 11.82, 11.84, 11.98, 11.92, 11.97, 11.95, 11.9, 11.96, 11.93, 11.88, 11.87, 11.84, 11.79, 11.8, 11.74, 11.78, 11.77, 11.79, 11.82, 11.86, 11.91, 11.89, 11.88, 11.86, 11.87, 11.91, 11.86, 11.86, 11.88, 11.81, 11.8, 11.73, 11.64, 11.66, 11.7, 11.7, 11.8, 11.79, 11.83, 11.78, 11.69, 11.72, 11.68, 11.75, 11.77, 11.77, 11.78], "gpu_pcie_rx_mb": [0.93, 0.97, 0.94, 0.98, 1.01, 0.88, 0.98, 0.99, 0.98, 0.97, 0.98, 1.0, 0.98, 0.98, 0.98, 0.94, 0.94, 0.98, 1.01, 1.02, 0.98, 0.98, 0.93, 0.98, 0.93, 0.98, 1.29, 0.92, 0.94, 0.94, 0.99, 0.96, 0.98, 1.01, 1.01, 0.98, 0.93, 0.86, 1.0, 0.99, 0.99, 0.99, 0.89, 1.33, 0.93, 0.98, 0.93, 0.95, 0.98, 0.95, 0.95, 0.99, 1.0, 0.98, 0.99, 0.98, 1.04, 1.0, 0.98, 0.93], "cpu": [0.2, 0.0, 0.1, 0.0, 0.1, 0.0, 0.2, 0.0, 0.0, 0.1, 0.0, 0.1, 0.1, 0.0, 0.1, 0.1, 0.0, 0.1, 0.1, 0.1, 0.0, 0.1, 0.0, 0.0, 0.1, 0.1, 0.1, 0.0, 0.1, 0.1, 0.1, 0.0, 0.0, 0.1, 0.8, 0.0, 0.0, 0.0, 0.1, 0.0, 0.1, 0.1, 0.1, 0.1, 0.1, 0.0, 0.0, 0.1, 0.1, 0.0, 0.0, 0.0, 0.1, 0.1, 0.1, 0.1, 0.0, 0.1, 0.0, 0.1], "ram_used": [66181525504, 66181525504, 66181525504, 66181525504, 66181525504, 66181525504, 66181525504, 66181525504, 66181525504, 66181525504, 66181525504, 66182758400, 66182758400, 66182758400, 66182758400, 66182758400, 66181898240, 66181898240, 66181898240, 66181898240, 66181898240, 66181898240, 66181898240, 66181898240, 66181898240, 66181898240, 66181652480, 66181652480, 66181652480, 66181652480, 66181652480, 66181652480, 66181652480, 66181652480, 66185916416, 66193498112, 66193498112, 66193756160, 66193756160, 66193756160, 66193756160, 66193756160, 66194001920, 66194001920, 66197663744, 66198024192, 66198024192, 66198024192, 66198437888, 66199052288, 66199052288, 66199052288, 66199052288, 66199310336, 66198978560, 66198978560, 66198978560, 66198978560, 66198978560, 66198978560], "disk_read_mb": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 34.72, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.03, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "tok_s": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "prefill_tok_s_mean": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}, "time": 1791783773.387179} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/README.md b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/README.md new file mode 100644 index 00000000000..0e1e96444f0 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/README.md @@ -0,0 +1,129 @@ +# Community benchmark B1: `--layer-split auto` on an unequal 16+12 GB pair (IQ3_S) + +Measured on 2026-10-11 by @mmbeanz. Same box, same pinned profile, +same suite as run A (`../A/`) — only the split policy changed: explicit `40` -> `auto`, +and `--pcie-frac` removed so the engine probes and assigns per-stage shares. + +**Headline.** On this unequal pair (RTX 5060 Ti 16 GB on PCIe 5.0 **x8** + RTX 3060 12 GB +on PCIe 3.0 **x4**), `auto` chooses **K=3** and loses badly on the prompt path +(**-25% at 32K**) and on short-context decode (**-19%**), while long-context decode ties. +Separately: **expert-cache hit rate is flat in held count** (85.9% at 8,842 experts vs +86.5% at 5,935, a 33% smaller cache) — which contradicts *both* of the planner's coverage +curves on this model/box. + +## Hardware and software + +Identical to run A — see `../A/README.md` for the hand-written table (7900X 12c/24t, +64 GB DDR5-6400, 990 EVO Plus 4 TB DRAM-less NVMe, driver 615.71.09, CUDA 13.4, +systemd-nspawn on bare metal, measured H2D 28.2-28.7 GB/s and 3.15-3.2 GB/s, `LnkSta` +under load: 5.0 x8 / 3.0 x4). Strata 0.1.43 from upstream `7695859a` + a `setup.py`-only +local commit. + +## Model and configuration + +- Model: Qwen3.8-Flash-Next IQ3_S, ISTA-DASLab GSQ-RCO pack (sha256 in `../sha256sums.txt`), + 512K context, yarn factor 2.0. +- Pinned boot profile `data/expert-profile.bin`, no `--expert-profile-save`; engine logs + `profile ...: 24576 ranked pairs`. +- Full config in `config.json`. Effective launch line: + +```text +strata --serve --pack /home/llm/Strata-data/packs/iq3_s \ + --native Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf \ + --ple-gguf Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf \ + --expert-profile /home/llm/Strata/data/expert-profile.bin \ + --expert-cache auto --prefill auto --spec 4 --mtp rt \ + --max-context 524288 --rope-scaling yarn --rope-scale 2.0 \ + --kv int8 --kv-resident 20480 --vision --remote-expert-opt --trim-stage-weights \ + --spec-min-p 0.70 --layer-split auto +``` + +### What `auto` decided on this pair (from the engine log, dry start of the same config) + +``` +layer split: card order 1,0 (the faster card last, #1352) -> CUDA0 = RTX 3060, CUDA1 = RTX 5060 Ti +PCIe probe: CUDA0 3.2 GB/s -> miss cost x6.26 ; CUDA1 28.7 GB/s -> miss cost x1.00 +CUDA0 28 SMs at 1.78 GHz -> 1.46 ms/layer ; CUDA1 36 SMs at 2.59 GHz -> 0.78 ms/layer +K=2 would give a stage fewer than 3 layers ... chose K=3 instead (48.5 ms per decode window, not 48.2) +K=3 - predicted 48.5 ms; the caches hold 6135 of 24576 profiled pairs (~95.2% of the routed mass), best of 89 placements +NEAR TIE - K=6 is predicted 49.4 ms, within 1.9% of K=3; the cards differ, so the prediction may not rank them + (it prices decode, not the prompt). To try the other: --layer-split 6 +prompt chunk auto: 3328 tokens, a 199-slot ring (run A with K=40: 8192) +``` + +Engine facts at boot (`GET /metrics`): `expert_slots 5935` (11,394 MiB), `expert_slots_primary +1536` (2,762 MiB — the 3060 holds *all* experts of its 3 layers and idles `vram_free_mib +3548`), `pcie_frac 0.09` (auto per-stage: 0.09 on the 3060, 0.55 on the 5060 Ti). + +## Method + +Same as run A: `strata_bench.py --url --config config.json --long --runs 3` +against an operator-started server, cold boot, 0 requests before the suite; +`GET /metrics?requests=all` snapshots before/after (`B1-before.json`, +`B1-after-full.json`). Client on a separate LAN box (RTT 0.25 ms); the `machine` block in +`results.json` blends client and server fields (see `../A/README.md`, *Tool findings*) — +hardware above is hand-written. Suite time ~205 s. + +## Results + +| Configuration | Prompt tok/s (median, range) | Decode tok/s (median, range) | TTFT s (median, range) | +| --- | --- | --- | --- | +| short chat (9 runs) | n/a | 66.6 (62.5-75.9) | 0.48 (0.43-0.53) | +| 4K prompt, fresh | 835.6 (827.1-838.6) | 62.0* | 4.84 (4.82-4.89) | +| 16K prompt, fresh | 1227.4 (1222.1-1242.2) | 57.9* | 13.11 (12.96-13.17) | +| 32K prompt, fresh | 1290.9 (1288.1-1292.6) | 54.9* | 24.86 (24.83-24.92) | +| agent follow-up turns (15) | see `results.json` | 73.3 (50.9-85.5) | 3.92 (3.34-4.93) | + +\* from the engine's per-request `decode_tok_s` in `/metrics?requests=all` (24-token cap). + +Draft acceptance 1,046 / 1,485 = **70.4%**. Agent conversation-cache hit 68.8% (same as A). + +### A (K=40) vs B1 (auto K=3) — same box, same day, same profile + +| metric | A (held 8,842) | B1 (held 5,935) | delta | +| --- | ---: | ---: | ---: | +| prompt 32K tok/s | 1,720.8 | 1,290.9 | **-25.0%** | +| prompt 16K tok/s | 1,412.7 | 1,227.4 | **-13.1%** | +| prompt 4K tok/s | 868.1 | 835.6 | -3.7% | +| TTFT 32K | 18.66 s | 24.86 s | **+33%** | +| decode 32K tok/s | 53.2 | 54.9 | +3.2% (tie) | +| decode 16K tok/s | 58.3 | 57.9 | -0.7% (tie) | +| decode 4K tok/s | 70.7 | 62.0 | **-12.3%** | +| decode 609 tok/s | 71.6 | 57.7 | **-19.4%** | +| expert hit_rate (median, all 40) | 0.859 | 0.865 | +0.006 | +| `pcie_share` (median / max) | 0 / 0 | 0.08 / 0.247 | streaming on, no gain | + +The engine's own warning was correct: its placement prediction **"prices decode, not the +prompt"** — and on this pair the prompt path is where `auto` bleeds. K=3 leaves the 3060 +with 1,536 slots, which trips the #1616 chunk clamp (prompt chunk 3,328 instead of 8,192), +and the 3060's 3,548 MiB of free VRAM is stranded because it already holds every expert of +its 3 layers. + +### Expert-cache coverage: the flat-hit-rate result + +| held experts | f | old `(r+1)^-1.2` predicts | new `H(f)=1-(1-f)^3` predicts | **measured hit_rate** | +| ---: | ---: | ---: | ---: | ---: | +| 8,842 (run A, K=40) | 0.360 | 97.0% | 73.8% | **85.9%** | +| 5,935 (run B1, K=3) | 0.241 | 95.6% | 56.4% | **86.5%** | + +Cutting the cache by **33%** changed measured hit rate by **+0.6 points** — i.e. on this +box, model and traffic, hit rate is essentially **independent of held count** across +f = 0.24-0.36. The old curve has the right *shape* (near-flat) and sits ~10 points too +high; the new `H(f)` fit has the wrong shape here (it predicts a 17-point drop that did not +happen) and the wrong level. Fitting `b` to each point separately gives b = 4.4 (A) and +b = 7.3 (B1) — i.e. no single `b` works, because the measured curve is flat where both +functional forms are steep. + +Caveat, stated plainly: `hit_rate` is per-request over the whole request (prefill + decode) +with the adaptive tier churning and the prompt path borrowing cache slots, so it is not a +clean measurement of "coverage of the static held set". But it is the quantity the operator +sees, and it is what the planner's curve is implicitly compared against. + +## Correctness and limitations + +- Speed only; no needle/quality check in this run (needle suite: `../needles/`). +- One suite per config (3 runs each), single day; A and B1 measured the same evening. +- `auto` also changed `pcie_frac` (0.00 -> per-stage 0.09/0.55), so B1 vs A is a *policy* + comparison, not a single-variable one. The isolation run is **B1b** (`auto` + explicit + `--pcie-frac 0.00`); the engine-endorsed alternative is **B1c** (`--layer-split 6`). +- No concurrency, no vision latency, no mmap/low-RAM tier, Windows/HIP untested. diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/config.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/config.json new file mode 100644 index 00000000000..7291fb4f5b3 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/config.json @@ -0,0 +1,44 @@ +{ + "exe": "/home/llm/Strata/engine/strata", + "args": [ + "--pack", + "/home/llm/Strata-data/packs/iq3_s", + "--native", + "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf", + "--ple-gguf", + "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf", + "--expert-profile", + "/home/llm/Strata/data/expert-profile.bin", + "--expert-cache", + "auto", + "--prefill", + "auto", + "--spec", + "4", + "--mtp", + "/home/llm/Strata-data/mtp/rt", + "--max-context", + "524288", + "--rope-scaling", + "yarn", + "--rope-scale", + "2.0", + "--kv", + "int8", + "--kv-resident", + "20480", + "--vision", + "--remote-expert-opt", + "--trim-stage-weights", + "--spec-min-p", + "0.70" + ], + "model_name": "qwen3.8-flash-next-iq3_s", + "port": 8080, + "gpu": [ + 0, + 1 + ], + "layer_split": "auto", + "draft_vocab": "en" +} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/results.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/results.json new file mode 100644 index 00000000000..3854e4d6c31 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1/results.json @@ -0,0 +1,999 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-11", + "params": { + "runs": 3, + "think": false, + "clients": 0, + "long": true, + "started_server": false, + "command": "python tools/strata_bench.py --long" + }, + "elapsed_s": 232.9608438014984, + "machine": { + "gpu": [ + "NVIDIA GeForce RTX 5060 Ti", + "NVIDIA GeForce RTX 3060" + ], + "gpu_vram_mib": 28599.0, + "gpu_driver": "615.71.09", + "cuda": "13.4 (nvcc V13.4.92)", + "cpu": "AMD Ryzen 9 7900X 12-Core Processor", + "cpu_cores": 12, + "cpu_threads": 24, + "ram_gib": 61.9, + "os": "Linux 7.2.9-1-cachyos", + "python": "3.14.7", + "strata_version": "0.1.43", + "strata_commit": "7695859a", + "engine_version": "0.1.43", + "model": "qwen3.8-flash-next-iq3_s", + "quant": "IQ3_S", + "context": 524288, + "parallel_slots": 1, + "config": { + "exe": "strata", + "args": [ + "--pack", + "iq3_s", + "--native", + "Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf", + "--ple-gguf", + "Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf", + "--expert-profile", + "expert-profile.bin", + "--expert-cache", + "auto", + "--prefill", + "auto", + "--spec", + "4", + "--mtp", + "rt", + "--max-context", + "524288", + "--rope-scaling", + "yarn", + "--rope-scale", + "2.0", + "--kv", + "int8", + "--kv-resident", + "20480", + "--vision", + "--remote-expert-opt", + "--trim-stage-weights", + "--spec-min-p", + "0.70" + ], + "model_name": "qwen3.8-flash-next-iq3_s", + "port": 8080, + "gpu": [ + 0, + 1 + ], + "layer_split": "auto", + "draft_vocab": "en" + }, + "launch": "strata --pack iq3_s --native Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf --ple-gguf Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf --expert-profile expert-profile.bin --expert-cache auto --prefill auto --spec 4 --mtp rt --max-context 524288 --rope-scaling yarn --rope-scale 2.0 --kv int8 --kv-resident 20480 --vision --remote-expert-opt --trim-stage-weights --spec-min-p 0.70", + "engine_info": { + "model": "qwen3.8-flash-next-iq3_s", + "max_context": 524288, + "images": true, + "version": "0.1.43", + "logit_bias": 1, + "context": 524288, + "kv": "int8", + "kv_resident": 20480, + "expert_slots": 5935, + "expert_cache_mib": 11394, + "expert_slots_primary": 1536, + "expert_cache_primary_mib": 2762, + "spec": 6, + "mtp_max": 4, + "lookup": 3, + "vram_free_mib": 3548, + "cvec": 0, + "arena_mib": 47962, + "pool_workers": 11, + "pcie_frac": "0.09", + "spec_min_p": "0.70", + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "conversation_cache_min_tokens": 0, + "tail_role_token": -1, + "vram_elastic": 0, + "engine": "0.1.43" + }, + "vram_after": { + "elastic": false + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.527, + "total_s": 1.859, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 101.5, + "prompt_tps_source": "engine", + "decode_tps": 74.1, + "decode_tps_source": "engine", + "prompt_ms": 502.5, + "decode_ms": 1350.2, + "drafts_offered": 85, + "drafts_accepted": 70, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.467, + "total_s": 1.978, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 97.5, + "prompt_tps_source": "engine", + "decode_tps": 65.4, + "decode_tps_source": "engine", + "prompt_ms": 441.1, + "decode_ms": 1529.4, + "drafts_offered": 76, + "drafts_accepted": 52, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.501, + "total_s": 2.083, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 96.6, + "prompt_tps_source": "engine", + "decode_tps": 62.5, + "decode_tps_source": "engine", + "prompt_ms": 476.0, + "decode_ms": 1599.9, + "drafts_offered": 70, + "drafts_accepted": 50, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.518, + "total_s": 1.82, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 103.2, + "prompt_tps_source": "engine", + "decode_tps": 75.8, + "decode_tps_source": "engine", + "prompt_ms": 494.0, + "decode_ms": 1319.8, + "drafts_offered": 80, + "drafts_accepted": 70, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.438, + "total_s": 1.925, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 104.1, + "prompt_tps_source": "engine", + "decode_tps": 66.4, + "decode_tps_source": "engine", + "prompt_ms": 412.9, + "decode_ms": 1506.4, + "drafts_offered": 74, + "drafts_accepted": 48, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.481, + "total_s": 1.998, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 100.8, + "prompt_tps_source": "engine", + "decode_tps": 65.1, + "decode_tps_source": "engine", + "prompt_ms": 456.2, + "decode_ms": 1535.4, + "drafts_offered": 72, + "drafts_accepted": 50, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.519, + "total_s": 1.818, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 103.2, + "prompt_tps_source": "engine", + "decode_tps": 75.9, + "decode_tps_source": "engine", + "prompt_ms": 494.2, + "decode_ms": 1317.3, + "drafts_offered": 79, + "drafts_accepted": 70, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.431, + "total_s": 1.897, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 106.0, + "prompt_tps_source": "engine", + "decode_tps": 67.4, + "decode_tps_source": "engine", + "prompt_ms": 405.7, + "decode_ms": 1484.7, + "drafts_offered": 79, + "drafts_accepted": 50, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.47, + "total_s": 1.952, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 103.3, + "prompt_tps_source": "engine", + "decode_tps": 66.6, + "decode_tps_source": "engine", + "prompt_ms": 445.2, + "decode_ms": 1500.8, + "drafts_offered": 67, + "drafts_accepted": 50, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 66.6, + "min": 62.5, + "max": 75.9, + "n": 9 + }, + "ttft_s": { + "median": 0.481, + "min": 0.431, + "max": 0.527, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 46, + "min": 43, + "max": 51, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 4.887, + "total_s": 5.267, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 827.1, + "prompt_tps_source": "engine", + "decode_tps": 60.1, + "decode_tps_source": "engine", + "prompt_ms": 4859.2, + "decode_ms": 399.6, + "drafts_offered": 15, + "drafts_accepted": 12, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.837, + "total_s": 5.204, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 835.6, + "prompt_tps_source": "engine", + "decode_tps": 62.0, + "decode_tps_source": "engine", + "prompt_ms": 4809.9, + "decode_ms": 387.1, + "drafts_offered": 13, + "drafts_accepted": 10, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.819, + "total_s": 5.162, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 838.6, + "prompt_tps_source": "engine", + "decode_tps": 66.3, + "decode_tps_source": "engine", + "prompt_ms": 4792.3, + "decode_ms": 361.8, + "drafts_offered": 13, + "drafts_accepted": 10, + "run": 2 + } + ], + "prompt_tps": { + "median": 835.6, + "min": 827.1, + "max": 838.6, + "n": 3 + }, + "ttft_s": { + "median": 4.837, + "min": 4.819, + "max": 4.887, + "n": 3 + }, + "prompt_tokens": { + "median": 4019, + "min": 4019, + "max": 4019, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 12.959, + "total_s": 13.423, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1242.2, + "prompt_tps_source": "engine", + "decode_tps": 49.7, + "decode_tps_source": "engine", + "prompt_ms": 12926.9, + "decode_ms": 482.9, + "drafts_offered": 23, + "drafts_accepted": 10, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 13.114, + "total_s": 13.49, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1227.4, + "prompt_tps_source": "engine", + "decode_tps": 60.9, + "decode_tps_source": "engine", + "prompt_ms": 13083.1, + "decode_ms": 394.1, + "drafts_offered": 19, + "drafts_accepted": 10, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 13.171, + "total_s": 13.568, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1222.1, + "prompt_tps_source": "engine", + "decode_tps": 57.9, + "decode_tps_source": "engine", + "prompt_ms": 13139.8, + "decode_ms": 414.7, + "drafts_offered": 23, + "drafts_accepted": 11, + "run": 2 + } + ], + "prompt_tps": { + "median": 1227.4, + "min": 1222.1, + "max": 1242.2, + "n": 3 + }, + "ttft_s": { + "median": 13.114, + "min": 12.959, + "max": 13.171, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "32K": { + "target_tokens": 32768, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 24.865, + "total_s": 25.403, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 1290.9, + "prompt_tps_source": "engine", + "decode_tps": 43.1, + "decode_tps_source": "engine", + "prompt_ms": 24826.3, + "decode_ms": 557.0, + "drafts_offered": 24, + "drafts_accepted": 10, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 24.921, + "total_s": 25.34, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 1288.1, + "prompt_tps_source": "engine", + "decode_tps": 54.9, + "decode_tps_source": "engine", + "prompt_ms": 24881.3, + "decode_ms": 437.0, + "drafts_offered": 18, + "drafts_accepted": 9, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 24.832, + "total_s": 25.122, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 1292.6, + "prompt_tps_source": "engine", + "decode_tps": 77.9, + "decode_tps_source": "engine", + "prompt_ms": 24794.0, + "decode_ms": 307.9, + "drafts_offered": 12, + "drafts_accepted": 12, + "run": 2 + } + ], + "prompt_tps": { + "median": 1290.9, + "min": 1288.1, + "max": 1292.6, + "n": 3 + }, + "ttft_s": { + "median": 24.865, + "min": 24.832, + "max": 24.921, + "n": 3 + }, + "prompt_tokens": { + "median": 32049, + "min": 32049, + "max": 32049, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 2.11, + "total_s": 2.508, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 24, + "prompt_tps": 292.9, + "prompt_tps_source": "engine", + "decode_tps": 57.5, + "decode_tps_source": "engine", + "prompt_ms": 2079.5, + "decode_ms": 417.2, + "drafts_offered": 22, + "drafts_accepted": 13, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.053, + "total_s": 4.964, + "prompt_tokens": 3715, + "reused_tokens": 633, + "read_tokens": 3082, + "generated_tokens": 64, + "prompt_tps": 767.6, + "prompt_tps_source": "engine", + "decode_tps": 68.9, + "decode_tps_source": "engine", + "prompt_ms": 4015.1, + "decode_ms": 929.2, + "drafts_offered": 49, + "drafts_accepted": 40, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.345, + "total_s": 4.2, + "prompt_tokens": 5248, + "reused_tokens": 3779, + "read_tokens": 1469, + "generated_tokens": 64, + "prompt_tps": 443.1, + "prompt_tps_source": "engine", + "decode_tps": 73.3, + "decode_tps_source": "engine", + "prompt_ms": 3315.1, + "decode_ms": 872.6, + "drafts_offered": 52, + "drafts_accepted": 40, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.693, + "total_s": 4.949, + "prompt_tokens": 8692, + "reused_tokens": 5311, + "read_tokens": 3381, + "generated_tokens": 20, + "prompt_tps": 725.3, + "prompt_tps_source": "engine", + "decode_tps": 73.1, + "decode_tps_source": "engine", + "prompt_ms": 4661.8, + "decode_ms": 273.5, + "drafts_offered": 14, + "drafts_accepted": 11, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.925, + "total_s": 4.165, + "prompt_tokens": 11571, + "reused_tokens": 8711, + "read_tokens": 2860, + "generated_tokens": 19, + "prompt_tps": 735.0, + "prompt_tps_source": "engine", + "decode_tps": 73.6, + "decode_tps_source": "engine", + "prompt_ms": 3891.4, + "decode_ms": 258.3, + "drafts_offered": 20, + "drafts_accepted": 12, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.907, + "total_s": 4.419, + "prompt_tokens": 14435, + "reused_tokens": 11589, + "read_tokens": 2846, + "generated_tokens": 31, + "prompt_tps": 734.5, + "prompt_tps_source": "engine", + "decode_tps": 58.6, + "decode_tps_source": "engine", + "prompt_ms": 3874.6, + "decode_ms": 529.1, + "drafts_offered": 27, + "drafts_accepted": 14, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 2.122, + "total_s": 2.425, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 19, + "prompt_tps": 290.2, + "prompt_tps_source": "engine", + "decode_tps": 59.3, + "decode_tps_source": "engine", + "prompt_ms": 2098.4, + "decode_ms": 320.3, + "drafts_offered": 22, + "drafts_accepted": 12, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.007, + "total_s": 4.751, + "prompt_tokens": 3710, + "reused_tokens": 629, + "read_tokens": 3081, + "generated_tokens": 64, + "prompt_tps": 774.4, + "prompt_tps_source": "engine", + "decode_tps": 84.0, + "decode_tps_source": "engine", + "prompt_ms": 3978.4, + "decode_ms": 762.1, + "drafts_offered": 53, + "drafts_accepted": 42, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.351, + "total_s": 4.292, + "prompt_tokens": 5243, + "reused_tokens": 3773, + "read_tokens": 1470, + "generated_tokens": 64, + "prompt_tps": 442.1, + "prompt_tps_source": "engine", + "decode_tps": 66.8, + "decode_tps_source": "engine", + "prompt_ms": 3325.2, + "decode_ms": 958.5, + "drafts_offered": 58, + "drafts_accepted": 35, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.686, + "total_s": 5.234, + "prompt_tokens": 8687, + "reused_tokens": 5306, + "read_tokens": 3381, + "generated_tokens": 43, + "prompt_tps": 725.8, + "prompt_tps_source": "engine", + "decode_tps": 76.1, + "decode_tps_source": "engine", + "prompt_ms": 4658.6, + "decode_ms": 565.1, + "drafts_offered": 32, + "drafts_accepted": 24, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.894, + "total_s": 4.143, + "prompt_tokens": 11589, + "reused_tokens": 8729, + "read_tokens": 2860, + "generated_tokens": 22, + "prompt_tps": 739.9, + "prompt_tps_source": "engine", + "decode_tps": 82.6, + "decode_tps_source": "engine", + "prompt_ms": 3865.2, + "decode_ms": 266.5, + "drafts_offered": 19, + "drafts_accepted": 14, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.889, + "total_s": 4.619, + "prompt_tokens": 14456, + "reused_tokens": 11610, + "read_tokens": 2846, + "generated_tokens": 45, + "prompt_tps": 737.6, + "prompt_tps_source": "engine", + "decode_tps": 60.2, + "decode_tps_source": "engine", + "prompt_ms": 3858.3, + "decode_ms": 747.8, + "drafts_offered": 40, + "drafts_accepted": 20, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 2.123, + "total_s": 2.468, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 21, + "prompt_tps": 290.2, + "prompt_tps_source": "engine", + "decode_tps": 57.7, + "decode_tps_source": "engine", + "prompt_ms": 2098.6, + "decode_ms": 363.7, + "drafts_offered": 24, + "drafts_accepted": 13, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.04, + "total_s": 4.799, + "prompt_tokens": 3712, + "reused_tokens": 631, + "read_tokens": 3081, + "generated_tokens": 64, + "prompt_tps": 767.5, + "prompt_tps_source": "engine", + "decode_tps": 82.4, + "decode_tps_source": "engine", + "prompt_ms": 4014.2, + "decode_ms": 776.9, + "drafts_offered": 48, + "drafts_accepted": 40, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.343, + "total_s": 4.154, + "prompt_tokens": 5245, + "reused_tokens": 3775, + "read_tokens": 1470, + "generated_tokens": 64, + "prompt_tps": 443.2, + "prompt_tps_source": "engine", + "decode_tps": 77.2, + "decode_tps_source": "engine", + "prompt_ms": 3316.7, + "decode_ms": 828.6, + "drafts_offered": 55, + "drafts_accepted": 42, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.929, + "total_s": 5.816, + "prompt_tokens": 8689, + "reused_tokens": 5238, + "read_tokens": 3451, + "generated_tokens": 64, + "prompt_tps": 704.4, + "prompt_tps_source": "engine", + "decode_tps": 70.8, + "decode_tps_source": "engine", + "prompt_ms": 4899.1, + "decode_ms": 904.1, + "drafts_offered": 52, + "drafts_accepted": 34, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.895, + "total_s": 4.077, + "prompt_tokens": 11613, + "reused_tokens": 8753, + "read_tokens": 2860, + "generated_tokens": 17, + "prompt_tps": 739.9, + "prompt_tps_source": "engine", + "decode_tps": 85.5, + "decode_tps_source": "engine", + "prompt_ms": 3865.3, + "decode_ms": 198.8, + "drafts_offered": 15, + "drafts_accepted": 11, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.915, + "total_s": 4.193, + "prompt_tokens": 14475, + "reused_tokens": 11629, + "read_tokens": 2846, + "generated_tokens": 15, + "prompt_tps": 732.5, + "prompt_tps_source": "engine", + "decode_tps": 50.9, + "decode_tps_source": "engine", + "prompt_ms": 3885.5, + "decode_ms": 294.9, + "drafts_offered": 16, + "drafts_accepted": 5, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 68.764, + "min": 68.656, + "max": 68.781, + "n": 3 + }, + "reread_tokens": { + "median": 4, + "min": 3, + "max": 73, + "n": 3 + }, + "new_tokens": { + "median": 13635, + "min": 13634, + "max": 13635, + "n": 3 + }, + "turn_ttft_s": { + "median": 3.915, + "min": 3.343, + "max": 4.929, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 14456, + "min": 14435, + "max": 14475, + "n": 3 + }, + "turn_decode_tps": { + "median": 73.3, + "min": 50.9, + "max": 85.5, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 2.122, + "min": 2.11, + "max": 2.123, + "n": 3 + } + }, + "clients": {}, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/B1c-after-full.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/B1c-after-full.json new file mode 100644 index 00000000000..93782bb0a6e --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/B1c-after-full.json @@ -0,0 +1 @@ +{"engine": {"model": "qwen3.8-flash-next-iq3_s", "max_context": 524288, "images": true, "version": "0.1.43", "logit_bias": 1, "context": 524288, "kv": "int8", "kv_resident": 20480, "expert_slots": 5611, "expert_cache_mib": 10665, "expert_slots_primary": 3072, "expert_cache_primary_mib": 5650, "spec": 6, "mtp_max": 4, "lookup": 3, "vram_free_mib": 3162, "cvec": 0, "arena_mib": 47962, "pool_workers": 11, "pcie_frac": "0.55", "spec_min_p": "0.70", "conversation_cache_mib": 0, "conversation_cache_slots": 4, "conversation_cache_min_free_mib": 2560, "conversation_cache_min_tokens": 0, "tail_role_token": -1, "vram_elastic": 0, "engine": "0.1.43"}, "live": {"state": "idle", "queued": 0, "phase": null, "prompt_tokens": null, "prompt_read": null, "prompt_total": null, "generated": null, "max_tokens": null, "elapsed_s": null, "tok_s": null, "tok_s_mean": null, "prefill_tok_s_mean": null, "tok_s_window_s": null}, "requests": [{"projection": null, "time": 1791785372.6024227, "duration_s": 79.3, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 78570.6, "decode_ms": 739.9, "decode_tok_s": 32.4, "hit_rate": 0.854, "pcie_share": 0.001, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 21, "drafts_accepted": 10}, {"projection": null, "time": 1791785293.3335884, "duration_s": 79.2, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 78499.3, "decode_ms": 744.1, "decode_tok_s": 32.3, "hit_rate": 0.791, "pcie_share": 0.001, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 16, "drafts_accepted": 8}, {"projection": null, "time": 1791785213.7421708, "duration_s": 79.6, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 78778.7, "decode_ms": 785.2, "decode_tok_s": 30.6, "hit_rate": 0.575, "pcie_share": 0.002, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 17, "drafts_accepted": 9}, {"projection": null, "time": 1791785196.952123, "duration_s": 16.8, "finish": "stop", "prompt_tokens": 14469, "reused": 11623, "output_tokens": 49, "prompt_total": 14469, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 49, "prompt_ms": 15312.2, "decode_ms": 1454.5, "decode_tok_s": 33.7, "hit_rate": 0.727, "pcie_share": 0.005, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 49, "drafts_accepted": 24}, {"projection": null, "time": 1791785181.1321237, "duration_s": 15.8, "finish": "stop", "prompt_tokens": 11604, "reused": 8744, "output_tokens": 20, "prompt_total": 11604, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 20, "prompt_ms": 15348.0, "decode_ms": 458.7, "decode_tok_s": 43.6, "hit_rate": 0.792, "pcie_share": 0.003, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 18, "drafts_accepted": 12}, {"projection": null, "time": 1791785163.7413445, "duration_s": 17.4, "finish": "length", "prompt_tokens": 8680, "reused": 5229, "output_tokens": 64, "prompt_total": 8680, "prompt_read": 3451, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 15610.6, "decode_ms": 1768.7, "decode_tok_s": 36.2, "hit_rate": 0.756, "pcie_share": 0.003, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 55, "drafts_accepted": 32}, {"projection": null, "time": 1791785147.693134, "duration_s": 16.0, "finish": "length", "prompt_tokens": 5236, "reused": 3767, "output_tokens": 64, "prompt_total": 5236, "prompt_read": 1469, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 14569.0, "decode_ms": 1467.6, "decode_tok_s": 43.6, "hit_rate": 0.694, "pcie_share": 0.006, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 51, "drafts_accepted": 41}, {"projection": null, "time": 1791785130.921754, "duration_s": 16.8, "finish": "length", "prompt_tokens": 3703, "reused": 623, "output_tokens": 64, "prompt_total": 3703, "prompt_read": 3080, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 15318.1, "decode_ms": 1444.3, "decode_tok_s": 44.3, "hit_rate": 0.705, "pcie_share": 0.006, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 49, "drafts_accepted": 39}, {"projection": null, "time": 1791785119.6294875, "duration_s": 11.3, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 12, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 12, "prompt_ms": 10943.6, "decode_ms": 340.5, "decode_tok_s": 35.2, "hit_rate": 0.487, "pcie_share": 0.011, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 14, "drafts_accepted": 9}, {"projection": null, "time": 1791785103.8549085, "duration_s": 15.8, "finish": "stop", "prompt_tokens": 14453, "reused": 11607, "output_tokens": 12, "prompt_total": 14453, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 12, "prompt_ms": 15318.0, "decode_ms": 442.6, "decode_tok_s": 27.1, "hit_rate": 0.756, "pcie_share": 0.002, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 10, "drafts_accepted": 3}, {"projection": null, "time": 1791785088.2610857, "duration_s": 15.6, "finish": "stop", "prompt_tokens": 11593, "reused": 8733, "output_tokens": 15, "prompt_total": 11593, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 15, "prompt_ms": 15293.0, "decode_ms": 287.6, "decode_tok_s": 52.2, "hit_rate": 0.791, "pcie_share": 0.003, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 12, "drafts_accepted": 9}, {"projection": null, "time": 1791785071.4616587, "duration_s": 16.8, "finish": "stop", "prompt_tokens": 8687, "reused": 5236, "output_tokens": 47, "prompt_total": 8687, "prompt_read": 3451, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 47, "prompt_ms": 15568.5, "decode_ms": 1217.7, "decode_tok_s": 38.6, "hit_rate": 0.766, "pcie_share": 0.002, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 39, "drafts_accepted": 25}, {"projection": null, "time": 1791785055.3026884, "duration_s": 16.2, "finish": "length", "prompt_tokens": 5243, "reused": 3773, "output_tokens": 64, "prompt_total": 5243, "prompt_read": 1470, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 14478.1, "decode_ms": 1670.1, "decode_tok_s": 38.3, "hit_rate": 0.691, "pcie_share": 0.006, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 61, "drafts_accepted": 39}, {"projection": null, "time": 1791785038.7035003, "duration_s": 16.6, "finish": "length", "prompt_tokens": 3710, "reused": 630, "output_tokens": 64, "prompt_total": 3710, "prompt_read": 3080, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 15331.6, "decode_ms": 1257.6, "decode_tok_s": 50.9, "hit_rate": 0.658, "pcie_share": 0.008, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 49, "drafts_accepted": 44}, {"projection": null, "time": 1791785027.262812, "duration_s": 11.4, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 19, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 19, "prompt_ms": 10891.7, "decode_ms": 541.2, "decode_tok_s": 35.1, "hit_rate": 0.537, "pcie_share": 0.008, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 20, "drafts_accepted": 13}, {"projection": null, "time": 1791785009.4407933, "duration_s": 17.8, "finish": "stop", "prompt_tokens": 14454, "reused": 11608, "output_tokens": 32, "prompt_total": 14454, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 32, "prompt_ms": 16827.3, "decode_ms": 982.0, "decode_tok_s": 32.6, "hit_rate": 0.755, "pcie_share": 0.003, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 27, "drafts_accepted": 14}, {"projection": null, "time": 1791784991.6134655, "duration_s": 17.8, "finish": "stop", "prompt_tokens": 11594, "reused": 8734, "output_tokens": 15, "prompt_total": 11594, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 15, "prompt_ms": 17451.8, "decode_ms": 358.7, "decode_tok_s": 41.8, "hit_rate": 0.788, "pcie_share": 0.003, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 14, "drafts_accepted": 9}, {"projection": null, "time": 1791784972.373388, "duration_s": 19.2, "finish": "stop", "prompt_tokens": 8686, "reused": 5306, "output_tokens": 49, "prompt_total": 8686, "prompt_read": 3380, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 49, "prompt_ms": 17852.7, "decode_ms": 1371.0, "decode_tok_s": 35.7, "hit_rate": 0.741, "pcie_share": 0.003, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 44, "drafts_accepted": 26}, {"projection": null, "time": 1791784954.6767802, "duration_s": 17.7, "finish": "length", "prompt_tokens": 5242, "reused": 3772, "output_tokens": 64, "prompt_total": 5242, "prompt_read": 1470, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 15934.1, "decode_ms": 1748.4, "decode_tok_s": 36.6, "hit_rate": 0.631, "pcie_share": 0.008, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 52, "drafts_accepted": 37}, {"projection": null, "time": 1791784934.8033667, "duration_s": 19.9, "finish": "length", "prompt_tokens": 3709, "reused": 629, "output_tokens": 64, "prompt_total": 3709, "prompt_read": 3080, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 18260.8, "decode_ms": 1597.6, "decode_tok_s": 40.1, "hit_rate": 0.493, "pcie_share": 0.012, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 47, "drafts_accepted": 40}, {"projection": null, "time": 1791784922.5789607, "duration_s": 12.2, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 18, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 18, "prompt_ms": 11572.1, "decode_ms": 633.0, "decode_tok_s": 28.4, "hit_rate": 0.464, "pcie_share": 0.012, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 22, "drafts_accepted": 12}, {"projection": null, "time": 1791784875.5082505, "duration_s": 47.1, "finish": "length", "prompt_tokens": 16058, "reused": 0, "output_tokens": 24, "prompt_total": 16058, "prompt_read": 16058, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 46387.7, "decode_ms": 666.1, "decode_tok_s": 36.0, "hit_rate": 0.879, "pcie_share": 0.001, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 17, "drafts_accepted": 10}, {"projection": null, "time": 1791784828.2874928, "duration_s": 47.2, "finish": "length", "prompt_tokens": 16058, "reused": 0, "output_tokens": 24, "prompt_total": 16058, "prompt_read": 16058, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 46309.1, "decode_ms": 895.0, "decode_tok_s": 26.8, "hit_rate": 0.828, "pcie_share": 0.001, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 22, "drafts_accepted": 10}, {"projection": null, "time": 1791784780.1967735, "duration_s": 48.1, "finish": "length", "prompt_tokens": 16058, "reused": 0, "output_tokens": 24, "prompt_total": 16058, "prompt_read": 16058, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 47389.5, "decode_ms": 685.0, "decode_tok_s": 35.0, "hit_rate": 0.778, "pcie_share": 0.001, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 14, "drafts_accepted": 12}, {"projection": null, "time": 1791784764.3791242, "duration_s": 15.8, "finish": "length", "prompt_tokens": 4019, "reused": 0, "output_tokens": 24, "prompt_total": 4019, "prompt_read": 4019, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 15224.5, "decode_ms": 578.1, "decode_tok_s": 41.5, "hit_rate": 0.8, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 13, "drafts_accepted": 11}, {"projection": null, "time": 1791784748.5692549, "duration_s": 15.8, "finish": "length", "prompt_tokens": 4019, "reused": 0, "output_tokens": 24, "prompt_total": 4019, "prompt_read": 4019, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 15181.2, "decode_ms": 620.2, "decode_tok_s": 38.7, "hit_rate": 0.681, "pcie_share": 0.001, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 13, "drafts_accepted": 10}, {"projection": null, "time": 1791784731.1269796, "duration_s": 17.4, "finish": "length", "prompt_tokens": 4019, "reused": 0, "output_tokens": 24, "prompt_total": 4019, "prompt_read": 4019, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 16702.4, "decode_ms": 730.8, "decode_tok_s": 32.8, "hit_rate": 0.444, "pcie_share": 0.003, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 12, "drafts_accepted": 10}, {"projection": null, "time": 1791784728.0026345, "duration_s": 3.1, "finish": "length", "prompt_tokens": 46, "reused": 0, "output_tokens": 100, "prompt_total": 46, "prompt_read": 46, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 708.2, "decode_ms": 2407.9, "decode_tok_s": 41.5, "hit_rate": 0.65, "pcie_share": 0.005, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 81, "drafts_accepted": 53}, {"projection": null, "time": 1791784724.5885704, "duration_s": 3.4, "finish": "length", "prompt_tokens": 43, "reused": 0, "output_tokens": 100, "prompt_total": 43, "prompt_read": 43, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 599.3, "decode_ms": 2808.5, "decode_tok_s": 35.6, "hit_rate": 0.708, "pcie_share": 0.003, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 71, "drafts_accepted": 45}, {"projection": null, "time": 1791784721.8807886, "duration_s": 2.7, "finish": "length", "prompt_tokens": 51, "reused": 0, "output_tokens": 100, "prompt_total": 51, "prompt_read": 51, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 696.7, "decode_ms": 2004.8, "decode_tok_s": 49.9, "hit_rate": 0.509, "pcie_share": 0.014, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 75, "drafts_accepted": 70}, {"projection": null, "time": 1791784718.7596223, "duration_s": 3.1, "finish": "length", "prompt_tokens": 46, "reused": 0, "output_tokens": 100, "prompt_total": 46, "prompt_read": 46, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 698.0, "decode_ms": 2416.8, "decode_tok_s": 41.4, "hit_rate": 0.64, "pcie_share": 0.005, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 80, "drafts_accepted": 54}, {"projection": null, "time": 1791784715.4855578, "duration_s": 3.3, "finish": "length", "prompt_tokens": 43, "reused": 0, "output_tokens": 100, "prompt_total": 43, "prompt_read": 43, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 573.2, "decode_ms": 2694.1, "decode_tok_s": 37.1, "hit_rate": 0.657, "pcie_share": 0.005, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 76, "drafts_accepted": 50}, {"projection": null, "time": 1791784712.7316835, "duration_s": 2.7, "finish": "length", "prompt_tokens": 51, "reused": 0, "output_tokens": 100, "prompt_total": 51, "prompt_read": 51, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 693.6, "decode_ms": 2053.2, "decode_tok_s": 48.7, "hit_rate": 0.492, "pcie_share": 0.015, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 79, "drafts_accepted": 70}, {"projection": null, "time": 1791784709.3067627, "duration_s": 3.4, "finish": "length", "prompt_tokens": 46, "reused": 0, "output_tokens": 100, "prompt_total": 46, "prompt_read": 46, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 697.8, "decode_ms": 2720.8, "decode_tok_s": 36.8, "hit_rate": 0.613, "pcie_share": 0.007, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 79, "drafts_accepted": 54}, {"projection": null, "time": 1791784705.8215296, "duration_s": 3.5, "finish": "length", "prompt_tokens": 43, "reused": 0, "output_tokens": 100, "prompt_total": 43, "prompt_read": 43, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 680.4, "decode_ms": 2797.8, "decode_tok_s": 35.7, "hit_rate": 0.654, "pcie_share": 0.006, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 80, "drafts_accepted": 50}, {"projection": null, "time": 1791784702.947718, "duration_s": 2.9, "finish": "length", "prompt_tokens": 51, "reused": 0, "output_tokens": 100, "prompt_total": 51, "prompt_read": 51, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 749.2, "decode_ms": 2117.5, "decode_tok_s": 47.2, "hit_rate": 0.561, "pcie_share": 0.013, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 86, "drafts_accepted": 70}, {"projection": null, "time": 1791784695.4038546, "duration_s": 7.5, "finish": "length", "prompt_tokens": 1028, "reused": 0, "output_tokens": 8, "prompt_total": 1028, "prompt_read": 1028, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 8, "prompt_ms": 7207.0, "decode_ms": 330.7, "decode_tok_s": 24.2, "hit_rate": 0.408, "pcie_share": 0.012, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 8, "drafts_accepted": 3}, {"projection": null, "time": 1791784687.939419, "duration_s": 7.5, "finish": "length", "prompt_tokens": 906, "reused": 0, "output_tokens": 1, "prompt_total": 906, "prompt_read": 906, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 1, "prompt_ms": 7434.3, "decode_ms": 23.4, "decode_tok_s": 42.7, "hit_rate": 0.515, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 0, "drafts_accepted": 0}, {"projection": null, "time": 1791784681.8580742, "duration_s": 6.1, "finish": "length", "prompt_tokens": 260, "reused": 0, "output_tokens": 1, "prompt_total": 260, "prompt_read": 260, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 1, "prompt_ms": 6048.3, "decode_ms": 23.3, "decode_tok_s": 42.9, "hit_rate": 0.515, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 0, "drafts_accepted": 0}, {"projection": null, "time": 1791784680.3306918, "duration_s": 1.5, "finish": "length", "prompt_tokens": 34, "reused": 0, "output_tokens": 24, "prompt_total": 34, "prompt_read": 34, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 878.1, "decode_ms": 639.7, "decode_tok_s": 37.5, "hit_rate": 0.309, "pcie_share": 0.019, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 18, "drafts_accepted": 17}], "conversation_cache": {"enabled": false, "budget_mib": 0, "slots": 4, "parked": 0, "bytes": 0, "evictions": 0, "parks": 0, "restores": 0, "last_event": null, "last_tokens": null, "last_at": null, "requests": 40, "requests_reused": 15, "reused_tokens": 90014, "prompt_tokens": 291916, "last_reused": 0, "last_prompt": 32049}, "requests_kept": 40, "totals": {"since": 1791784504.3221655, "requests": 40, "prompt_tokens": 291916, "reused": 90014, "output_tokens": 1886, "prompt_ms": 722588.2999999999, "decode_ms": 48524.69999999998, "drafts_offered": 1511, "drafts_accepted": 1054}, "hardware": {"gpus": [{"index": 0, "util": 0, "mem_used": 13834977280, "mem_total": 17103323136, "temp": 40, "power": 9.502}, {"index": 1, "util": 100, "mem_used": 12555190272, "mem_total": 12884901888, "temp": 56, "power": 51.511}], "gpu_util": 50.0, "gpu_mem_used": 26390167552, "gpu_mem_total": 29988225024, "gpu_temp": 56, "gpu_power": 61.013000000000005, "gpu_power_limit": 350.0, "gpu_pcie_gen": 5, "gpu_pcie_gen_max": 5, "gpu_pcie_width": 8, "gpu_pcie_rx_mb": 3300.107421875, "gpu_pcie_tx_mb": 278.5888671875, "cpu": 4.3, "ram_used": 64850591744, "ram_total": 66472062976, "disk_read_mb": 0.0, "disk_write_mb": 0.0, "tok_s": 0.0, "tok_s_mean": 0, "prefill_tok_s_mean": 416.3}, "hardware_static": {"gpu_name": "NVIDIA GeForce RTX 5060 Ti + NVIDIA GeForce RTX 3060", "gpu_count": 2, "gpu_note": null, "cpu_name": "AMD Ryzen 9 7900X 12-Core Processor", "cores": 12, "threads": 24, "psutil": true}, "history": {"gpu_util": [50.0, 70.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0], "gpu_mem_used": [26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552], "gpu_temp": [59, 59, 61, 61, 61, 60, 59, 62, 59, 58, 61, 62, 62, 62, 60, 63, 62, 59, 62, 60, 62, 62, 60, 62, 61, 61, 60, 60, 62, 60, 64, 61, 62, 62, 60, 61, 62, 63, 61, 60, 63, 59, 60, 60, 62, 59, 59, 59, 59, 58, 58, 58, 58, 59, 57, 58, 57, 57, 60, 56], "gpu_power": [107.58, 165.05, 138.0, 126.73, 120.77, 110.11, 109.62, 114.25, 111.14, 99.71, 118.85, 116.57, 107.86, 106.97, 118.02, 117.86, 178.65, 143.97, 112.49, 122.34, 114.23, 112.51, 118.67, 113.34, 113.96, 105.04, 114.19, 112.9, 109.75, 106.32, 139.25, 134.47, 122.99, 117.26, 113.29, 114.22, 113.79, 111.1, 112.06, 114.79, 112.21, 111.0, 110.09, 108.53, 102.34, 116.71, 86.99, 77.22, 71.83, 71.28, 70.82, 65.68, 68.93, 72.78, 70.25, 70.22, 64.85, 68.63, 77.04, 61.01], "gpu_pcie_rx_mb": [62.81, 10.0, 3291.09, 3302.29, 3305.95, 3311.11, 101.26, 3293.95, 3304.19, 3295.99, 3299.32, 3296.7, 3308.09, 3379.63, 3302.08, 3340.88, 3292.71, 3294.49, 3300.27, 3310.39, 3359.49, 3300.48, 18.24, 3288.16, 3301.26, 3289.79, 11.95, 3299.14, 3302.7, 3301.05, 3309.07, 3285.82, 3340.51, 3334.4, 24.53, 3294.06, 3350.75, 3298.47, 13.52, 3305.22, 3294.83, 3296.78, 3309.19, 3293.07, 63.0, 3287.93, 3288.07, 3292.09, 3297.08, 3295.55, 3298.84, 3302.65, 3296.83, 3292.87, 3303.9, 3296.78, 3295.59, 3301.98, 3298.07, 3300.11], "cpu": [4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.3, 4.3], "ram_used": [64648450048, 64648450048, 64648450048, 64648450048, 64648450048, 64648450048, 64648450048, 64648450048, 64648450048, 64648450048, 64648028160, 64648028160, 64648445952, 64648495104, 64648495104, 64648495104, 64648237056, 64648237056, 64648237056, 64648134656, 64648134656, 64648134656, 64648134656, 64648134656, 64648134656, 64648134656, 64648548352, 64648908800, 64648908800, 64648908800, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745164800, 64745025536, 64850591744, 64850591744], "disk_read_mb": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 15.69, 0.0, 23.19, 0.0, 0.0, 0.03, 0.0, 0.01, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.01, 0.0, 0.0, 0.0, 0.0, 0.0, 0.01, 0.0, 0.03, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 8.66, 0.0, 0.0, 0.0], "tok_s": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "prefill_tok_s_mean": [null, 499.0, 499.0, 499.0, 499.0, 499.0, 499.0, 499.0, 499.0, 499.0, 499.0, 499.0, 499.0, 499.0, 499.0, 485.3, 485.3, 485.3, 485.3, 485.3, 485.3, 485.3, 485.3, 485.3, 485.3, 485.3, 485.3, 485.3, 485.3, 485.3, 479.4, 479.4, 479.4, 479.4, 479.4, 479.4, 479.4, 479.4, 479.4, 479.4, 479.4, 479.4, 479.4, 479.4, 479.4, 476.5, 476.5, 476.5, 476.5, 476.5, 476.5, 476.5, 476.5, 476.5, 476.5, 476.5, 476.5, 476.5, 416.3, 416.3]}, "time": 1791785451.9656045} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/B1c-after.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/B1c-after.json new file mode 100644 index 00000000000..a33eba6450b --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/B1c-after.json @@ -0,0 +1 @@ +{"engine": {"model": "qwen3.8-flash-next-iq3_s", "max_context": 524288, "images": true, "version": "0.1.43", "logit_bias": 1, "context": 524288, "kv": "int8", "kv_resident": 20480, "expert_slots": 5611, "expert_cache_mib": 10665, "expert_slots_primary": 3072, "expert_cache_primary_mib": 5650, "spec": 6, "mtp_max": 4, "lookup": 3, "vram_free_mib": 3162, "cvec": 0, "arena_mib": 47962, "pool_workers": 11, "pcie_frac": "0.55", "spec_min_p": "0.70", "conversation_cache_mib": 0, "conversation_cache_slots": 4, "conversation_cache_min_free_mib": 2560, "conversation_cache_min_tokens": 0, "tail_role_token": -1, "vram_elastic": 0, "engine": "0.1.43"}, "live": {"state": "idle", "queued": 0, "phase": null, "prompt_tokens": null, "prompt_read": null, "prompt_total": null, "generated": null, "max_tokens": null, "elapsed_s": null, "tok_s": null, "tok_s_mean": null, "prefill_tok_s_mean": null, "tok_s_window_s": null}, "requests": [{"projection": null, "time": 1791785372.6024227, "duration_s": 79.3, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 78570.6, "decode_ms": 739.9, "decode_tok_s": 32.4, "hit_rate": 0.854, "pcie_share": 0.001, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 21, "drafts_accepted": 10}, {"projection": null, "time": 1791785293.3335884, "duration_s": 79.2, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 78499.3, "decode_ms": 744.1, "decode_tok_s": 32.3, "hit_rate": 0.791, "pcie_share": 0.001, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 16, "drafts_accepted": 8}, {"projection": null, "time": 1791785213.7421708, "duration_s": 79.6, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 78778.7, "decode_ms": 785.2, "decode_tok_s": 30.6, "hit_rate": 0.575, "pcie_share": 0.002, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 17, "drafts_accepted": 9}, {"projection": null, "time": 1791785196.952123, "duration_s": 16.8, "finish": "stop", "prompt_tokens": 14469, "reused": 11623, "output_tokens": 49, "prompt_total": 14469, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 49, "prompt_ms": 15312.2, "decode_ms": 1454.5, "decode_tok_s": 33.7, "hit_rate": 0.727, "pcie_share": 0.005, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 49, "drafts_accepted": 24}, {"projection": null, "time": 1791785181.1321237, "duration_s": 15.8, "finish": "stop", "prompt_tokens": 11604, "reused": 8744, "output_tokens": 20, "prompt_total": 11604, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 20, "prompt_ms": 15348.0, "decode_ms": 458.7, "decode_tok_s": 43.6, "hit_rate": 0.792, "pcie_share": 0.003, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 18, "drafts_accepted": 12}, {"projection": null, "time": 1791785163.7413445, "duration_s": 17.4, "finish": "length", "prompt_tokens": 8680, "reused": 5229, "output_tokens": 64, "prompt_total": 8680, "prompt_read": 3451, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 15610.6, "decode_ms": 1768.7, "decode_tok_s": 36.2, "hit_rate": 0.756, "pcie_share": 0.003, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 55, "drafts_accepted": 32}, {"projection": null, "time": 1791785147.693134, "duration_s": 16.0, "finish": "length", "prompt_tokens": 5236, "reused": 3767, "output_tokens": 64, "prompt_total": 5236, "prompt_read": 1469, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 14569.0, "decode_ms": 1467.6, "decode_tok_s": 43.6, "hit_rate": 0.694, "pcie_share": 0.006, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 51, "drafts_accepted": 41}, {"projection": null, "time": 1791785130.921754, "duration_s": 16.8, "finish": "length", "prompt_tokens": 3703, "reused": 623, "output_tokens": 64, "prompt_total": 3703, "prompt_read": 3080, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 15318.1, "decode_ms": 1444.3, "decode_tok_s": 44.3, "hit_rate": 0.705, "pcie_share": 0.006, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 49, "drafts_accepted": 39}, {"projection": null, "time": 1791785119.6294875, "duration_s": 11.3, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 12, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 12, "prompt_ms": 10943.6, "decode_ms": 340.5, "decode_tok_s": 35.2, "hit_rate": 0.487, "pcie_share": 0.011, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 14, "drafts_accepted": 9}, {"projection": null, "time": 1791785103.8549085, "duration_s": 15.8, "finish": "stop", "prompt_tokens": 14453, "reused": 11607, "output_tokens": 12, "prompt_total": 14453, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 12, "prompt_ms": 15318.0, "decode_ms": 442.6, "decode_tok_s": 27.1, "hit_rate": 0.756, "pcie_share": 0.002, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 10, "drafts_accepted": 3}, {"projection": null, "time": 1791785088.2610857, "duration_s": 15.6, "finish": "stop", "prompt_tokens": 11593, "reused": 8733, "output_tokens": 15, "prompt_total": 11593, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 15, "prompt_ms": 15293.0, "decode_ms": 287.6, "decode_tok_s": 52.2, "hit_rate": 0.791, "pcie_share": 0.003, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 12, "drafts_accepted": 9}, {"projection": null, "time": 1791785071.4616587, "duration_s": 16.8, "finish": "stop", "prompt_tokens": 8687, "reused": 5236, "output_tokens": 47, "prompt_total": 8687, "prompt_read": 3451, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 47, "prompt_ms": 15568.5, "decode_ms": 1217.7, "decode_tok_s": 38.6, "hit_rate": 0.766, "pcie_share": 0.002, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 39, "drafts_accepted": 25}], "conversation_cache": {"enabled": false, "budget_mib": 0, "slots": 4, "parked": 0, "bytes": 0, "evictions": 0, "parks": 0, "restores": 0, "last_event": null, "last_tokens": null, "last_at": null, "requests": 40, "requests_reused": 15, "reused_tokens": 90014, "prompt_tokens": 291916, "last_reused": 0, "last_prompt": 32049}, "requests_kept": 40, "totals": {"since": 1791784504.3221655, "requests": 40, "prompt_tokens": 291916, "reused": 90014, "output_tokens": 1886, "prompt_ms": 722588.2999999999, "decode_ms": 48524.69999999998, "drafts_offered": 1511, "drafts_accepted": 1054}, "hardware": {"gpus": [{"index": 0, "util": 0, "mem_used": 13834977280, "mem_total": 17103323136, "temp": 40, "power": 9.502}, {"index": 1, "util": 100, "mem_used": 12555190272, "mem_total": 12884901888, "temp": 56, "power": 51.511}], "gpu_util": 50.0, "gpu_mem_used": 26390167552, "gpu_mem_total": 29988225024, "gpu_temp": 56, "gpu_power": 61.013000000000005, "gpu_power_limit": 350.0, "gpu_pcie_gen": 5, "gpu_pcie_gen_max": 5, "gpu_pcie_width": 8, "gpu_pcie_rx_mb": 3300.107421875, "gpu_pcie_tx_mb": 278.5888671875, "cpu": 4.3, "ram_used": 64850591744, "ram_total": 66472062976, "disk_read_mb": 0.0, "disk_write_mb": 0.0, "tok_s": 0.0, "tok_s_mean": 0, "prefill_tok_s_mean": 416.3}, "hardware_static": {"gpu_name": "NVIDIA GeForce RTX 5060 Ti + NVIDIA GeForce RTX 3060", "gpu_count": 2, "gpu_note": null, "cpu_name": "AMD Ryzen 9 7900X 12-Core Processor", "cores": 12, "threads": 24, "psutil": true}, "history": {"gpu_util": [50.0, 70.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0], "gpu_mem_used": [26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552, 26390167552], "gpu_temp": [59, 59, 61, 61, 61, 60, 59, 62, 59, 58, 61, 62, 62, 62, 60, 63, 62, 59, 62, 60, 62, 62, 60, 62, 61, 61, 60, 60, 62, 60, 64, 61, 62, 62, 60, 61, 62, 63, 61, 60, 63, 59, 60, 60, 62, 59, 59, 59, 59, 58, 58, 58, 58, 59, 57, 58, 57, 57, 60, 56], "gpu_power": [107.58, 165.05, 138.0, 126.73, 120.77, 110.11, 109.62, 114.25, 111.14, 99.71, 118.85, 116.57, 107.86, 106.97, 118.02, 117.86, 178.65, 143.97, 112.49, 122.34, 114.23, 112.51, 118.67, 113.34, 113.96, 105.04, 114.19, 112.9, 109.75, 106.32, 139.25, 134.47, 122.99, 117.26, 113.29, 114.22, 113.79, 111.1, 112.06, 114.79, 112.21, 111.0, 110.09, 108.53, 102.34, 116.71, 86.99, 77.22, 71.83, 71.28, 70.82, 65.68, 68.93, 72.78, 70.25, 70.22, 64.85, 68.63, 77.04, 61.01], "gpu_pcie_rx_mb": [62.81, 10.0, 3291.09, 3302.29, 3305.95, 3311.11, 101.26, 3293.95, 3304.19, 3295.99, 3299.32, 3296.7, 3308.09, 3379.63, 3302.08, 3340.88, 3292.71, 3294.49, 3300.27, 3310.39, 3359.49, 3300.48, 18.24, 3288.16, 3301.26, 3289.79, 11.95, 3299.14, 3302.7, 3301.05, 3309.07, 3285.82, 3340.51, 3334.4, 24.53, 3294.06, 3350.75, 3298.47, 13.52, 3305.22, 3294.83, 3296.78, 3309.19, 3293.07, 63.0, 3287.93, 3288.07, 3292.09, 3297.08, 3295.55, 3298.84, 3302.65, 3296.83, 3292.87, 3303.9, 3296.78, 3295.59, 3301.98, 3298.07, 3300.11], "cpu": [4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.3, 4.3], "ram_used": [64648450048, 64648450048, 64648450048, 64648450048, 64648450048, 64648450048, 64648450048, 64648450048, 64648450048, 64648450048, 64648028160, 64648028160, 64648445952, 64648495104, 64648495104, 64648495104, 64648237056, 64648237056, 64648237056, 64648134656, 64648134656, 64648134656, 64648134656, 64648134656, 64648134656, 64648134656, 64648548352, 64648908800, 64648908800, 64648908800, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745635840, 64745164800, 64745025536, 64850591744, 64850591744], "disk_read_mb": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 15.69, 0.0, 23.19, 0.0, 0.0, 0.03, 0.0, 0.01, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.01, 0.0, 0.0, 0.0, 0.0, 0.0, 0.01, 0.0, 0.03, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 8.66, 0.0, 0.0, 0.0], "tok_s": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "prefill_tok_s_mean": [null, 499.0, 499.0, 499.0, 499.0, 499.0, 499.0, 499.0, 499.0, 499.0, 499.0, 499.0, 499.0, 499.0, 499.0, 485.3, 485.3, 485.3, 485.3, 485.3, 485.3, 485.3, 485.3, 485.3, 485.3, 485.3, 485.3, 485.3, 485.3, 485.3, 479.4, 479.4, 479.4, 479.4, 479.4, 479.4, 479.4, 479.4, 479.4, 479.4, 479.4, 479.4, 479.4, 479.4, 479.4, 476.5, 476.5, 476.5, 476.5, 476.5, 476.5, 476.5, 476.5, 476.5, 476.5, 476.5, 476.5, 476.5, 416.3, 416.3]}, "time": 1791785451.9776397} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/B1c-before.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/B1c-before.json new file mode 100644 index 00000000000..62ef642dcd9 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/B1c-before.json @@ -0,0 +1 @@ +{"engine": {"model": "qwen3.8-flash-next-iq3_s", "max_context": 524288, "images": true, "version": "0.1.43", "logit_bias": 1, "context": 524288, "kv": "int8", "kv_resident": 20480, "expert_slots": 5611, "expert_cache_mib": 10665, "expert_slots_primary": 3072, "expert_cache_primary_mib": 5650, "spec": 6, "mtp_max": 4, "lookup": 3, "vram_free_mib": 3162, "cvec": 0, "arena_mib": 47962, "pool_workers": 11, "pcie_frac": "0.55", "spec_min_p": "0.70", "conversation_cache_mib": 0, "conversation_cache_slots": 4, "conversation_cache_min_free_mib": 2560, "conversation_cache_min_tokens": 0, "tail_role_token": -1, "vram_elastic": 0, "engine": "0.1.43"}, "live": {"state": "idle", "queued": 0, "phase": null, "prompt_tokens": null, "prompt_read": null, "prompt_total": null, "generated": null, "max_tokens": null, "elapsed_s": null, "tok_s": null, "tok_s_mean": null, "prefill_tok_s_mean": null, "tok_s_window_s": null}, "requests": [], "conversation_cache": {"enabled": false, "budget_mib": 0, "slots": 4, "parked": 0, "bytes": 0, "evictions": 0, "parks": 0, "restores": 0, "last_event": null, "last_tokens": null, "last_at": null, "requests": 0, "requests_reused": 0, "reused_tokens": 0, "prompt_tokens": 0, "last_reused": null, "last_prompt": null}, "requests_kept": 0, "totals": {"since": 1791784504.3221655, "requests": 0, "prompt_tokens": 0, "reused": 0, "output_tokens": 0, "prompt_ms": 0.0, "decode_ms": 0.0, "drafts_offered": 0, "drafts_accepted": 0}, "hardware": {"gpus": [{"index": 0, "util": 0, "mem_used": 13786742784, "mem_total": 17103323136, "temp": 36, "power": 4.435}, {"index": 1, "util": 0, "mem_used": 12429361152, "mem_total": 12884901888, "temp": 42, "power": 7.645}], "gpu_util": 0.0, "gpu_mem_used": 26216103936, "gpu_mem_total": 29988225024, "gpu_temp": 42, "gpu_power": 12.079999999999998, "gpu_power_limit": 350.0, "gpu_pcie_gen": 1, "gpu_pcie_gen_max": 5, "gpu_pcie_width": 8, "gpu_pcie_rx_mb": 0.98828125, "gpu_pcie_tx_mb": 1.373046875, "cpu": 0.1, "ram_used": 66033725440, "ram_total": 66472062976, "disk_read_mb": 0.0, "disk_write_mb": 0.0, "tok_s": 0.0, "tok_s_mean": 0, "prefill_tok_s_mean": 0.0}, "hardware_static": {"gpu_name": "NVIDIA GeForce RTX 5060 Ti + NVIDIA GeForce RTX 3060", "gpu_count": 2, "gpu_note": null, "cpu_name": "AMD Ryzen 9 7900X 12-Core Processor", "cores": 12, "threads": 24, "psutil": true}, "history": {"gpu_util": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "gpu_mem_used": [26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936, 26216103936], "gpu_temp": [43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 42, 43, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42], "gpu_power": [12.3, 12.21, 12.09, 12.13, 12.04, 12.0, 12.08, 12.18, 12.18, 12.09, 11.96, 11.91, 11.99, 12.05, 11.93, 12.0, 12.02, 12.0, 12.03, 11.97, 11.98, 12.04, 12.0, 11.98, 11.86, 11.96, 12.01, 12.01, 12.08, 12.14, 12.2, 12.26, 12.24, 12.26, 12.23, 12.15, 12.11, 12.12, 12.09, 12.09, 12.04, 12.15, 12.16, 12.12, 12.05, 12.04, 12.03, 12.07, 12.04, 12.06, 12.1, 12.11, 12.11, 12.08, 12.01, 12.01, 12.01, 12.05, 12.07, 12.08], "gpu_pcie_rx_mb": [1.01, 1.0, 1.23, 0.97, 0.97, 0.99, 0.98, 0.94, 1.0, 0.98, 0.92, 1.01, 0.98, 0.98, 0.98, 0.95, 0.92, 0.98, 0.88, 1.03, 0.93, 0.99, 0.99, 1.0, 1.0, 0.96, 0.99, 0.98, 0.98, 0.93, 1.0, 0.98, 0.99, 0.94, 0.96, 0.93, 1.02, 0.93, 0.95, 0.94, 0.98, 1.0, 0.94, 0.98, 1.03, 1.03, 1.1, 0.98, 0.98, 0.99, 0.99, 0.94, 0.94, 0.98, 0.94, 0.94, 0.99, 0.94, 1.0, 0.99], "cpu": [0.0, 0.0, 0.1, 0.1, 0.1, 0.0, 0.1, 0.0, 0.0, 0.0, 0.2, 0.0, 0.0, 0.0, 0.1, 0.1, 0.0, 0.1, 0.0, 0.1, 0.0, 0.1, 0.0, 0.2, 0.0, 0.1, 0.1, 0.0, 0.1, 0.1, 0.1, 0.0, 0.0, 0.0, 0.2, 0.0, 0.1, 0.1, 0.0, 0.1, 0.0, 0.1, 0.0, 0.1, 0.0, 0.1, 0.1, 0.0, 0.0, 0.0, 0.1, 0.1, 0.0, 0.1, 0.1, 0.1, 0.0, 0.0, 0.1, 0.1], "ram_used": [66031726592, 66031828992, 66031828992, 66031828992, 66031828992, 66031828992, 66031828992, 66031828992, 66032926720, 66032926720, 66032926720, 66032926720, 66032926720, 66032926720, 66032926720, 66032926720, 66032926720, 66032984064, 66032963584, 66032963584, 66032963584, 66032963584, 66032963584, 66032963584, 66032963584, 66032963584, 66032963584, 66032963584, 66035081216, 66035081216, 66035081216, 66035081216, 66035081216, 66035081216, 66035081216, 66035081216, 66035015680, 66035015680, 66035015680, 66035015680, 66035015680, 66035429376, 66035789824, 66035789824, 66035789824, 66035789824, 66035789824, 66035789824, 66035789824, 66035789824, 66035789824, 66034135040, 66033725440, 66033725440, 66033725440, 66033725440, 66033725440, 66033725440, 66033725440, 66033725440], "disk_read_mb": [0.0, 0.03, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.04, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 12.69, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "tok_s": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "prefill_tok_s_mean": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}, "time": 1791784655.1063397} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/README.md b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/README.md new file mode 100644 index 00000000000..320b5c4ec21 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/README.md @@ -0,0 +1,157 @@ +# Community benchmark B1c: `--layer-split 6` — the engine's suggestion, followed literally, is catastrophic + +Measured on 2026-10-11 by @mmbeanz. Same box / pinned profile / suite as +`../A/` (K=40) and `../B1/` (auto -> K=3). + +**Headline.** `auto` flagged K=6 as a near-tie and printed *"To try the other: +`--layer-split 6`"*. We did exactly that, and it is **-68% vs B1 and -76% vs A on the +prompt path**. The reason is a trap in how the suggestion reads: **`--layer-split N` counts +layers for CUDA0, and an explicit split does NOT get the #1352 card reorder that `auto` +applies.** In `auto`'s frame CUDA0 was the slow RTX 3060 (3 layers); pasted literally, CUDA0 +is the fast RTX 5060 Ti, so `6` put **6 layers on the fast card and 42 on the slow one** — +the inverted, pathological assignment. The advice is correct in its own frame and +unusable in the copy-pasted one. + +## Evidence: which card is which (from the engine logs) + +| run | split arg | reorder? | CUDA0 | CUDA1 | layers on the slow card | +| --- | --- | --- | --- | --- | ---: | +| A | `40` (explicit) | no | **5060 Ti**, layers 0-39 | 3060, layers 40-47 | 8 | +| B1 | `auto` | **yes**, order 1,0 (#1352) | **3060**, layers 0-2 | 5060 Ti, layers 3-47 | 3 | +| B1c | `6` (explicit) | **no** | **5060 Ti**, layers 0-5 | 3060, layers 6-47 | **42** | + +Card identity is unambiguous in the logs (total VRAM and link probe): + +``` +B1 : layer split, CUDA1: 9.21 GiB free of 15.57 -> CUDA1 = 5060 Ti (16 GB) + layer split: CUDA1 PCIe probe 28.7 GB/s -> the gen5 link, i.e. the 5060 Ti + layer split: CUDA1 runs layers 3-47, expert cache 4399 slots +B1c: layer split, CUDA1: 5.68 GiB free of 11.68 -> CUDA1 = 3060 (12 GB) + layer split: CUDA1 PCIe probe 3.2 GB/s -> the gen3 x4 link, i.e. the 3060 + layer split: CUDA1 runs layers 6-47, expert cache 2539 slots + WARNING: prompt chunk 7424 ... CUDA1 (NVIDIA GeForce RTX 3060) has 2539 expert-cache slots +``` + +That last WARNING line names the card outright: in B1c the **3060** carries layers 6-47. + +## What the misassignment costs, per resource + +**1. Layer time.** The 3060 is 1.9x slower per layer (engine's own figures: 1.46 ms/layer +vs 0.78 ms/layer). B1c puts 42 of 48 layers on it: `42 x 1.46 + 6 x 0.78 ~ 66 ms` per +window, versus `40 x 0.78 + 8 x 1.46 ~ 42.9 ms` in A. Decode collapses accordingly. + +**2. Stranded VRAM on the fast card.** A stage can only cache **its own** layers' experts, +and 6 layers x 512 experts = 3,072 — so the 5060 Ti's cache is capped by layer count, not +memory: + +``` +B1c: expert cache auto: 13.28 GiB free, 700 MiB reserved -> 3072 slots (5.52 GiB used) +``` + +**~7 GiB of the 16 GB card's free VRAM cannot be used for experts at all**, while the 3060 +gets only 2,539 slots (4.90 GiB) for its 42 layers (21,504 pairs = 11.8%). + +**3. Hit rate.** 0.669 median (min 0.309) vs 0.859 (A) / 0.865 (B1). + +**4. Prompt path — and a correction to the B1 story.** B1c's chunk was **7,424** (only +lightly clamped), yet prompt throughput was 407.9 tok/s. So the prompt collapse here is +**not** the chunk clamp: it is prefill spending most of its layer work on the slow card. +B1's -25% prompt loss (chunk 3,328, fast card doing 45 layers) *is* the chunk clamp. The +two effects are separable, and B1c isolates the layer-assignment one. + +## Results + +| Configuration | Prompt tok/s (median) | Decode tok/s (median) | TTFT s (median) | +| --- | ---: | ---: | ---: | +| short chat (9 runs) | n/a | 41.4 (35.6-49.9) | 0.72 (0.60-0.78) | +| 4K prompt, fresh | 264.0 (240.6-264.7) | 38.7* | 15.26 (15.21-16.73) | +| 16K prompt, fresh | 346.2 (338.9-346.8) | ~50* | 46.42 (46.34-47.43) | +| 32K prompt, fresh | 407.9 (406.8-408.3) | 32.3* | 78.61 (78.54-78.82) | +| agent follow-up turns (15) | see `results.json` | 38.6 (27.1-52.2) | 15.38 (14.51-18.30) | + +\* engine per-request `decode_tok_s` from `/metrics?requests=all` (24-token cap). + +### Three-way comparison + +| metric | A (K=40, fast card 40 layers) | B1 (auto K=3, fast card 45) | B1c (fast card 6, slow card 42) | +| --- | ---: | ---: | ---: | +| prompt 32K tok/s | **1,720.8** | 1,290.9 | **407.9** | +| prompt 4K tok/s | 868.1 | 835.6 | 264.0 | +| TTFT 32K (s) | 18.66 | 24.86 | 78.61 | +| short-chat decode | 74.4 | 66.6 | 41.4 | +| decode 32K | 53.2 | 54.9 | 32.3 | +| expert hit_rate (median / min) | 0.859 / 0.550 | 0.865 / 0.543 | **0.669 / 0.309** | +| held experts (total) | 8,842 | 5,935 | 5,611 | +| slots on the card doing most layers | 4,984 (40 layers) | 4,399 (45 layers) | 2,539 (42 layers) | +| prompt chunk (tokens) | 8,192 | 3,328 | 7,424 | +| `pcie_share` median / max | 0 / 0 | 0.08 / 0.247 | 0.004 / 0.019 | + +## Coverage: what actually tracks hit rate + +| run | held total | f | old `(r+1)^-1.2` | new `H(f)` b=3 | **measured hit_rate** | slots per layer on the busy card | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| A | 8,842 | 0.360 | 97.0% | 73.8% | **85.9%** | 124.6 | +| B1 | 5,935 | 0.241 | 95.6% | 56.4% | **86.5%** | 97.8 | +| B1c | 5,611 | 0.228 | 95.4% | 54.0% | **66.9%** | 60.5 | + +B1 and B1c hold nearly the same **total** number of experts (5,935 vs 5,611) and differ by +**20 points of hit rate**. A count-based coverage curve cannot express that: both curves +predict 95.6/95.4% (old) and 56.4/54.0% (new) for the two runs, i.e. essentially no +difference. What does track the measurement is **cache per layer on the card that runs most +of the layers** (124.6 -> 84%, 97.8 -> 86%, 60.5 -> 62%). Decomposing per stage under the +assumption that each layer gets a similar share of lookups: + +| run | card doing most layers | its slots | implied coverage of its own routed mass | +| --- | --- | ---: | ---: | +| A | 5060 Ti, 40 layers (83% of lookups) | 4,984 | ~84% | +| B1 | 5060 Ti, 45 layers (94% of lookups) | 4,399 | ~86% | +| B1c | 3060, 42 layers (88% of lookups) | 2,539 | ~62% | + +This qualifies the 0.1.43 re-fit comment *"hit rate follows the held COUNT and almost +nothing else"*: on this pair the total count is the one thing hit rate does **not** follow. +Placement across stages is what matters, because a stage can only cache its own layers. + +> **Update from `../B1d/` (K=42):** the "slots per layer on the busy card" heuristic above +> (124.6 -> 84%, 97.8 -> 86%, 60.5 -> 62%) is **not monotonic** once a fourth point is added +> (K=42: 116.0 slots/layer but hit 0.818). The model that does hold is **fraction of each +> layer's own pairs that the owning card can cache** — see `../B1d/README.md`, which also +> adds permutation significance tests (A vs B1: no difference in hit rate despite a 33% +> difference in held experts). + +Caveats: `hit_rate` is the engine's **decode-phase** expert-cache hit rate (the `/metrics` +per-request values match the log's `decode expert cache hit rate` lines exactly: min 55.0% / +max 96.9% in A); prefill routing is not covered. The per-stage implied coverage assumes +roughly uniform lookups per layer. + +## Tool / docs findings from this run + +1. **The `--layer-split N` suggestion from `auto` is not copy-pasteable.** `auto` reorders + cards so the faster one is last (#1352); an explicit `--layer-split` does not reorder, so + `N` counts layers for a *different* card than the search had in mind. Following the + engine's own advice produced a 3-4x slowdown. Suggested fix: print the suggestion in the + as-given frame (here `--layer-split 42`), or state the card order next to it. +2. A stage's expert cache is capped by `its own layer count x 512`, which can strand large + amounts of VRAM on a card given few layers (7 GiB stranded on a 16 GB card here). Worth a + log line, since the current one (`13.28 GiB free ... -> 3072 slots`) reads as if memory + were the limit. +3. Previously reported (see `../A/README.md`): `gpu_slug()` names multi-GPU folders after + the first card only; `collect_machine()` blends client/server fields under `--url`; + `/metrics` defaults to 12 requests (`?requests=all` for the rest); the generated README + cites `engine-log-excerpt.txt` in `--url` mode where none is written. + +## Correct next test: B1d = `--layer-split 42` + +To test the engine's intended "K=6 on the slow card" in the as-given card order, the +equivalent explicit split here is **`--layer-split 42`** (5060 Ti: layers 0-41; 3060: +layers 42-47, i.e. 6 layers, 3,072 slots, ~4.0 GiB, and the 5060 Ti keeps ~4,000 slots for +42 layers). Predicted window ~41.6 ms vs A's ~42.9 ms, with more of the 3060's VRAM used and +the fast card's cache roughly preserved. If that recovers A's prompt speed and beats its +decode, it is a concrete improvement to report; if not, K=40 stands as the tuned answer. + +## Limitations + +- Speed only; no needle/quality check (needle suite: `../needles/`). +- One suite per config (3 runs), same evening. +- A vs B1 vs B1c differ in split policy, card order and chunk size simultaneously; B1d is + the clean single-variable test of the K=6 idea. +- No concurrency, no vision latency, no mmap/low-RAM tier, Windows/HIP untested. diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/config.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/config.json new file mode 100644 index 00000000000..1a60474093d --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/config.json @@ -0,0 +1,44 @@ +{ + "exe": "/home/llm/Strata/engine/strata", + "args": [ + "--pack", + "/home/llm/Strata-data/packs/iq3_s", + "--native", + "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf", + "--ple-gguf", + "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf", + "--expert-profile", + "/home/llm/Strata/data/expert-profile.bin", + "--expert-cache", + "auto", + "--prefill", + "auto", + "--spec", + "4", + "--mtp", + "/home/llm/Strata-data/mtp/rt", + "--max-context", + "524288", + "--rope-scaling", + "yarn", + "--rope-scale", + "2.0", + "--kv", + "int8", + "--kv-resident", + "20480", + "--vision", + "--remote-expert-opt", + "--trim-stage-weights", + "--spec-min-p", + "0.70" + ], + "model_name": "qwen3.8-flash-next-iq3_s", + "port": 8080, + "gpu": [ + 0, + 1 + ], + "layer_split": "6", + "draft_vocab": "en" +} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/results.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/results.json new file mode 100644 index 00000000000..c41940d9a60 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1c/results.json @@ -0,0 +1,999 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-11", + "params": { + "runs": 3, + "think": false, + "clients": 0, + "long": true, + "started_server": false, + "command": "python tools/strata_bench.py --long" + }, + "elapsed_s": 771.6396803855896, + "machine": { + "gpu": [ + "NVIDIA GeForce RTX 5060 Ti", + "NVIDIA GeForce RTX 3060" + ], + "gpu_vram_mib": 28599.0, + "gpu_driver": "615.71.09", + "cuda": "13.4 (nvcc V13.4.92)", + "cpu": "AMD Ryzen 9 7900X 12-Core Processor", + "cpu_cores": 12, + "cpu_threads": 24, + "ram_gib": 61.9, + "os": "Linux 7.2.9-1-cachyos", + "python": "3.14.7", + "strata_version": "0.1.43", + "strata_commit": "7695859a", + "engine_version": "0.1.43", + "model": "qwen3.8-flash-next-iq3_s", + "quant": "IQ3_S", + "context": 524288, + "parallel_slots": 1, + "config": { + "exe": "strata", + "args": [ + "--pack", + "iq3_s", + "--native", + "Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf", + "--ple-gguf", + "Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf", + "--expert-profile", + "expert-profile.bin", + "--expert-cache", + "auto", + "--prefill", + "auto", + "--spec", + "4", + "--mtp", + "rt", + "--max-context", + "524288", + "--rope-scaling", + "yarn", + "--rope-scale", + "2.0", + "--kv", + "int8", + "--kv-resident", + "20480", + "--vision", + "--remote-expert-opt", + "--trim-stage-weights", + "--spec-min-p", + "0.70" + ], + "model_name": "qwen3.8-flash-next-iq3_s", + "port": 8080, + "gpu": [ + 0, + 1 + ], + "layer_split": "6", + "draft_vocab": "en" + }, + "launch": "strata --pack iq3_s --native Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf --ple-gguf Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf --expert-profile expert-profile.bin --expert-cache auto --prefill auto --spec 4 --mtp rt --max-context 524288 --rope-scaling yarn --rope-scale 2.0 --kv int8 --kv-resident 20480 --vision --remote-expert-opt --trim-stage-weights --spec-min-p 0.70", + "engine_info": { + "model": "qwen3.8-flash-next-iq3_s", + "max_context": 524288, + "images": true, + "version": "0.1.43", + "logit_bias": 1, + "context": 524288, + "kv": "int8", + "kv_resident": 20480, + "expert_slots": 5611, + "expert_cache_mib": 10665, + "expert_slots_primary": 3072, + "expert_cache_primary_mib": 5650, + "spec": 6, + "mtp_max": 4, + "lookup": 3, + "vram_free_mib": 3162, + "cvec": 0, + "arena_mib": 47962, + "pool_workers": 11, + "pcie_frac": "0.55", + "spec_min_p": "0.70", + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "conversation_cache_min_tokens": 0, + "tail_role_token": -1, + "vram_elastic": 0, + "engine": "0.1.43" + }, + "vram_after": { + "elastic": false + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.777, + "total_s": 2.873, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 68.1, + "prompt_tps_source": "engine", + "decode_tps": 47.2, + "decode_tps_source": "engine", + "prompt_ms": 749.2, + "decode_ms": 2117.5, + "drafts_offered": 86, + "drafts_accepted": 70, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.709, + "total_s": 3.485, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 63.2, + "prompt_tps_source": "engine", + "decode_tps": 35.7, + "decode_tps_source": "engine", + "prompt_ms": 680.4, + "decode_ms": 2797.8, + "drafts_offered": 80, + "drafts_accepted": 50, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.726, + "total_s": 3.425, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 65.9, + "prompt_tps_source": "engine", + "decode_tps": 36.8, + "decode_tps_source": "engine", + "prompt_ms": 697.8, + "decode_ms": 2720.8, + "drafts_offered": 79, + "drafts_accepted": 54, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.722, + "total_s": 2.753, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 73.5, + "prompt_tps_source": "engine", + "decode_tps": 48.7, + "decode_tps_source": "engine", + "prompt_ms": 693.6, + "decode_ms": 2053.2, + "drafts_offered": 79, + "drafts_accepted": 70, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.602, + "total_s": 3.274, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 75.0, + "prompt_tps_source": "engine", + "decode_tps": 37.1, + "decode_tps_source": "engine", + "prompt_ms": 573.2, + "decode_ms": 2694.1, + "drafts_offered": 76, + "drafts_accepted": 50, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.727, + "total_s": 3.122, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 65.9, + "prompt_tps_source": "engine", + "decode_tps": 41.4, + "decode_tps_source": "engine", + "prompt_ms": 698.0, + "decode_ms": 2416.8, + "drafts_offered": 80, + "drafts_accepted": 54, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.725, + "total_s": 2.708, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 73.2, + "prompt_tps_source": "engine", + "decode_tps": 49.9, + "decode_tps_source": "engine", + "prompt_ms": 696.7, + "decode_ms": 2004.8, + "drafts_offered": 75, + "drafts_accepted": 70, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.628, + "total_s": 3.414, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 71.8, + "prompt_tps_source": "engine", + "decode_tps": 35.6, + "decode_tps_source": "engine", + "prompt_ms": 599.3, + "decode_ms": 2808.5, + "drafts_offered": 71, + "drafts_accepted": 45, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.737, + "total_s": 3.122, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 65.0, + "prompt_tps_source": "engine", + "decode_tps": 41.5, + "decode_tps_source": "engine", + "prompt_ms": 708.2, + "decode_ms": 2407.9, + "drafts_offered": 81, + "drafts_accepted": 53, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 41.4, + "min": 35.6, + "max": 49.9, + "n": 9 + }, + "ttft_s": { + "median": 0.725, + "min": 0.602, + "max": 0.777, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 46, + "min": 43, + "max": 51, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 16.734, + "total_s": 17.441, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 240.6, + "prompt_tps_source": "engine", + "decode_tps": 32.8, + "decode_tps_source": "engine", + "prompt_ms": 16702.4, + "decode_ms": 730.8, + "drafts_offered": 12, + "drafts_accepted": 10, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 15.213, + "total_s": 15.81, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 264.7, + "prompt_tps_source": "engine", + "decode_tps": 38.7, + "decode_tps_source": "engine", + "prompt_ms": 15181.2, + "decode_ms": 620.2, + "drafts_offered": 13, + "drafts_accepted": 10, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 15.255, + "total_s": 15.81, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 264.0, + "prompt_tps_source": "engine", + "decode_tps": 41.5, + "decode_tps_source": "engine", + "prompt_ms": 15224.5, + "decode_ms": 578.1, + "drafts_offered": 13, + "drafts_accepted": 11, + "run": 2 + } + ], + "prompt_tps": { + "median": 264.0, + "min": 240.6, + "max": 264.7, + "n": 3 + }, + "ttft_s": { + "median": 15.255, + "min": 15.213, + "max": 16.734, + "n": 3 + }, + "prompt_tokens": { + "median": 4019, + "min": 4019, + "max": 4019, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 47.426, + "total_s": 48.088, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 338.9, + "prompt_tps_source": "engine", + "decode_tps": 35.0, + "decode_tps_source": "engine", + "prompt_ms": 47389.5, + "decode_ms": 685.0, + "drafts_offered": 14, + "drafts_accepted": 12, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 46.345, + "total_s": 47.217, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 346.8, + "prompt_tps_source": "engine", + "decode_tps": 26.8, + "decode_tps_source": "engine", + "prompt_ms": 46309.1, + "decode_ms": 895.0, + "drafts_offered": 22, + "drafts_accepted": 10, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 46.424, + "total_s": 47.068, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 346.2, + "prompt_tps_source": "engine", + "decode_tps": 36.0, + "decode_tps_source": "engine", + "prompt_ms": 46387.7, + "decode_ms": 666.1, + "drafts_offered": 17, + "drafts_accepted": 10, + "run": 2 + } + ], + "prompt_tps": { + "median": 346.2, + "min": 338.9, + "max": 346.8, + "n": 3 + }, + "ttft_s": { + "median": 46.424, + "min": 46.345, + "max": 47.426, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "32K": { + "target_tokens": 32768, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 78.823, + "total_s": 79.584, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 406.8, + "prompt_tps_source": "engine", + "decode_tps": 30.6, + "decode_tps_source": "engine", + "prompt_ms": 78778.7, + "decode_ms": 785.2, + "drafts_offered": 17, + "drafts_accepted": 9, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 78.544, + "total_s": 79.265, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 408.3, + "prompt_tps_source": "engine", + "decode_tps": 32.3, + "decode_tps_source": "engine", + "prompt_ms": 78499.3, + "decode_ms": 744.1, + "drafts_offered": 16, + "drafts_accepted": 8, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 78.614, + "total_s": 79.331, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 407.9, + "prompt_tps_source": "engine", + "decode_tps": 32.4, + "decode_tps_source": "engine", + "prompt_ms": 78570.6, + "decode_ms": 739.9, + "drafts_offered": 21, + "drafts_accepted": 10, + "run": 2 + } + ], + "prompt_tps": { + "median": 407.9, + "min": 406.8, + "max": 408.3, + "n": 3 + }, + "ttft_s": { + "median": 78.614, + "min": 78.544, + "max": 78.823, + "n": 3 + }, + "prompt_tokens": { + "median": 32049, + "min": 32049, + "max": 32049, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 11.606, + "total_s": 12.217, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 18, + "prompt_tps": 52.6, + "prompt_tps_source": "engine", + "decode_tps": 28.4, + "decode_tps_source": "engine", + "prompt_ms": 11572.1, + "decode_ms": 633.0, + "drafts_offered": 22, + "drafts_accepted": 12, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 18.303, + "total_s": 19.878, + "prompt_tokens": 3709, + "reused_tokens": 629, + "read_tokens": 3080, + "generated_tokens": 64, + "prompt_tps": 168.7, + "prompt_tps_source": "engine", + "decode_tps": 40.1, + "decode_tps_source": "engine", + "prompt_ms": 18260.8, + "decode_ms": 1597.6, + "drafts_offered": 47, + "drafts_accepted": 40, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 15.971, + "total_s": 17.698, + "prompt_tokens": 5242, + "reused_tokens": 3772, + "read_tokens": 1470, + "generated_tokens": 64, + "prompt_tps": 92.3, + "prompt_tps_source": "engine", + "decode_tps": 36.6, + "decode_tps_source": "engine", + "prompt_ms": 15934.1, + "decode_ms": 1748.4, + "drafts_offered": 52, + "drafts_accepted": 37, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 17.889, + "total_s": 19.238, + "prompt_tokens": 8686, + "reused_tokens": 5306, + "read_tokens": 3380, + "generated_tokens": 49, + "prompt_tps": 189.3, + "prompt_tps_source": "engine", + "decode_tps": 35.7, + "decode_tps_source": "engine", + "prompt_ms": 17852.7, + "decode_ms": 1371.0, + "drafts_offered": 44, + "drafts_accepted": 26, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 17.494, + "total_s": 17.83, + "prompt_tokens": 11594, + "reused_tokens": 8734, + "read_tokens": 2860, + "generated_tokens": 15, + "prompt_tps": 163.9, + "prompt_tps_source": "engine", + "decode_tps": 41.8, + "decode_tps_source": "engine", + "prompt_ms": 17451.8, + "decode_ms": 358.7, + "drafts_offered": 14, + "drafts_accepted": 9, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 16.864, + "total_s": 17.823, + "prompt_tokens": 14454, + "reused_tokens": 11608, + "read_tokens": 2846, + "generated_tokens": 32, + "prompt_tps": 169.1, + "prompt_tps_source": "engine", + "decode_tps": 32.6, + "decode_tps_source": "engine", + "prompt_ms": 16827.3, + "decode_ms": 982.0, + "drafts_offered": 27, + "drafts_accepted": 14, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 10.921, + "total_s": 11.44, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 19, + "prompt_tps": 55.9, + "prompt_tps_source": "engine", + "decode_tps": 35.1, + "decode_tps_source": "engine", + "prompt_ms": 10891.7, + "decode_ms": 541.2, + "drafts_offered": 20, + "drafts_accepted": 13, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 15.362, + "total_s": 16.597, + "prompt_tokens": 3710, + "reused_tokens": 630, + "read_tokens": 3080, + "generated_tokens": 64, + "prompt_tps": 200.9, + "prompt_tps_source": "engine", + "decode_tps": 50.9, + "decode_tps_source": "engine", + "prompt_ms": 15331.6, + "decode_ms": 1257.6, + "drafts_offered": 49, + "drafts_accepted": 44, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 14.51, + "total_s": 16.159, + "prompt_tokens": 5243, + "reused_tokens": 3773, + "read_tokens": 1470, + "generated_tokens": 64, + "prompt_tps": 101.5, + "prompt_tps_source": "engine", + "decode_tps": 38.3, + "decode_tps_source": "engine", + "prompt_ms": 14478.1, + "decode_ms": 1670.1, + "drafts_offered": 61, + "drafts_accepted": 39, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 15.602, + "total_s": 16.797, + "prompt_tokens": 8687, + "reused_tokens": 5236, + "read_tokens": 3451, + "generated_tokens": 47, + "prompt_tps": 221.7, + "prompt_tps_source": "engine", + "decode_tps": 38.6, + "decode_tps_source": "engine", + "prompt_ms": 15568.5, + "decode_ms": 1217.7, + "drafts_offered": 39, + "drafts_accepted": 25, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 15.33, + "total_s": 15.594, + "prompt_tokens": 11593, + "reused_tokens": 8733, + "read_tokens": 2860, + "generated_tokens": 15, + "prompt_tps": 187.0, + "prompt_tps_source": "engine", + "decode_tps": 52.2, + "decode_tps_source": "engine", + "prompt_ms": 15293.0, + "decode_ms": 287.6, + "drafts_offered": 12, + "drafts_accepted": 9, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 15.357, + "total_s": 15.777, + "prompt_tokens": 14453, + "reused_tokens": 11607, + "read_tokens": 2846, + "generated_tokens": 12, + "prompt_tps": 185.8, + "prompt_tps_source": "engine", + "decode_tps": 27.1, + "decode_tps_source": "engine", + "prompt_ms": 15318.0, + "decode_ms": 442.6, + "drafts_offered": 10, + "drafts_accepted": 3, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 10.972, + "total_s": 11.291, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 12, + "prompt_tps": 55.6, + "prompt_tps_source": "engine", + "decode_tps": 35.2, + "decode_tps_source": "engine", + "prompt_ms": 10943.6, + "decode_ms": 340.5, + "drafts_offered": 14, + "drafts_accepted": 9, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 15.349, + "total_s": 16.771, + "prompt_tokens": 3703, + "reused_tokens": 623, + "read_tokens": 3080, + "generated_tokens": 64, + "prompt_tps": 201.1, + "prompt_tps_source": "engine", + "decode_tps": 44.3, + "decode_tps_source": "engine", + "prompt_ms": 15318.1, + "decode_ms": 1444.3, + "drafts_offered": 49, + "drafts_accepted": 39, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 14.6, + "total_s": 16.046, + "prompt_tokens": 5236, + "reused_tokens": 3767, + "read_tokens": 1469, + "generated_tokens": 64, + "prompt_tps": 100.8, + "prompt_tps_source": "engine", + "decode_tps": 43.6, + "decode_tps_source": "engine", + "prompt_ms": 14569.0, + "decode_ms": 1467.6, + "drafts_offered": 51, + "drafts_accepted": 41, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 15.645, + "total_s": 17.391, + "prompt_tokens": 8680, + "reused_tokens": 5229, + "read_tokens": 3451, + "generated_tokens": 64, + "prompt_tps": 221.1, + "prompt_tps_source": "engine", + "decode_tps": 36.2, + "decode_tps_source": "engine", + "prompt_ms": 15610.6, + "decode_ms": 1768.7, + "drafts_offered": 55, + "drafts_accepted": 32, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 15.383, + "total_s": 15.818, + "prompt_tokens": 11604, + "reused_tokens": 8744, + "read_tokens": 2860, + "generated_tokens": 20, + "prompt_tps": 186.3, + "prompt_tps_source": "engine", + "decode_tps": 43.6, + "decode_tps_source": "engine", + "prompt_ms": 15348.0, + "decode_ms": 458.7, + "drafts_offered": 18, + "drafts_accepted": 12, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 15.349, + "total_s": 16.78, + "prompt_tokens": 14469, + "reused_tokens": 11623, + "read_tokens": 2846, + "generated_tokens": 49, + "prompt_tps": 185.9, + "prompt_tps_source": "engine", + "decode_tps": 33.7, + "decode_tps_source": "engine", + "prompt_ms": 15312.2, + "decode_ms": 1454.5, + "drafts_offered": 49, + "drafts_accepted": 24, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 68.63, + "min": 68.624, + "max": 68.786, + "n": 3 + }, + "reread_tokens": { + "median": 72, + "min": 3, + "max": 74, + "n": 3 + }, + "new_tokens": { + "median": 13633, + "min": 13633, + "max": 13634, + "n": 3 + }, + "turn_ttft_s": { + "median": 15.383, + "min": 14.51, + "max": 18.303, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 14454, + "min": 14453, + "max": 14469, + "n": 3 + }, + "turn_decode_tps": { + "median": 38.6, + "min": 27.1, + "max": 52.2, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 10.972, + "min": 10.921, + "max": 11.606, + "n": 3 + } + }, + "clients": {}, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/B1d-after-full.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/B1d-after-full.json new file mode 100644 index 00000000000..f71772aa6be --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/B1d-after-full.json @@ -0,0 +1 @@ +{"engine": {"model": "qwen3.8-flash-next-iq3_s", "max_context": 524288, "images": true, "version": "0.1.43", "logit_bias": 1, "context": 524288, "kv": "int8", "kv_resident": 20480, "expert_slots": 7946, "expert_cache_mib": 16078, "expert_slots_primary": 4874, "expert_cache_primary_mib": 9316, "spec": 6, "mtp_max": 4, "lookup": 3, "vram_free_mib": 324, "cvec": 0, "arena_mib": 47962, "pool_workers": 11, "pcie_frac": "0.00", "spec_min_p": "0.70", "conversation_cache_mib": 0, "conversation_cache_slots": 4, "conversation_cache_min_free_mib": 2560, "conversation_cache_min_tokens": 0, "tail_role_token": -1, "vram_elastic": 0, "engine": "0.1.43"}, "live": {"state": "idle", "queued": 0, "phase": null, "prompt_tokens": null, "prompt_read": null, "prompt_total": null, "generated": null, "max_tokens": null, "elapsed_s": null, "tok_s": null, "tok_s_mean": null, "prefill_tok_s_mean": null, "tok_s_window_s": null}, "requests": [{"projection": null, "time": 1791786815.341009, "duration_s": 19.4, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 19037.9, "decode_ms": 403.8, "decode_tok_s": 59.4, "hit_rate": 0.924, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 19, "drafts_accepted": 10}, {"projection": null, "time": 1791786795.8590991, "duration_s": 19.5, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 19049.2, "decode_ms": 407.5, "decode_tok_s": 58.9, "hit_rate": 0.846, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 19, "drafts_accepted": 10}, {"projection": null, "time": 1791786776.4215596, "duration_s": 19.4, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 19024.9, "decode_ms": 387.3, "decode_tok_s": 62.0, "hit_rate": 0.679, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 13, "drafts_accepted": 11}, {"projection": null, "time": 1791786771.8809876, "duration_s": 4.5, "finish": "stop", "prompt_tokens": 14455, "reused": 11609, "output_tokens": 24, "prompt_total": 14455, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 4117.0, "decode_ms": 400.8, "decode_tok_s": 59.9, "hit_rate": 0.908, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 24, "drafts_accepted": 9}, {"projection": null, "time": 1791786767.5233965, "duration_s": 4.3, "finish": "stop", "prompt_tokens": 11590, "reused": 8730, "output_tokens": 20, "prompt_total": 11590, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 20, "prompt_ms": 4113.3, "decode_ms": 231.0, "decode_tok_s": 86.6, "hit_rate": 0.925, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 18, "drafts_accepted": 12}, {"projection": null, "time": 1791786762.60912, "duration_s": 4.9, "finish": "stop", "prompt_tokens": 8692, "reused": 5312, "output_tokens": 39, "prompt_total": 8692, "prompt_read": 3380, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 39, "prompt_ms": 4450.9, "decode_ms": 451.5, "decode_tok_s": 86.4, "hit_rate": 0.868, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 30, "drafts_accepted": 23}, {"projection": null, "time": 1791786758.6010194, "duration_s": 4.0, "finish": "length", "prompt_tokens": 5248, "reused": 3778, "output_tokens": 64, "prompt_total": 5248, "prompt_read": 1470, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 3199.3, "decode_ms": 798.3, "decode_tok_s": 80.2, "hit_rate": 0.853, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 56, "drafts_accepted": 37}, {"projection": null, "time": 1791786753.7350805, "duration_s": 4.9, "finish": "length", "prompt_tokens": 3715, "reused": 633, "output_tokens": 64, "prompt_total": 3715, "prompt_read": 3082, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 4211.5, "decode_ms": 645.7, "decode_tok_s": 99.1, "hit_rate": 0.866, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 49, "drafts_accepted": 42}, {"projection": null, "time": 1791786751.4604647, "duration_s": 2.3, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 24, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 1967.3, "decode_ms": 299.7, "decode_tok_s": 80.1, "hit_rate": 0.753, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 24, "drafts_accepted": 15}, {"projection": null, "time": 1791786747.104432, "duration_s": 4.3, "finish": "stop", "prompt_tokens": 14421, "reused": 11575, "output_tokens": 12, "prompt_total": 14421, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 12, "prompt_ms": 4102.1, "decode_ms": 242.4, "decode_tok_s": 49.5, "hit_rate": 0.883, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 14, "drafts_accepted": 3}, {"projection": null, "time": 1791786742.9023848, "duration_s": 4.2, "finish": "stop", "prompt_tokens": 11570, "reused": 8710, "output_tokens": 6, "prompt_total": 11570, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 6, "prompt_ms": 4108.3, "decode_ms": 81.5, "decode_tok_s": 73.6, "hit_rate": 0.921, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 6, "drafts_accepted": 3}, {"projection": null, "time": 1791786738.0931792, "duration_s": 4.8, "finish": "stop", "prompt_tokens": 8692, "reused": 5241, "output_tokens": 19, "prompt_total": 8692, "prompt_read": 3451, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 19, "prompt_ms": 4501.5, "decode_ms": 296.5, "decode_tok_s": 64.1, "hit_rate": 0.886, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 18, "drafts_accepted": 8}, {"projection": null, "time": 1791786734.1077127, "duration_s": 4.0, "finish": "length", "prompt_tokens": 5248, "reused": 3778, "output_tokens": 64, "prompt_total": 5248, "prompt_read": 1470, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 3178.6, "decode_ms": 796.7, "decode_tok_s": 80.3, "hit_rate": 0.843, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 49, "drafts_accepted": 38}, {"projection": null, "time": 1791786729.2536955, "duration_s": 4.8, "finish": "length", "prompt_tokens": 3715, "reused": 635, "output_tokens": 64, "prompt_total": 3715, "prompt_read": 3080, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 4213.3, "decode_ms": 631.9, "decode_tok_s": 101.3, "hit_rate": 0.861, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 45, "drafts_accepted": 42}, {"projection": null, "time": 1791786726.9514961, "duration_s": 2.3, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 24, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 1970.1, "decode_ms": 324.4, "decode_tok_s": 74.0, "hit_rate": 0.771, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 25, "drafts_accepted": 16}, {"projection": null, "time": 1791786722.3553638, "duration_s": 4.6, "finish": "stop", "prompt_tokens": 14437, "reused": 11591, "output_tokens": 31, "prompt_total": 14437, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 31, "prompt_ms": 4124.4, "decode_ms": 460.8, "decode_tok_s": 67.3, "hit_rate": 0.859, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 25, "drafts_accepted": 14}, {"projection": null, "time": 1791786717.911397, "duration_s": 4.4, "finish": "stop", "prompt_tokens": 11571, "reused": 8711, "output_tokens": 21, "prompt_total": 11571, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 21, "prompt_ms": 4210.9, "decode_ms": 218.7, "decode_tok_s": 96.0, "hit_rate": 0.887, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 17, "drafts_accepted": 14}, {"projection": null, "time": 1791786713.0193892, "duration_s": 4.9, "finish": "stop", "prompt_tokens": 8689, "reused": 5309, "output_tokens": 23, "prompt_total": 8689, "prompt_read": 3380, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 23, "prompt_ms": 4540.5, "decode_ms": 336.8, "decode_tok_s": 68.3, "hit_rate": 0.83, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 21, "drafts_accepted": 11}, {"projection": null, "time": 1791786708.9677649, "duration_s": 4.0, "finish": "length", "prompt_tokens": 5245, "reused": 3705, "output_tokens": 64, "prompt_total": 5245, "prompt_read": 1540, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 3265.9, "decode_ms": 771.0, "decode_tok_s": 83.0, "hit_rate": 0.78, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 53, "drafts_accepted": 39}, {"projection": null, "time": 1791786703.8284214, "duration_s": 5.1, "finish": "length", "prompt_tokens": 3712, "reused": 631, "output_tokens": 64, "prompt_total": 3712, "prompt_read": 3081, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 4365.8, "decode_ms": 760.3, "decode_tok_s": 84.2, "hit_rate": 0.656, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 49, "drafts_accepted": 42}, {"projection": null, "time": 1791786701.5299911, "duration_s": 2.3, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 21, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 21, "prompt_ms": 1954.0, "decode_ms": 324.4, "decode_tok_s": 64.7, "hit_rate": 0.668, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 25, "drafts_accepted": 13}, {"projection": null, "time": 1791786689.598547, "duration_s": 11.9, "finish": "length", "prompt_tokens": 16058, "reused": 0, "output_tokens": 24, "prompt_total": 16058, "prompt_read": 16058, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 11503.4, "decode_ms": 411.8, "decode_tok_s": 58.3, "hit_rate": 0.957, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 21, "drafts_accepted": 10}, {"projection": null, "time": 1791786677.6831448, "duration_s": 11.9, "finish": "length", "prompt_tokens": 16058, "reused": 0, "output_tokens": 24, "prompt_total": 16058, "prompt_read": 16058, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 11492.4, "decode_ms": 406.4, "decode_tok_s": 59.1, "hit_rate": 0.924, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 24, "drafts_accepted": 12}, {"projection": null, "time": 1791786665.7828763, "duration_s": 11.9, "finish": "length", "prompt_tokens": 16058, "reused": 0, "output_tokens": 24, "prompt_total": 16058, "prompt_read": 16058, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 11466.8, "decode_ms": 417.8, "decode_tok_s": 57.4, "hit_rate": 0.874, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 21, "drafts_accepted": 10}, {"projection": null, "time": 1791786660.9214087, "duration_s": 4.8, "finish": "length", "prompt_tokens": 4019, "reused": 0, "output_tokens": 24, "prompt_total": 4019, "prompt_read": 4019, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 4509.8, "decode_ms": 337.3, "decode_tok_s": 71.2, "hit_rate": 0.888, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 13, "drafts_accepted": 10}, {"projection": null, "time": 1791786656.046748, "duration_s": 4.9, "finish": "length", "prompt_tokens": 4019, "reused": 0, "output_tokens": 24, "prompt_total": 4019, "prompt_read": 4019, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 4521.6, "decode_ms": 343.4, "decode_tok_s": 69.9, "hit_rate": 0.816, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 13, "drafts_accepted": 10}, {"projection": null, "time": 1791786651.1384912, "duration_s": 4.9, "finish": "length", "prompt_tokens": 4019, "reused": 0, "output_tokens": 24, "prompt_total": 4019, "prompt_read": 4019, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 4560.5, "decode_ms": 339.2, "decode_tok_s": 70.8, "hit_rate": 0.608, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 15, "drafts_accepted": 12}, {"projection": null, "time": 1791786649.4537249, "duration_s": 1.7, "finish": "length", "prompt_tokens": 46, "reused": 0, "output_tokens": 100, "prompt_total": 46, "prompt_read": 46, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 367.5, "decode_ms": 1309.2, "decode_tok_s": 76.4, "hit_rate": 0.785, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 72, "drafts_accepted": 51}, {"projection": null, "time": 1791786647.7643852, "duration_s": 1.7, "finish": "length", "prompt_tokens": 43, "reused": 0, "output_tokens": 100, "prompt_total": 43, "prompt_read": 43, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 339.0, "decode_ms": 1344.3, "decode_tok_s": 74.4, "hit_rate": 0.808, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 87, "drafts_accepted": 51}, {"projection": null, "time": 1791786646.2673142, "duration_s": 1.5, "finish": "length", "prompt_tokens": 51, "reused": 0, "output_tokens": 100, "prompt_total": 51, "prompt_read": 51, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 417.0, "decode_ms": 1074.1, "decode_tok_s": 93.1, "hit_rate": 0.606, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 75, "drafts_accepted": 70}, {"projection": null, "time": 1791786644.5758328, "duration_s": 1.7, "finish": "length", "prompt_tokens": 46, "reused": 0, "output_tokens": 100, "prompt_total": 46, "prompt_read": 46, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 373.5, "decode_ms": 1311.6, "decode_tok_s": 76.2, "hit_rate": 0.782, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 72, "drafts_accepted": 51}, {"projection": null, "time": 1791786642.849822, "duration_s": 1.7, "finish": "length", "prompt_tokens": 43, "reused": 0, "output_tokens": 100, "prompt_total": 43, "prompt_read": 43, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 344.5, "decode_ms": 1375.5, "decode_tok_s": 72.7, "hit_rate": 0.82, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 75, "drafts_accepted": 46}, {"projection": null, "time": 1791786641.3196201, "duration_s": 1.5, "finish": "length", "prompt_tokens": 51, "reused": 0, "output_tokens": 100, "prompt_total": 51, "prompt_read": 51, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 414.8, "decode_ms": 1109.2, "decode_tok_s": 90.2, "hit_rate": 0.626, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 84, "drafts_accepted": 70}, {"projection": null, "time": 1791786639.4923959, "duration_s": 1.8, "finish": "length", "prompt_tokens": 46, "reused": 0, "output_tokens": 100, "prompt_total": 46, "prompt_read": 46, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 395.3, "decode_ms": 1425.5, "decode_tok_s": 70.2, "hit_rate": 0.704, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 79, "drafts_accepted": 49}, {"projection": null, "time": 1791786637.7179136, "duration_s": 1.8, "finish": "length", "prompt_tokens": 43, "reused": 0, "output_tokens": 100, "prompt_total": 43, "prompt_read": 43, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 369.5, "decode_ms": 1398.0, "decode_tok_s": 71.5, "hit_rate": 0.731, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 77, "drafts_accepted": 48}, {"projection": null, "time": 1791786636.203871, "duration_s": 1.5, "finish": "length", "prompt_tokens": 51, "reused": 0, "output_tokens": 100, "prompt_total": 51, "prompt_read": 51, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 426.3, "decode_ms": 1080.8, "decode_tok_s": 92.5, "hit_rate": 0.637, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 82, "drafts_accepted": 70}, {"projection": null, "time": 1791786634.0822344, "duration_s": 2.1, "finish": "length", "prompt_tokens": 1028, "reused": 0, "output_tokens": 8, "prompt_total": 1028, "prompt_read": 1028, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 8, "prompt_ms": 1976.4, "decode_ms": 138.9, "decode_tok_s": 57.6, "hit_rate": 0.609, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 7, "drafts_accepted": 3}, {"projection": null, "time": 1791786632.1291761, "duration_s": 1.9, "finish": "length", "prompt_tokens": 906, "reused": 0, "output_tokens": 1, "prompt_total": 906, "prompt_read": 906, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 1, "prompt_ms": 1927.3, "decode_ms": 18.3, "decode_tok_s": 54.6, "hit_rate": 0.703, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 0, "drafts_accepted": 0}, {"projection": null, "time": 1791786630.788006, "duration_s": 1.3, "finish": "length", "prompt_tokens": 260, "reused": 0, "output_tokens": 1, "prompt_total": 260, "prompt_read": 260, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 1, "prompt_ms": 1315.1, "decode_ms": 18.7, "decode_tok_s": 53.5, "hit_rate": 0.702, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 0, "drafts_accepted": 0}, {"projection": null, "time": 1791786630.0465045, "duration_s": 0.7, "finish": "length", "prompt_tokens": 34, "reused": 0, "output_tokens": 24, "prompt_total": 34, "prompt_read": 34, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 385.9, "decode_ms": 345.9, "decode_tok_s": 69.4, "hit_rate": 0.427, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 18, "drafts_accepted": 17}], "conversation_cache": {"enabled": false, "budget_mib": 0, "slots": 4, "parked": 0, "bytes": 0, "evictions": 0, "parks": 0, "restores": 0, "last_event": null, "last_tokens": null, "last_at": null, "requests": 40, "requests_reused": 15, "reused_tokens": 89948, "prompt_tokens": 291853, "last_reused": 0, "last_prompt": 32049}, "requests_kept": 40, "totals": {"since": 1791786506.1800041, "requests": 40, "prompt_tokens": 291853, "reused": 89948, "output_tokens": 1798, "prompt_ms": 180813.30000000002, "decode_ms": 23476.9, "drafts_offered": 1434, "drafts_accepted": 1002}, "hardware": {"gpus": [{"index": 0, "util": 0, "mem_used": 16836001792, "mem_total": 17103323136, "temp": 51, "power": 26.191}, {"index": 1, "util": 100, "mem_used": 10562895872, "mem_total": 12884901888, "temp": 55, "power": 46.913}], "gpu_util": 50.0, "gpu_mem_used": 27398897664, "gpu_mem_total": 29988225024, "gpu_temp": 55, "gpu_power": 73.104, "gpu_power_limit": 350.0, "gpu_pcie_gen": 5, "gpu_pcie_gen_max": 5, "gpu_pcie_width": 8, "gpu_pcie_rx_mb": 3303.107421875, "gpu_pcie_tx_mb": 378.5078125, "cpu": 4.2, "ram_used": 65281994752, "ram_total": 66472062976, "disk_read_mb": 0.15867704562788276, "disk_write_mb": 0.0, "tok_s": 0.0, "tok_s_mean": 0, "prefill_tok_s_mean": 1851.6}, "hardware_static": {"gpu_name": "NVIDIA GeForce RTX 5060 Ti + NVIDIA GeForce RTX 3060", "gpu_count": 2, "gpu_note": null, "cpu_name": "AMD Ryzen 9 7900X 12-Core Processor", "cores": 12, "threads": 24, "psutil": true}, "history": {"gpu_util": [50.0, 50.0, 48.5, 50.0, 50.0, 50.0, 51.5, 50.0, 50.0, 50.0, 100.0, 99.0, 50.0, 50.0, 98.5, 96.5, 50.0, 100.0, 99.5, 50.0, 97.5, 50.0, 84.5, 50.0, 44.0, 50.0, 50.0, 50.0, 100.0, 100.0, 50.0, 100.0, 99.0, 96.5, 50.0, 100.0, 99.5, 50.0, 81.0, 50.0, 99.0, 50.0, 34.5, 50.0, 50.0, 50.0, 99.5, 100.0, 50.0, 98.0, 99.5, 98.0, 50.0, 99.5, 100.0, 50.0, 66.0, 50.0, 100.0, 50.0], "gpu_mem_used": [27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664], "gpu_temp": [58, 58, 59, 63, 60, 57, 58, 63, 64, 67, 69, 68, 68, 69, 69, 68, 68, 69, 67, 70, 62, 60, 58, 56, 56, 67, 68, 69, 66, 70, 69, 64, 68, 68, 68, 65, 70, 69, 61, 59, 57, 55, 55, 65, 68, 67, 69, 71, 70, 63, 71, 71, 69, 70, 70, 70, 60, 58, 56, 55], "gpu_power": [145.56, 124.14, 100.13, 188.76, 140.19, 109.31, 109.67, 181.74, 184.51, 183.37, 209.51, 254.57, 238.39, 190.94, 213.31, 243.21, 205.2, 206.21, 254.23, 253.31, 189.99, 142.18, 134.52, 73.38, 111.32, 181.77, 186.0, 183.67, 209.81, 254.88, 233.84, 191.69, 240.83, 247.41, 209.52, 209.49, 254.08, 254.56, 185.25, 148.72, 133.8, 73.0, 118.71, 189.23, 187.05, 184.57, 211.46, 256.15, 233.73, 191.06, 242.7, 248.76, 209.78, 209.86, 254.55, 254.44, 185.42, 146.12, 132.15, 73.1], "gpu_pcie_rx_mb": [3295.24, 3293.38, 26.56, 21768.4, 3292.83, 3291.55, 6.5, 2.07, 26236.72, 1.44, 22074.06, 17135.75, 7672.05, 25637.73, 20742.08, 4380.14, 1.6, 32762.14, 29111.9, 11048.57, 3296.1, 3315.07, 7717.54, 3294.92, 26283.24, 8594.0, 25505.7, 5.02, 9878.44, 29809.27, 1.84, 32878.34, 20425.85, 27574.02, 6155.5, 33125.56, 28353.54, 23832.45, 3293.14, 3297.96, 3295.3, 3292.37, 9678.03, 27920.56, 9.95, 1.89, 32812.93, 63.74, 53.88, 12632.15, 3305.78, 6.03, 2.02, 5519.9, 6060.16, 9753.1, 3288.95, 3299.06, 3290.38, 3303.11], "cpu": [4.2, 5.8, 16.4, 4.2, 4.2, 6.2, 23.6, 4.2, 4.2, 4.2, 4.4, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.3, 4.1, 4.3, 23.4, 4.2, 4.2, 4.3, 4.2, 4.2, 4.3, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 24.0, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2], "ram_used": [65682419712, 65663201280, 65663717376, 65663717376, 65663717376, 65641394176, 65042579456, 65042579456, 65042579456, 65042567168, 65047609344, 65047609344, 65047609344, 65137496064, 65137496064, 65150869504, 65150730240, 65150730240, 65150730240, 65150701568, 65150459904, 65150459904, 65255989248, 65256538112, 65054031872, 65054289920, 65054289920, 65054289920, 65046265856, 65046581248, 65047613440, 65154277376, 65154277376, 65164664832, 65164664832, 65164922880, 65164902400, 65164902400, 65165078528, 65165078528, 65269936128, 65269936128, 65073586176, 65073586176, 65073586176, 65073586176, 65073831936, 65073684480, 65073684480, 65178800128, 65178800128, 65178800128, 65179054080, 65179054080, 65179054080, 65179299840, 65179299840, 65179279360, 65281748992, 65281994752], "disk_read_mb": [0.16, 0.14, 1.77, 0.16, 0.16, 0.2, 7.53, 0.14, 0.16, 0.16, 83.3, 0.14, 0.16, 29.7, 0.15, 5.91, 0.16, 0.17, 0.16, 0.16, 3.84, 0.36, 0.16, 0.98, 6.15, 0.16, 0.16, 0.16, 0.18, 0.15, 0.16, 0.16, 0.17, 0.17, 0.16, 1.44, 0.16, 0.18, 0.16, 0.16, 0.16, 0.14, 2.39, 0.16, 0.14, 0.16, 0.16, 0.76, 0.16, 0.16, 0.14, 0.2, 0.16, 0.16, 0.17, 0.16, 0.14, 0.17, 0.16, 0.16], "tok_s": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "prefill_tok_s_mean": [null, 929.9, null, null, null, 925.9, null, null, null, null, null, null, 1399.1, 1399.1, 1399.1, 1653.6, 1653.6, 1653.6, 1653.6, 1787.6, 1787.6, 1787.6, 1853.2, 1853.2, null, null, null, null, null, null, 1400.9, 1400.9, 1400.9, 1652.0, 1652.0, 1652.0, 1652.0, 1785.1, 1785.1, 1785.1, 1850.3, 1850.3, null, null, null, null, null, null, 1400.5, 1400.5, 1400.5, 1652.3, 1652.3, 1652.3, 1652.3, 1786.4, 1786.4, 1786.4, 1851.6, 1851.6]}, "time": 1791786834.8303797} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/B1d-after.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/B1d-after.json new file mode 100644 index 00000000000..8aa32bce63d --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/B1d-after.json @@ -0,0 +1 @@ +{"engine": {"model": "qwen3.8-flash-next-iq3_s", "max_context": 524288, "images": true, "version": "0.1.43", "logit_bias": 1, "context": 524288, "kv": "int8", "kv_resident": 20480, "expert_slots": 7946, "expert_cache_mib": 16078, "expert_slots_primary": 4874, "expert_cache_primary_mib": 9316, "spec": 6, "mtp_max": 4, "lookup": 3, "vram_free_mib": 324, "cvec": 0, "arena_mib": 47962, "pool_workers": 11, "pcie_frac": "0.00", "spec_min_p": "0.70", "conversation_cache_mib": 0, "conversation_cache_slots": 4, "conversation_cache_min_free_mib": 2560, "conversation_cache_min_tokens": 0, "tail_role_token": -1, "vram_elastic": 0, "engine": "0.1.43"}, "live": {"state": "idle", "queued": 0, "phase": null, "prompt_tokens": null, "prompt_read": null, "prompt_total": null, "generated": null, "max_tokens": null, "elapsed_s": null, "tok_s": null, "tok_s_mean": null, "prefill_tok_s_mean": null, "tok_s_window_s": null}, "requests": [{"projection": null, "time": 1791786815.341009, "duration_s": 19.4, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 19037.9, "decode_ms": 403.8, "decode_tok_s": 59.4, "hit_rate": 0.924, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 19, "drafts_accepted": 10}, {"projection": null, "time": 1791786795.8590991, "duration_s": 19.5, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 19049.2, "decode_ms": 407.5, "decode_tok_s": 58.9, "hit_rate": 0.846, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 19, "drafts_accepted": 10}, {"projection": null, "time": 1791786776.4215596, "duration_s": 19.4, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 19024.9, "decode_ms": 387.3, "decode_tok_s": 62.0, "hit_rate": 0.679, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 13, "drafts_accepted": 11}, {"projection": null, "time": 1791786771.8809876, "duration_s": 4.5, "finish": "stop", "prompt_tokens": 14455, "reused": 11609, "output_tokens": 24, "prompt_total": 14455, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 4117.0, "decode_ms": 400.8, "decode_tok_s": 59.9, "hit_rate": 0.908, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 24, "drafts_accepted": 9}, {"projection": null, "time": 1791786767.5233965, "duration_s": 4.3, "finish": "stop", "prompt_tokens": 11590, "reused": 8730, "output_tokens": 20, "prompt_total": 11590, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 20, "prompt_ms": 4113.3, "decode_ms": 231.0, "decode_tok_s": 86.6, "hit_rate": 0.925, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 18, "drafts_accepted": 12}, {"projection": null, "time": 1791786762.60912, "duration_s": 4.9, "finish": "stop", "prompt_tokens": 8692, "reused": 5312, "output_tokens": 39, "prompt_total": 8692, "prompt_read": 3380, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 39, "prompt_ms": 4450.9, "decode_ms": 451.5, "decode_tok_s": 86.4, "hit_rate": 0.868, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 30, "drafts_accepted": 23}, {"projection": null, "time": 1791786758.6010194, "duration_s": 4.0, "finish": "length", "prompt_tokens": 5248, "reused": 3778, "output_tokens": 64, "prompt_total": 5248, "prompt_read": 1470, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 3199.3, "decode_ms": 798.3, "decode_tok_s": 80.2, "hit_rate": 0.853, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 56, "drafts_accepted": 37}, {"projection": null, "time": 1791786753.7350805, "duration_s": 4.9, "finish": "length", "prompt_tokens": 3715, "reused": 633, "output_tokens": 64, "prompt_total": 3715, "prompt_read": 3082, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 4211.5, "decode_ms": 645.7, "decode_tok_s": 99.1, "hit_rate": 0.866, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 49, "drafts_accepted": 42}, {"projection": null, "time": 1791786751.4604647, "duration_s": 2.3, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 24, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 1967.3, "decode_ms": 299.7, "decode_tok_s": 80.1, "hit_rate": 0.753, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 24, "drafts_accepted": 15}, {"projection": null, "time": 1791786747.104432, "duration_s": 4.3, "finish": "stop", "prompt_tokens": 14421, "reused": 11575, "output_tokens": 12, "prompt_total": 14421, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 12, "prompt_ms": 4102.1, "decode_ms": 242.4, "decode_tok_s": 49.5, "hit_rate": 0.883, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 14, "drafts_accepted": 3}, {"projection": null, "time": 1791786742.9023848, "duration_s": 4.2, "finish": "stop", "prompt_tokens": 11570, "reused": 8710, "output_tokens": 6, "prompt_total": 11570, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 6, "prompt_ms": 4108.3, "decode_ms": 81.5, "decode_tok_s": 73.6, "hit_rate": 0.921, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 6, "drafts_accepted": 3}, {"projection": null, "time": 1791786738.0931792, "duration_s": 4.8, "finish": "stop", "prompt_tokens": 8692, "reused": 5241, "output_tokens": 19, "prompt_total": 8692, "prompt_read": 3451, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 19, "prompt_ms": 4501.5, "decode_ms": 296.5, "decode_tok_s": 64.1, "hit_rate": 0.886, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 18, "drafts_accepted": 8}], "conversation_cache": {"enabled": false, "budget_mib": 0, "slots": 4, "parked": 0, "bytes": 0, "evictions": 0, "parks": 0, "restores": 0, "last_event": null, "last_tokens": null, "last_at": null, "requests": 40, "requests_reused": 15, "reused_tokens": 89948, "prompt_tokens": 291853, "last_reused": 0, "last_prompt": 32049}, "requests_kept": 40, "totals": {"since": 1791786506.1800041, "requests": 40, "prompt_tokens": 291853, "reused": 89948, "output_tokens": 1798, "prompt_ms": 180813.30000000002, "decode_ms": 23476.9, "drafts_offered": 1434, "drafts_accepted": 1002}, "hardware": {"gpus": [{"index": 0, "util": 0, "mem_used": 16836001792, "mem_total": 17103323136, "temp": 51, "power": 26.191}, {"index": 1, "util": 100, "mem_used": 10562895872, "mem_total": 12884901888, "temp": 55, "power": 46.913}], "gpu_util": 50.0, "gpu_mem_used": 27398897664, "gpu_mem_total": 29988225024, "gpu_temp": 55, "gpu_power": 73.104, "gpu_power_limit": 350.0, "gpu_pcie_gen": 5, "gpu_pcie_gen_max": 5, "gpu_pcie_width": 8, "gpu_pcie_rx_mb": 3303.107421875, "gpu_pcie_tx_mb": 378.5078125, "cpu": 4.2, "ram_used": 65281994752, "ram_total": 66472062976, "disk_read_mb": 0.15867704562788276, "disk_write_mb": 0.0, "tok_s": 0.0, "tok_s_mean": 0, "prefill_tok_s_mean": 1851.6}, "hardware_static": {"gpu_name": "NVIDIA GeForce RTX 5060 Ti + NVIDIA GeForce RTX 3060", "gpu_count": 2, "gpu_note": null, "cpu_name": "AMD Ryzen 9 7900X 12-Core Processor", "cores": 12, "threads": 24, "psutil": true}, "history": {"gpu_util": [50.0, 50.0, 48.5, 50.0, 50.0, 50.0, 51.5, 50.0, 50.0, 50.0, 100.0, 99.0, 50.0, 50.0, 98.5, 96.5, 50.0, 100.0, 99.5, 50.0, 97.5, 50.0, 84.5, 50.0, 44.0, 50.0, 50.0, 50.0, 100.0, 100.0, 50.0, 100.0, 99.0, 96.5, 50.0, 100.0, 99.5, 50.0, 81.0, 50.0, 99.0, 50.0, 34.5, 50.0, 50.0, 50.0, 99.5, 100.0, 50.0, 98.0, 99.5, 98.0, 50.0, 99.5, 100.0, 50.0, 66.0, 50.0, 100.0, 50.0], "gpu_mem_used": [27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27396800512, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664, 27398897664], "gpu_temp": [58, 58, 59, 63, 60, 57, 58, 63, 64, 67, 69, 68, 68, 69, 69, 68, 68, 69, 67, 70, 62, 60, 58, 56, 56, 67, 68, 69, 66, 70, 69, 64, 68, 68, 68, 65, 70, 69, 61, 59, 57, 55, 55, 65, 68, 67, 69, 71, 70, 63, 71, 71, 69, 70, 70, 70, 60, 58, 56, 55], "gpu_power": [145.56, 124.14, 100.13, 188.76, 140.19, 109.31, 109.67, 181.74, 184.51, 183.37, 209.51, 254.57, 238.39, 190.94, 213.31, 243.21, 205.2, 206.21, 254.23, 253.31, 189.99, 142.18, 134.52, 73.38, 111.32, 181.77, 186.0, 183.67, 209.81, 254.88, 233.84, 191.69, 240.83, 247.41, 209.52, 209.49, 254.08, 254.56, 185.25, 148.72, 133.8, 73.0, 118.71, 189.23, 187.05, 184.57, 211.46, 256.15, 233.73, 191.06, 242.7, 248.76, 209.78, 209.86, 254.55, 254.44, 185.42, 146.12, 132.15, 73.1], "gpu_pcie_rx_mb": [3295.24, 3293.38, 26.56, 21768.4, 3292.83, 3291.55, 6.5, 2.07, 26236.72, 1.44, 22074.06, 17135.75, 7672.05, 25637.73, 20742.08, 4380.14, 1.6, 32762.14, 29111.9, 11048.57, 3296.1, 3315.07, 7717.54, 3294.92, 26283.24, 8594.0, 25505.7, 5.02, 9878.44, 29809.27, 1.84, 32878.34, 20425.85, 27574.02, 6155.5, 33125.56, 28353.54, 23832.45, 3293.14, 3297.96, 3295.3, 3292.37, 9678.03, 27920.56, 9.95, 1.89, 32812.93, 63.74, 53.88, 12632.15, 3305.78, 6.03, 2.02, 5519.9, 6060.16, 9753.1, 3288.95, 3299.06, 3290.38, 3303.11], "cpu": [4.2, 5.8, 16.4, 4.2, 4.2, 6.2, 23.6, 4.2, 4.2, 4.2, 4.4, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.3, 4.1, 4.3, 23.4, 4.2, 4.2, 4.3, 4.2, 4.2, 4.3, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 24.0, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2], "ram_used": [65682419712, 65663201280, 65663717376, 65663717376, 65663717376, 65641394176, 65042579456, 65042579456, 65042579456, 65042567168, 65047609344, 65047609344, 65047609344, 65137496064, 65137496064, 65150869504, 65150730240, 65150730240, 65150730240, 65150701568, 65150459904, 65150459904, 65255989248, 65256538112, 65054031872, 65054289920, 65054289920, 65054289920, 65046265856, 65046581248, 65047613440, 65154277376, 65154277376, 65164664832, 65164664832, 65164922880, 65164902400, 65164902400, 65165078528, 65165078528, 65269936128, 65269936128, 65073586176, 65073586176, 65073586176, 65073586176, 65073831936, 65073684480, 65073684480, 65178800128, 65178800128, 65178800128, 65179054080, 65179054080, 65179054080, 65179299840, 65179299840, 65179279360, 65281748992, 65281994752], "disk_read_mb": [0.16, 0.14, 1.77, 0.16, 0.16, 0.2, 7.53, 0.14, 0.16, 0.16, 83.3, 0.14, 0.16, 29.7, 0.15, 5.91, 0.16, 0.17, 0.16, 0.16, 3.84, 0.36, 0.16, 0.98, 6.15, 0.16, 0.16, 0.16, 0.18, 0.15, 0.16, 0.16, 0.17, 0.17, 0.16, 1.44, 0.16, 0.18, 0.16, 0.16, 0.16, 0.14, 2.39, 0.16, 0.14, 0.16, 0.16, 0.76, 0.16, 0.16, 0.14, 0.2, 0.16, 0.16, 0.17, 0.16, 0.14, 0.17, 0.16, 0.16], "tok_s": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "prefill_tok_s_mean": [null, 929.9, null, null, null, 925.9, null, null, null, null, null, null, 1399.1, 1399.1, 1399.1, 1653.6, 1653.6, 1653.6, 1653.6, 1787.6, 1787.6, 1787.6, 1853.2, 1853.2, null, null, null, null, null, null, 1400.9, 1400.9, 1400.9, 1652.0, 1652.0, 1652.0, 1652.0, 1785.1, 1785.1, 1785.1, 1850.3, 1850.3, null, null, null, null, null, null, 1400.5, 1400.5, 1400.5, 1652.3, 1652.3, 1652.3, 1652.3, 1786.4, 1786.4, 1786.4, 1851.6, 1851.6]}, "time": 1791786834.8427439} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/B1d-before.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/B1d-before.json new file mode 100644 index 00000000000..b421e7d2f3a --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/B1d-before.json @@ -0,0 +1 @@ +{"engine": {"model": "qwen3.8-flash-next-iq3_s", "max_context": 524288, "images": true, "version": "0.1.43", "logit_bias": 1, "context": 524288, "kv": "int8", "kv_resident": 20480, "expert_slots": 7946, "expert_cache_mib": 16078, "expert_slots_primary": 4874, "expert_cache_primary_mib": 9316, "spec": 6, "mtp_max": 4, "lookup": 3, "vram_free_mib": 324, "cvec": 0, "arena_mib": 47962, "pool_workers": 11, "pcie_frac": "0.00", "spec_min_p": "0.70", "conversation_cache_mib": 0, "conversation_cache_slots": 4, "conversation_cache_min_free_mib": 2560, "conversation_cache_min_tokens": 0, "tail_role_token": -1, "vram_elastic": 0, "engine": "0.1.43"}, "live": {"state": "idle", "queued": 0, "phase": null, "prompt_tokens": null, "prompt_read": null, "prompt_total": null, "generated": null, "max_tokens": null, "elapsed_s": null, "tok_s": null, "tok_s_mean": null, "prefill_tok_s_mean": null, "tok_s_window_s": null}, "requests": [], "conversation_cache": {"enabled": false, "budget_mib": 0, "slots": 4, "parked": 0, "bytes": 0, "evictions": 0, "parks": 0, "restores": 0, "last_event": null, "last_tokens": null, "last_at": null, "requests": 0, "requests_reused": 0, "reused_tokens": 0, "prompt_tokens": 0, "last_reused": null, "last_prompt": null}, "requests_kept": 0, "totals": {"since": 1791786506.1800041, "requests": 0, "prompt_tokens": 0, "reused": 0, "output_tokens": 0, "prompt_ms": 0.0, "decode_ms": 0.0, "drafts_offered": 0, "drafts_accepted": 0}, "hardware": {"gpus": [{"index": 0, "util": 0, "mem_used": 16762601472, "mem_total": 17103323136, "temp": 35, "power": 4.56}, {"index": 1, "util": 0, "mem_used": 10508369920, "mem_total": 12884901888, "temp": 39, "power": 7.596}], "gpu_util": 0.0, "gpu_mem_used": 27270971392, "gpu_mem_total": 29988225024, "gpu_temp": 39, "gpu_power": 12.155999999999999, "gpu_power_limit": 350.0, "gpu_pcie_gen": 1, "gpu_pcie_gen_max": 5, "gpu_pcie_width": 8, "gpu_pcie_rx_mb": 1.0126953125, "gpu_pcie_tx_mb": 1.328125, "cpu": 0.1, "ram_used": 65586728960, "ram_total": 66472062976, "disk_read_mb": 0.0, "disk_write_mb": 0.0, "tok_s": 0.0, "tok_s_mean": 0, "prefill_tok_s_mean": 0.0}, "hardware_static": {"gpu_name": "NVIDIA GeForce RTX 5060 Ti + NVIDIA GeForce RTX 3060", "gpu_count": 2, "gpu_note": null, "cpu_name": "AMD Ryzen 9 7900X 12-Core Processor", "cores": 12, "threads": 24, "psutil": true}, "history": {"gpu_util": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "gpu_mem_used": [27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392, 27270971392], "gpu_temp": [41, 41, 40, 40, 41, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 39], "gpu_power": [12.21, 12.27, 12.26, 12.26, 12.34, 12.29, 12.33, 12.36, 12.3, 12.33, 12.28, 12.19, 12.22, 12.33, 12.28, 12.2, 12.24, 12.22, 12.31, 12.34, 12.29, 12.2, 12.18, 12.17, 12.17, 12.22, 12.27, 12.33, 12.32, 12.26, 12.29, 12.27, 12.3, 12.3, 12.28, 12.32, 12.21, 12.13, 12.22, 12.2, 12.19, 12.23, 12.23, 12.28, 12.21, 12.14, 12.1, 12.1, 12.17, 12.16, 12.18, 12.22, 12.27, 12.22, 12.26, 12.16, 12.15, 12.09, 12.08, 12.16], "gpu_pcie_rx_mb": [1.02, 1.17, 0.98, 0.96, 0.98, 0.93, 0.99, 1.0, 0.95, 1.0, 1.03, 1.0, 0.99, 1.0, 1.0, 0.99, 0.99, 1.0, 1.21, 0.99, 0.93, 0.98, 0.91, 0.99, 0.98, 0.94, 0.93, 0.95, 0.98, 0.99, 0.99, 0.91, 0.98, 0.99, 0.99, 1.19, 0.99, 0.94, 0.98, 0.87, 0.99, 0.93, 0.94, 0.99, 0.99, 0.96, 1.0, 0.99, 0.94, 0.99, 0.98, 0.98, 1.11, 0.93, 1.0, 1.0, 1.0, 0.99, 0.94, 1.01], "cpu": [0.0, 0.0, 0.1, 0.0, 0.1, 0.0, 0.1, 0.1, 0.0, 0.0, 0.1, 0.0, 0.1, 0.0, 0.1, 0.0, 0.1, 0.0, 0.1, 0.0, 0.1, 0.1, 0.1, 0.0, 0.1, 0.1, 0.1, 0.1, 0.0, 0.0, 0.1, 0.1, 0.0, 0.0, 0.0, 0.1, 0.1, 0.1, 0.0, 0.1, 0.0, 0.2, 0.0, 0.1, 0.0, 0.0, 0.1, 0.1, 0.1, 0.1, 0.0, 0.0, 0.1, 0.1, 0.0, 0.1, 0.0, 0.1, 0.0, 0.1], "ram_used": [65580511232, 65579319296, 65579319296, 65579057152, 65579057152, 65578356736, 65578356736, 65578356736, 65578356736, 65578356736, 65578614784, 65578614784, 65578614784, 65579855872, 65580421120, 65580421120, 65580421120, 65580552192, 65580437504, 65580437504, 65580437504, 65580437504, 65580437504, 65580437504, 65580437504, 65580437504, 65580437504, 65580437504, 65580437504, 65580437504, 65580437504, 65580437504, 65580437504, 65580437504, 65580437504, 65580437504, 65582100480, 65582120960, 65582120960, 65582120960, 65582120960, 65587695616, 65587339264, 65587339264, 65587339264, 65584939008, 65586499584, 65586499584, 65586499584, 65586499584, 65586499584, 65586499584, 65586499584, 65586499584, 65586499584, 65586499584, 65586499584, 65586499584, 65586499584, 65586728960], "disk_read_mb": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.16, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.14, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.98, 0.0, 0.01, 0.0, 0.0, 35.4, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "tok_s": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "prefill_tok_s_mean": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}, "time": 1791786621.2723434} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/README.md b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/README.md new file mode 100644 index 00000000000..42e4d84b872 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/README.md @@ -0,0 +1,145 @@ +# Community benchmark B1d: `--layer-split 42` — the as-given-frame equivalent of the engine's K=6 idea + +Measured on 2026-10-11 by @mmbeanz. This is the **clean single-variable +test**: identical to run A (`../A/`, `--layer-split 40`, `--pcie-frac 0.00`) except +`40` -> `42`. It asks the question `auto` raised ("K=6 on the slow card, 49.2 vs 48.4 ms, +near tie — try `--layer-split 6`") in the correct card frame, since an explicit split does +not get the #1352 reorder (see `../B1c/README.md`). + +**Headline.** K=42 is a **wash against K=40 on speed** (prompt 32K -2.2%, short-chat decode ++2.4%, decode 32K +11.7% on 3 samples, agent decode -0.5%) but its **expert hit rate is +significantly lower** (0.818 vs 0.859 median, permutation p = 0.024). Conclusion for this +pair: **K=40 stands as the tuned answer**; moving two more layers onto the 12 GB card costs +cache coverage on the 16 GB card without buying anything back. + +## Layout (from `GET /metrics`, card identity per the A/B1c log convention) + +| | CUDA0 = **5060 Ti** (16 GB, 0.78 ms/layer) | CUDA1 = **3060** (12 GB, 1.46 ms/layer) | total | +| --- | --- | --- | ---: | +| layers | 0-41 (42) | 42-47 (6) | 48 | +| expert slots | 4,874 (9,316 MiB) = 22.7% of its 21,504 pairs | 3,072 = **100%** of its 3,072 pairs | **7,946** (f = 0.323) | + +Compare A: 5060 Ti 40 layers / 4,984 slots (24.3% of 20,480); 3060 8 layers / 3,858 slots +(**94%** of its 4,096). Predicted decode window: `42x0.78 + 6x1.46 ~ 41.6 ms` vs A's +`40x0.78 + 8x1.46 ~ 42.9 ms` — the engine's model says 42 should be marginally faster, and +the short-context measurements agree; the prompt path and hit rate do not. + +## Hardware, software, model + +Identical to `../A/README.md` (hand-written tables): 7900X 12c/24t, 64 GB DDR5-6400, +5060 Ti 16 GB on PCIe 5.0 x8 (H2D 28.2-28.7 GB/s) + 3060 12 GB on PCIe 3.0 x4 behind the +chipset (H2D 3.15-3.2 GB/s), 990 EVO Plus 4 TB DRAM-less, driver 615.71.09, CUDA 13.4, +systemd-nspawn on bare metal, Strata 0.1.43 from `7695859a` + `setup.py`-only local commit. +IQ3_S GSQ-RCO pack (sha256 in `../sha256sums.txt`), 512K yarn 2.0, pinned +`data/expert-profile.bin` (24,576 ranked pairs), `pcie_frac 0.00`, `vram_free_mib 324`, +`pool_workers 11`. Config snapshot: `config.json`. + +## Method + +As in A/B1/B1c: `strata_bench.py --url --config config.json --long --runs 3`, +operator-started server, cold boot, 0 requests before the suite; +`GET /metrics?requests=all` before/after. Client on a separate LAN box (RTT 0.25 ms); the +`machine` block blends client/server fields (see `../A/README.md`, *Tool findings*). +Suite ~205 s. + +## Results + +| Configuration | Prompt tok/s (median, range) | Decode tok/s (median, range) | TTFT s (median, range) | +| --- | --- | --- | --- | +| short chat (9 runs) | n/a | 76.2 (70.2-93.1) | 0.40 (0.36-0.45) | +| 4K prompt, fresh | 888.8 (881.3-891.2) | 70.8* | 4.55 (4.54-4.59) | +| 16K prompt, fresh | 1397.3 (1395.9-1400.4) | 58.3* | 11.52 (11.50-11.54) | +| 32K prompt, fresh | 1683.4 (1682.4-1684.6) | 59.4* | 19.08 (19.06-19.09) | +| agent follow-up turns (15) | see `results.json` | 80.3 (49.5-101.3) | 4.16 (3.21-4.57) | + +\* engine per-request `decode_tok_s` from `/metrics?requests=all` (24-token cap, short +window — indicative only). + +### A (K=40) vs B1d (K=42) — one variable + +| metric | A | B1d | delta | +| --- | ---: | ---: | ---: | +| prompt 32K tok/s | 1,720.8 | 1,683.4 | -2.2% | +| prompt 16K tok/s | 1,412.7 | 1,397.3 | -1.1% | +| prompt 4K tok/s | 868.1 | 888.8 | +2.4% | +| TTFT 32K (s) | 18.66 | 19.08 | +2.2% | +| decode 32K tok/s | 53.2 | 59.4 | +11.7% (n=3, 24-token windows) | +| decode 16K tok/s | 58.3 | 58.3 | 0.0% | +| decode 4K tok/s | 70.7 | 70.8 | +0.1% | +| decode 609 tok/s | 71.6 | 74.0 | +3.4% | +| short-chat decode | 74.4 | 76.2 | +2.4% | +| agent decode | 80.7 | 80.3 | -0.5% | +| expert hit_rate (median) | 0.859 | **0.818** | **-4.1 pts** | +| held experts (total) | 8,842 | 7,946 | -10% | +| `pcie_share` | 0 / 0 | 0 / 0 | identical policy | + +Speed: within a few percent in both directions — a **wash**. Hit rate: significantly lower +(see below). + +## Statistics on the hit-rate measure + +Per-request decode expert-cache hit rate, n = 40 per run: + +| run | held | f | median | mean | IQR | 10-90 | +| --- | ---: | ---: | ---: | ---: | --- | --- | +| A (K=40) | 8,842 | 0.360 | 0.859 | 0.842 | 0.774-0.915 | 0.711-0.945 | +| B1 (auto K=3) | 5,935 | 0.241 | 0.865 | 0.840 | 0.783-0.913 | 0.730-0.922 | +| B1c (mis-set K=6) | 5,611 | 0.228 | 0.669 | 0.652 | 0.532-0.758 | 0.485-0.793 | +| B1d (K=42) | 7,946 | 0.323 | 0.818 | 0.787 | 0.703-0.876 | 0.624-0.921 | + +Two-sample permutation tests on the means (5,000 shuffles, two-sided): + +| pair | mean diff | p | verdict | +| --- | ---: | ---: | --- | +| A vs B1 | +0.002 | 0.93 | **not significant** | +| A vs B1d | +0.055 | 0.024 | significant | +| B1 vs B1d | +0.053 | 0.030 | significant | +| A vs B1c | +0.189 | <0.0001 | significant | +| B1d vs B1c | +0.135 | <0.0001 | significant | + +**A and B1 are statistically indistinguishable in hit rate despite a 33% difference in held +experts (8,842 vs 5,935).** That is the strongest evidence here against count-based coverage +curves: the old power law predicts 97.0% vs 95.6%, the new `H(f)` b=3 predicts 73.8% vs +56.4%, and the measurement says "no difference" at ~86%. + +## What the four points do support: per-layer coverage + +A stage can only cache **its own** layers' experts, so the quantity that behaves is +"what fraction of each layer's routed mass can the owning card hold": + +| run | small card (3060) | big card (5060 Ti) | measured hit | +| --- | --- | --- | ---: | +| A | 8 layers, 3,858/4,096 pairs = **94%** | 40 layers, 4,984/20,480 = 24.3% | 0.859 | +| B1 | 3 layers, 1,536/1,536 = **100%** | 45 layers, 4,399/23,040 = 19.1% | 0.865 | +| B1d | 6 layers, 3,072/3,072 = **100%** | 42 layers, 4,874/21,504 = 22.7% | 0.818 | +| B1c | **42 layers**, 2,539/21,504 = 11.8% | 6 layers, 3,072/3,072 = 100% | 0.669 | + +Reading A -> B1d: moving layers 40-41 from the 3060 (which held ~94% of their experts) to +the 5060 Ti (which holds ~23% of its pairs) costs those 2/48 = 4.2% of lookups roughly 70 +points of coverage -> ~3 points of overall hit rate. Measured: 4.1 points. Same order of +magnitude, right direction. B1c is the same effect taken to the extreme (42 layers on the +card that can barely cache any of them). + +Why A and B1 tie despite different totals is the open part: between 3 and 8 layers on the +small card the small card is essentially fully cached either way, and the big card's +fractional coverage (19.1% vs 24.3%) apparently does not move the aggregate — consistent +with the routing being concentrated enough that ~19% of pairs already captures most of the +mass on this model. That is a claim worth testing with a K sweep (36/40/44), not something +these four points can settle. + +## Practical conclusion for this pair + +**Keep `--layer-split 40`.** K=42 is not better (equal speed, significantly worse hit rate), +`auto`'s K=3 is clearly worse on speed (-25% prompt, -10..-19% short decode) even though its +hit rate matches, and the engine's literal `--layer-split 6` suggestion is catastrophic +because it inverts the card assignment. + +## Limitations + +- Speed only; no needle/quality check (needle suite: `../needles/`). +- Decode for the capped prompt tests is a 24-token window (n=3) — the +11.7% at 32K is not + established, only suggestive. +- Hit-rate significance tests treat per-request values as independent samples; the adaptive + tier makes consecutive requests correlated, so p-values are optimistic. +- A vs B1d is one variable; A/B1 vs B1c/B1d also differ in card order and PCIe share. +- No concurrency, no vision latency, no mmap/low-RAM tier, Windows/HIP untested. diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/config.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/config.json new file mode 100644 index 00000000000..b1625fa9c27 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/config.json @@ -0,0 +1,64 @@ +{ + "exe": "/home/llm/Strata/engine/strata", + "args": [ + "--pack", + "/home/llm/Strata-data/packs/iq3_s", + "--native", + "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf", + "--ple-gguf", + "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf", + "--expert-profile", + "/home/llm/Strata/data/expert-profile.bin", + "--expert-cache", + "auto", + "--prefill", + "auto", + "--spec", + "4", + "--mtp", + "/home/llm/Strata-data/mtp/rt", + "--max-context", + "524288", + "--rope-scaling", + "yarn", + "--rope-scale", + "2.0", + "--kv", + "int8", + "--kv-resident", + "20480", + "--vision", + "--remote-expert-opt", + "--trim-stage-weights", + "--pcie-frac", + "0.00", + "--spec-min-p", + "0.70" + ], + "cwd": "/home/llm/Strata", + "tokenizer": "/home/llm/Strata-data/packs/iq3_s/tokenizer", + "model_name": "qwen3.8-flash-next-iq3_s", + "log": "/home/llm/Strata/strata-iq3_s.log", + "lib_dirs": [ + "/opt/cuda/bin", + "/opt/cuda/lib64" + ], + "port": 8080, + "gpu": [ + 0, + 1 + ], + "gpus_asked": true, + "layer_split": "42", + "vision": { + "exe": "/home/llm/Strata/engine/strata-vision", + "mmproj": "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/mmproj-Qwen3.8-Flash-Next-BF16.gguf", + "model": "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf", + "gpu": true, + "max_tokens": 1024 + }, + "allowed_hosts": [ + "bertha" + ], + "draft_vocab": "en" +} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/results.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/results.json new file mode 100644 index 00000000000..5f3260c453b --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/B1d/results.json @@ -0,0 +1,1018 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-11", + "params": { + "runs": 3, + "think": false, + "clients": 0, + "long": true, + "started_server": false, + "command": "python tools/strata_bench.py --long" + }, + "elapsed_s": 204.7869575023651, + "machine": { + "gpu": [ + "NVIDIA GeForce RTX 5060 Ti", + "NVIDIA GeForce RTX 3060" + ], + "gpu_vram_mib": 28599.0, + "gpu_driver": "615.71.09", + "cuda": "13.4 (nvcc V13.4.92)", + "cpu": "AMD Ryzen 9 7900X 12-Core Processor", + "cpu_cores": 12, + "cpu_threads": 24, + "ram_gib": 61.9, + "os": "Linux 7.2.9-1-cachyos", + "python": "3.14.7", + "strata_version": "0.1.43", + "strata_commit": "7695859a", + "engine_version": "0.1.43", + "model": "qwen3.8-flash-next-iq3_s", + "quant": "IQ3_S", + "context": 524288, + "parallel_slots": 1, + "config": { + "exe": "strata", + "args": [ + "--pack", + "iq3_s", + "--native", + "Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf", + "--ple-gguf", + "Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf", + "--expert-profile", + "expert-profile-live.bin", + "--expert-profile-save", + "expert-profile-live.bin", + "--expert-cache", + "auto", + "--prefill", + "auto", + "--spec", + "4", + "--mtp", + "rt", + "--max-context", + "524288", + "--rope-scaling", + "yarn", + "--rope-scale", + "2.0", + "--kv", + "int8", + "--kv-resident", + "20480", + "--vision", + "--remote-expert-opt", + "--trim-stage-weights", + "--pcie-frac", + "0.00", + "--spec-min-p", + "0.70" + ], + "model_name": "qwen3.8-flash-next-iq3_s", + "lib_dirs": [ + "/opt/cuda/bin", + "/opt/cuda/lib64" + ], + "port": 8080, + "gpu": [ + 0, + 1 + ], + "gpus_asked": true, + "layer_split": "42", + "vision": { + "exe": "/home//Strata/engine/strata-vision", + "mmproj": "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/mmproj-Qwen3.8-Flash-Next-BF16.gguf", + "model": "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf", + "gpu": true, + "max_tokens": 1024 + }, + "allowed_hosts": [ + "bertha" + ], + "draft_vocab": "en" + }, + "launch": "strata --pack iq3_s --native Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf --ple-gguf Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf --expert-profile expert-profile-live.bin --expert-profile-save expert-profile-live.bin --expert-cache auto --prefill auto --spec 4 --mtp rt --max-context 524288 --rope-scaling yarn --rope-scale 2.0 --kv int8 --kv-resident 20480 --vision --remote-expert-opt --trim-stage-weights --pcie-frac 0.00 --spec-min-p 0.70", + "engine_info": { + "model": "qwen3.8-flash-next-iq3_s", + "max_context": 524288, + "images": true, + "version": "0.1.43", + "logit_bias": 1, + "context": 524288, + "kv": "int8", + "kv_resident": 20480, + "expert_slots": 7946, + "expert_cache_mib": 16078, + "expert_slots_primary": 4874, + "expert_cache_primary_mib": 9316, + "spec": 6, + "mtp_max": 4, + "lookup": 3, + "vram_free_mib": 324, + "cvec": 0, + "arena_mib": 47962, + "pool_workers": 11, + "pcie_frac": "0.00", + "spec_min_p": "0.70", + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "conversation_cache_min_tokens": 0, + "tail_role_token": -1, + "vram_elastic": 0, + "engine": "0.1.43" + }, + "vram_after": { + "elastic": false + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.45, + "total_s": 1.513, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 119.6, + "prompt_tps_source": "engine", + "decode_tps": 92.5, + "decode_tps_source": "engine", + "prompt_ms": 426.3, + "decode_ms": 1080.8, + "drafts_offered": 82, + "drafts_accepted": 70, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.394, + "total_s": 1.774, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 116.4, + "prompt_tps_source": "engine", + "decode_tps": 71.5, + "decode_tps_source": "engine", + "prompt_ms": 369.5, + "decode_ms": 1398.0, + "drafts_offered": 77, + "drafts_accepted": 48, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.42, + "total_s": 1.828, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 116.4, + "prompt_tps_source": "engine", + "decode_tps": 70.2, + "decode_tps_source": "engine", + "prompt_ms": 395.3, + "decode_ms": 1425.5, + "drafts_offered": 79, + "drafts_accepted": 49, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.438, + "total_s": 1.53, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 123.0, + "prompt_tps_source": "engine", + "decode_tps": 90.2, + "decode_tps_source": "engine", + "prompt_ms": 414.8, + "decode_ms": 1109.2, + "drafts_offered": 84, + "drafts_accepted": 70, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.368, + "total_s": 1.726, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 124.8, + "prompt_tps_source": "engine", + "decode_tps": 72.7, + "decode_tps_source": "engine", + "prompt_ms": 344.5, + "decode_ms": 1375.5, + "drafts_offered": 75, + "drafts_accepted": 46, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.397, + "total_s": 1.691, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 123.2, + "prompt_tps_source": "engine", + "decode_tps": 76.2, + "decode_tps_source": "engine", + "prompt_ms": 373.5, + "decode_ms": 1311.6, + "drafts_offered": 72, + "drafts_accepted": 51, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.441, + "total_s": 1.497, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 122.3, + "prompt_tps_source": "engine", + "decode_tps": 93.1, + "decode_tps_source": "engine", + "prompt_ms": 417.0, + "decode_ms": 1074.1, + "drafts_offered": 75, + "drafts_accepted": 70, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.362, + "total_s": 1.689, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 126.8, + "prompt_tps_source": "engine", + "decode_tps": 74.4, + "decode_tps_source": "engine", + "prompt_ms": 339.0, + "decode_ms": 1344.3, + "drafts_offered": 87, + "drafts_accepted": 51, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.391, + "total_s": 1.683, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 125.2, + "prompt_tps_source": "engine", + "decode_tps": 76.4, + "decode_tps_source": "engine", + "prompt_ms": 367.5, + "decode_ms": 1309.2, + "drafts_offered": 72, + "drafts_accepted": 51, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 76.2, + "min": 70.2, + "max": 93.1, + "n": 9 + }, + "ttft_s": { + "median": 0.397, + "min": 0.362, + "max": 0.45, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 46, + "min": 43, + "max": 51, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 4.586, + "total_s": 4.907, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 881.3, + "prompt_tps_source": "engine", + "decode_tps": 70.8, + "decode_tps_source": "engine", + "prompt_ms": 4560.5, + "decode_ms": 339.2, + "drafts_offered": 15, + "drafts_accepted": 12, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.547, + "total_s": 4.873, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 888.8, + "prompt_tps_source": "engine", + "decode_tps": 69.9, + "decode_tps_source": "engine", + "prompt_ms": 4521.6, + "decode_ms": 343.4, + "drafts_offered": 13, + "drafts_accepted": 10, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.536, + "total_s": 4.856, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 891.2, + "prompt_tps_source": "engine", + "decode_tps": 71.2, + "decode_tps_source": "engine", + "prompt_ms": 4509.8, + "decode_ms": 337.3, + "drafts_offered": 13, + "drafts_accepted": 10, + "run": 2 + } + ], + "prompt_tps": { + "median": 888.8, + "min": 881.3, + "max": 891.2, + "n": 3 + }, + "ttft_s": { + "median": 4.547, + "min": 4.536, + "max": 4.586, + "n": 3 + }, + "prompt_tokens": { + "median": 4019, + "min": 4019, + "max": 4019, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 11.498, + "total_s": 11.898, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1400.4, + "prompt_tps_source": "engine", + "decode_tps": 57.4, + "decode_tps_source": "engine", + "prompt_ms": 11466.8, + "decode_ms": 417.8, + "drafts_offered": 21, + "drafts_accepted": 10, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 11.523, + "total_s": 11.911, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1397.3, + "prompt_tps_source": "engine", + "decode_tps": 59.1, + "decode_tps_source": "engine", + "prompt_ms": 11492.4, + "decode_ms": 406.4, + "drafts_offered": 24, + "drafts_accepted": 12, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 11.535, + "total_s": 11.929, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1395.9, + "prompt_tps_source": "engine", + "decode_tps": 58.3, + "decode_tps_source": "engine", + "prompt_ms": 11503.4, + "decode_ms": 411.8, + "drafts_offered": 21, + "drafts_accepted": 10, + "run": 2 + } + ], + "prompt_tps": { + "median": 1397.3, + "min": 1395.9, + "max": 1400.4, + "n": 3 + }, + "ttft_s": { + "median": 11.523, + "min": 11.498, + "max": 11.535, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "32K": { + "target_tokens": 32768, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 19.063, + "total_s": 19.432, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 1684.6, + "prompt_tps_source": "engine", + "decode_tps": 62.0, + "decode_tps_source": "engine", + "prompt_ms": 19024.9, + "decode_ms": 387.3, + "drafts_offered": 13, + "drafts_accepted": 11, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 19.087, + "total_s": 19.476, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 1682.4, + "prompt_tps_source": "engine", + "decode_tps": 58.9, + "decode_tps_source": "engine", + "prompt_ms": 19049.2, + "decode_ms": 407.5, + "drafts_offered": 19, + "drafts_accepted": 10, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 19.076, + "total_s": 19.462, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 1683.4, + "prompt_tps_source": "engine", + "decode_tps": 59.4, + "decode_tps_source": "engine", + "prompt_ms": 19037.9, + "decode_ms": 403.8, + "drafts_offered": 19, + "drafts_accepted": 10, + "run": 2 + } + ], + "prompt_tps": { + "median": 1683.4, + "min": 1682.4, + "max": 1684.6, + "n": 3 + }, + "ttft_s": { + "median": 19.076, + "min": 19.063, + "max": 19.087, + "n": 3 + }, + "prompt_tokens": { + "median": 32049, + "min": 32049, + "max": 32049, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 1.983, + "total_s": 2.29, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 21, + "prompt_tps": 311.7, + "prompt_tps_source": "engine", + "decode_tps": 64.7, + "decode_tps_source": "engine", + "prompt_ms": 1954.0, + "decode_ms": 324.4, + "drafts_offered": 25, + "drafts_accepted": 13, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.404, + "total_s": 5.147, + "prompt_tokens": 3712, + "reused_tokens": 631, + "read_tokens": 3081, + "generated_tokens": 64, + "prompt_tps": 705.7, + "prompt_tps_source": "engine", + "decode_tps": 84.2, + "decode_tps_source": "engine", + "prompt_ms": 4365.8, + "decode_ms": 760.3, + "drafts_offered": 49, + "drafts_accepted": 42, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.296, + "total_s": 4.05, + "prompt_tokens": 5245, + "reused_tokens": 3705, + "read_tokens": 1540, + "generated_tokens": 64, + "prompt_tps": 471.5, + "prompt_tps_source": "engine", + "decode_tps": 83.0, + "decode_tps_source": "engine", + "prompt_ms": 3265.9, + "decode_ms": 771.0, + "drafts_offered": 53, + "drafts_accepted": 39, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.573, + "total_s": 4.892, + "prompt_tokens": 8689, + "reused_tokens": 5309, + "read_tokens": 3380, + "generated_tokens": 23, + "prompt_tps": 744.4, + "prompt_tps_source": "engine", + "decode_tps": 68.3, + "decode_tps_source": "engine", + "prompt_ms": 4540.5, + "decode_ms": 336.8, + "drafts_offered": 21, + "drafts_accepted": 11, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.243, + "total_s": 4.445, + "prompt_tokens": 11571, + "reused_tokens": 8711, + "read_tokens": 2860, + "generated_tokens": 21, + "prompt_tps": 679.2, + "prompt_tps_source": "engine", + "decode_tps": 96.0, + "decode_tps_source": "engine", + "prompt_ms": 4210.9, + "decode_ms": 218.7, + "drafts_offered": 17, + "drafts_accepted": 14, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.157, + "total_s": 4.6, + "prompt_tokens": 14437, + "reused_tokens": 11591, + "read_tokens": 2846, + "generated_tokens": 31, + "prompt_tps": 690.0, + "prompt_tps_source": "engine", + "decode_tps": 67.3, + "decode_tps_source": "engine", + "prompt_ms": 4124.4, + "decode_ms": 460.8, + "drafts_offered": 25, + "drafts_accepted": 14, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.994, + "total_s": 2.301, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 24, + "prompt_tps": 309.1, + "prompt_tps_source": "engine", + "decode_tps": 74.0, + "decode_tps_source": "engine", + "prompt_ms": 1970.1, + "decode_ms": 324.4, + "drafts_offered": 25, + "drafts_accepted": 16, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.239, + "total_s": 4.853, + "prompt_tokens": 3715, + "reused_tokens": 635, + "read_tokens": 3080, + "generated_tokens": 64, + "prompt_tps": 731.0, + "prompt_tps_source": "engine", + "decode_tps": 101.3, + "decode_tps_source": "engine", + "prompt_ms": 4213.3, + "decode_ms": 631.9, + "drafts_offered": 45, + "drafts_accepted": 42, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.205, + "total_s": 3.984, + "prompt_tokens": 5248, + "reused_tokens": 3778, + "read_tokens": 1470, + "generated_tokens": 64, + "prompt_tps": 462.5, + "prompt_tps_source": "engine", + "decode_tps": 80.3, + "decode_tps_source": "engine", + "prompt_ms": 3178.6, + "decode_ms": 796.7, + "drafts_offered": 49, + "drafts_accepted": 38, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.529, + "total_s": 4.808, + "prompt_tokens": 8692, + "reused_tokens": 5241, + "read_tokens": 3451, + "generated_tokens": 19, + "prompt_tps": 766.6, + "prompt_tps_source": "engine", + "decode_tps": 64.1, + "decode_tps_source": "engine", + "prompt_ms": 4501.5, + "decode_ms": 296.5, + "drafts_offered": 18, + "drafts_accepted": 8, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.137, + "total_s": 4.201, + "prompt_tokens": 11570, + "reused_tokens": 8710, + "read_tokens": 2860, + "generated_tokens": 6, + "prompt_tps": 696.2, + "prompt_tps_source": "engine", + "decode_tps": 73.6, + "decode_tps_source": "engine", + "prompt_ms": 4108.3, + "decode_ms": 81.5, + "drafts_offered": 6, + "drafts_accepted": 3, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.132, + "total_s": 4.357, + "prompt_tokens": 14421, + "reused_tokens": 11575, + "read_tokens": 2846, + "generated_tokens": 12, + "prompt_tps": 693.8, + "prompt_tps_source": "engine", + "decode_tps": 49.5, + "decode_tps_source": "engine", + "prompt_ms": 4102.1, + "decode_ms": 242.4, + "drafts_offered": 14, + "drafts_accepted": 3, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.992, + "total_s": 2.274, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 24, + "prompt_tps": 309.6, + "prompt_tps_source": "engine", + "decode_tps": 80.1, + "decode_tps_source": "engine", + "prompt_ms": 1967.3, + "decode_ms": 299.7, + "drafts_offered": 24, + "drafts_accepted": 15, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.237, + "total_s": 4.865, + "prompt_tokens": 3715, + "reused_tokens": 633, + "read_tokens": 3082, + "generated_tokens": 64, + "prompt_tps": 731.8, + "prompt_tps_source": "engine", + "decode_tps": 99.1, + "decode_tps_source": "engine", + "prompt_ms": 4211.5, + "decode_ms": 645.7, + "drafts_offered": 49, + "drafts_accepted": 42, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.226, + "total_s": 4.007, + "prompt_tokens": 5248, + "reused_tokens": 3778, + "read_tokens": 1470, + "generated_tokens": 64, + "prompt_tps": 459.5, + "prompt_tps_source": "engine", + "decode_tps": 80.2, + "decode_tps_source": "engine", + "prompt_ms": 3199.3, + "decode_ms": 798.3, + "drafts_offered": 56, + "drafts_accepted": 37, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.479, + "total_s": 4.913, + "prompt_tokens": 8692, + "reused_tokens": 5312, + "read_tokens": 3380, + "generated_tokens": 39, + "prompt_tps": 759.4, + "prompt_tps_source": "engine", + "decode_tps": 86.4, + "decode_tps_source": "engine", + "prompt_ms": 4450.9, + "decode_ms": 451.5, + "drafts_offered": 30, + "drafts_accepted": 23, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.143, + "total_s": 4.356, + "prompt_tokens": 11590, + "reused_tokens": 8730, + "read_tokens": 2860, + "generated_tokens": 20, + "prompt_tps": 695.3, + "prompt_tps_source": "engine", + "decode_tps": 86.6, + "decode_tps_source": "engine", + "prompt_ms": 4113.3, + "decode_ms": 231.0, + "drafts_offered": 18, + "drafts_accepted": 12, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.148, + "total_s": 4.531, + "prompt_tokens": 14455, + "reused_tokens": 11609, + "read_tokens": 2846, + "generated_tokens": 24, + "prompt_tps": 691.3, + "prompt_tps_source": "engine", + "decode_tps": 59.9, + "decode_tps_source": "engine", + "prompt_ms": 4117.0, + "decode_ms": 400.8, + "drafts_offered": 24, + "drafts_accepted": 9, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 68.601, + "min": 68.595, + "max": 68.792, + "n": 3 + }, + "reread_tokens": { + "median": 73, + "min": 3, + "max": 74, + "n": 3 + }, + "new_tokens": { + "median": 13634, + "min": 13633, + "max": 13635, + "n": 3 + }, + "turn_ttft_s": { + "median": 4.157, + "min": 3.205, + "max": 4.573, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 14437, + "min": 14421, + "max": 14455, + "n": 3 + }, + "turn_decode_tps": { + "median": 80.3, + "min": 49.5, + "max": 101.3, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 1.992, + "min": 1.983, + "max": 1.994, + "n": 3 + } + }, + "clients": {}, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/C-after-full.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/C-after-full.json new file mode 100644 index 00000000000..55ccccce406 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/C-after-full.json @@ -0,0 +1 @@ +{"engine": {"model": "qwen3.8-flash-next-iq3_s", "max_context": 524288, "images": true, "version": "0.1.43", "logit_bias": 1, "context": 524288, "kv": "int8", "kv_resident": 20480, "expert_slots": 8842, "expert_cache_mib": 17910, "expert_slots_primary": 4984, "expert_cache_primary_mib": 9474, "spec": 6, "mtp_max": 4, "lookup": 3, "vram_free_mib": 324, "cvec": 0, "arena_mib": 47962, "pool_workers": 11, "pcie_frac": "0.00", "spec_min_p": "0.70", "conversation_cache_mib": 0, "conversation_cache_slots": 4, "conversation_cache_min_free_mib": 2560, "conversation_cache_min_tokens": 0, "tail_role_token": -1, "vram_elastic": 0, "engine": "0.1.43"}, "live": {"state": "idle", "queued": 0, "phase": null, "prompt_tokens": null, "prompt_read": null, "prompt_total": null, "generated": null, "max_tokens": null, "elapsed_s": null, "tok_s": null, "tok_s_mean": null, "prefill_tok_s_mean": null, "tok_s_window_s": null}, "requests": [{"projection": null, "time": 1791787591.178125, "duration_s": 19.0, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 18593.1, "decode_ms": 411.4, "decode_tok_s": 58.3, "hit_rate": 0.953, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 19, "drafts_accepted": 10}, {"projection": null, "time": 1791787572.1203792, "duration_s": 19.0, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 18598.5, "decode_ms": 435.2, "decode_tok_s": 55.1, "hit_rate": 0.898, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 24, "drafts_accepted": 12}, {"projection": null, "time": 1791787553.131049, "duration_s": 19.0, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 18601.0, "decode_ms": 362.8, "decode_tok_s": 66.2, "hit_rate": 0.791, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 15, "drafts_accepted": 12}, {"projection": null, "time": 1791787548.5094302, "duration_s": 4.6, "finish": "stop", "prompt_tokens": 14460, "reused": 11614, "output_tokens": 24, "prompt_total": 14460, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 4221.6, "decode_ms": 376.6, "decode_tok_s": 63.7, "hit_rate": 0.925, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 19, "drafts_accepted": 10}, {"projection": null, "time": 1791787544.0395448, "duration_s": 4.5, "finish": "stop", "prompt_tokens": 11587, "reused": 8727, "output_tokens": 28, "prompt_total": 11587, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 28, "prompt_ms": 4189.1, "decode_ms": 268.1, "decode_tok_s": 104.4, "hit_rate": 0.945, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 19, "drafts_accepted": 19}, {"projection": null, "time": 1791787539.0323, "duration_s": 5.0, "finish": "stop", "prompt_tokens": 8687, "reused": 5306, "output_tokens": 41, "prompt_total": 8687, "prompt_read": 3381, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 41, "prompt_ms": 4503.9, "decode_ms": 491.9, "decode_tok_s": 83.4, "hit_rate": 0.916, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 27, "drafts_accepted": 23}, {"projection": null, "time": 1791787534.9737628, "duration_s": 4.1, "finish": "length", "prompt_tokens": 5243, "reused": 3774, "output_tokens": 64, "prompt_total": 5243, "prompt_read": 1469, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 3264.9, "decode_ms": 783.7, "decode_tok_s": 81.7, "hit_rate": 0.889, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 53, "drafts_accepted": 37}, {"projection": null, "time": 1791787529.9899266, "duration_s": 5.0, "finish": "length", "prompt_tokens": 3710, "reused": 630, "output_tokens": 64, "prompt_total": 3710, "prompt_read": 3080, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 4283.5, "decode_ms": 691.4, "decode_tok_s": 92.6, "hit_rate": 0.865, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 48, "drafts_accepted": 42}, {"projection": null, "time": 1791787527.716457, "duration_s": 2.3, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 19, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 19, "prompt_ms": 1999.2, "decode_ms": 266.8, "decode_tok_s": 71.2, "hit_rate": 0.792, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 20, "drafts_accepted": 13}, {"projection": null, "time": 1791787523.1084292, "duration_s": 4.6, "finish": "stop", "prompt_tokens": 14465, "reused": 11619, "output_tokens": 24, "prompt_total": 14465, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 4188.9, "decode_ms": 408.4, "decode_tok_s": 58.8, "hit_rate": 0.928, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 22, "drafts_accepted": 9}, {"projection": null, "time": 1791787518.6444337, "duration_s": 4.5, "finish": "stop", "prompt_tokens": 11594, "reused": 8734, "output_tokens": 26, "prompt_total": 11594, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 26, "prompt_ms": 4177.8, "decode_ms": 272.4, "decode_tok_s": 95.4, "hit_rate": 0.916, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 20, "drafts_accepted": 17}, {"projection": null, "time": 1791787513.5341477, "duration_s": 5.1, "finish": "stop", "prompt_tokens": 8687, "reused": 5307, "output_tokens": 48, "prompt_total": 8687, "prompt_read": 3380, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 48, "prompt_ms": 4532.3, "decode_ms": 565.8, "decode_tok_s": 84.8, "hit_rate": 0.919, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 37, "drafts_accepted": 28}, {"projection": null, "time": 1791787509.4483669, "duration_s": 4.1, "finish": "length", "prompt_tokens": 5243, "reused": 3773, "output_tokens": 64, "prompt_total": 5243, "prompt_read": 1470, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 3309.6, "decode_ms": 765.7, "decode_tok_s": 83.6, "hit_rate": 0.88, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 54, "drafts_accepted": 39}, {"projection": null, "time": 1791787504.4814332, "duration_s": 5.0, "finish": "length", "prompt_tokens": 3710, "reused": 630, "output_tokens": 64, "prompt_total": 3710, "prompt_read": 3080, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 4299.5, "decode_ms": 658.4, "decode_tok_s": 97.2, "hit_rate": 0.886, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 47, "drafts_accepted": 42}, {"projection": null, "time": 1791787502.2116232, "duration_s": 2.3, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 19, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 19, "prompt_ms": 1994.6, "decode_ms": 267.1, "decode_tok_s": 71.1, "hit_rate": 0.829, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 21, "drafts_accepted": 13}, {"projection": null, "time": 1791787497.5361862, "duration_s": 4.7, "finish": "stop", "prompt_tokens": 14441, "reused": 11595, "output_tokens": 25, "prompt_total": 14441, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 25, "prompt_ms": 4271.2, "decode_ms": 391.1, "decode_tok_s": 63.9, "hit_rate": 0.932, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 22, "drafts_accepted": 11}, {"projection": null, "time": 1791787492.9921672, "duration_s": 4.5, "finish": "stop", "prompt_tokens": 11575, "reused": 8715, "output_tokens": 21, "prompt_total": 11575, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 21, "prompt_ms": 4283.5, "decode_ms": 246.5, "decode_tok_s": 85.2, "hit_rate": 0.935, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 15, "drafts_accepted": 12}, {"projection": null, "time": 1791787487.8974795, "duration_s": 5.1, "finish": "stop", "prompt_tokens": 8687, "reused": 5307, "output_tokens": 29, "prompt_total": 8687, "prompt_read": 3380, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 29, "prompt_ms": 4688.4, "decode_ms": 393.2, "decode_tok_s": 73.8, "hit_rate": 0.889, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 19, "drafts_accepted": 14}, {"projection": null, "time": 1791787483.5927894, "duration_s": 4.3, "finish": "length", "prompt_tokens": 5243, "reused": 3773, "output_tokens": 64, "prompt_total": 5243, "prompt_read": 1470, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 3358.4, "decode_ms": 934.0, "decode_tok_s": 68.5, "hit_rate": 0.857, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 57, "drafts_accepted": 32}, {"projection": null, "time": 1791787478.414484, "duration_s": 5.2, "finish": "length", "prompt_tokens": 3710, "reused": 630, "output_tokens": 64, "prompt_total": 3710, "prompt_read": 3080, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 4428.1, "decode_ms": 739.1, "decode_tok_s": 86.6, "hit_rate": 0.723, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 48, "drafts_accepted": 41}, {"projection": null, "time": 1791787476.127828, "duration_s": 2.3, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 19, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 19, "prompt_ms": 2001.7, "decode_ms": 272.8, "decode_tok_s": 69.6, "hit_rate": 0.736, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 19, "drafts_accepted": 13}, {"projection": null, "time": 1791787464.3072362, "duration_s": 11.8, "finish": "length", "prompt_tokens": 16058, "reused": 0, "output_tokens": 24, "prompt_total": 16058, "prompt_read": 16058, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 11362.0, "decode_ms": 445.9, "decode_tok_s": 53.8, "hit_rate": 0.973, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 22, "drafts_accepted": 10}, {"projection": null, "time": 1791787452.486338, "duration_s": 11.8, "finish": "length", "prompt_tokens": 16058, "reused": 0, "output_tokens": 24, "prompt_total": 16058, "prompt_read": 16058, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 11391.9, "decode_ms": 414.1, "decode_tok_s": 58.0, "hit_rate": 0.954, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 24, "drafts_accepted": 12}, {"projection": null, "time": 1791787440.7137313, "duration_s": 11.8, "finish": "length", "prompt_tokens": 16058, "reused": 0, "output_tokens": 24, "prompt_total": 16058, "prompt_read": 16058, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 11331.5, "decode_ms": 425.2, "decode_tok_s": 56.4, "hit_rate": 0.904, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 25, "drafts_accepted": 12}, {"projection": null, "time": 1791787435.7596076, "duration_s": 4.9, "finish": "length", "prompt_tokens": 4019, "reused": 0, "output_tokens": 24, "prompt_total": 4019, "prompt_read": 4019, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 4614.2, "decode_ms": 325.0, "decode_tok_s": 73.8, "hit_rate": 0.926, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 13, "drafts_accepted": 11}, {"projection": null, "time": 1791787430.784213, "duration_s": 5.0, "finish": "length", "prompt_tokens": 4019, "reused": 0, "output_tokens": 24, "prompt_total": 4019, "prompt_read": 4019, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 4631.2, "decode_ms": 335.4, "decode_tok_s": 71.6, "hit_rate": 0.865, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 14, "drafts_accepted": 11}, {"projection": null, "time": 1791787425.8016279, "duration_s": 5.0, "finish": "length", "prompt_tokens": 4019, "reused": 0, "output_tokens": 24, "prompt_total": 4019, "prompt_read": 4019, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 4635.7, "decode_ms": 338.3, "decode_tok_s": 70.9, "hit_rate": 0.708, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 14, "drafts_accepted": 12}, {"projection": null, "time": 1791787424.027939, "duration_s": 1.8, "finish": "length", "prompt_tokens": 46, "reused": 0, "output_tokens": 100, "prompt_total": 46, "prompt_read": 46, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 367.9, "decode_ms": 1397.5, "decode_tok_s": 71.6, "hit_rate": 0.839, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 80, "drafts_accepted": 49}, {"projection": null, "time": 1791787422.4076836, "duration_s": 1.6, "finish": "length", "prompt_tokens": 43, "reused": 0, "output_tokens": 100, "prompt_total": 43, "prompt_read": 43, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 341.4, "decode_ms": 1272.3, "decode_tok_s": 78.6, "hit_rate": 0.835, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 74, "drafts_accepted": 53}, {"projection": null, "time": 1791787420.8739371, "duration_s": 1.5, "finish": "length", "prompt_tokens": 51, "reused": 0, "output_tokens": 100, "prompt_total": 51, "prompt_read": 51, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 415.4, "decode_ms": 1112.1, "decode_tok_s": 89.9, "hit_rate": 0.695, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 81, "drafts_accepted": 70}, {"projection": null, "time": 1791787419.1655023, "duration_s": 1.7, "finish": "length", "prompt_tokens": 46, "reused": 0, "output_tokens": 100, "prompt_total": 46, "prompt_read": 46, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 374.6, "decode_ms": 1327.5, "decode_tok_s": 75.3, "hit_rate": 0.824, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 70, "drafts_accepted": 51}, {"projection": null, "time": 1791787417.4558668, "duration_s": 1.7, "finish": "length", "prompt_tokens": 43, "reused": 0, "output_tokens": 100, "prompt_total": 43, "prompt_read": 43, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 343.1, "decode_ms": 1360.2, "decode_tok_s": 73.5, "hit_rate": 0.861, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 79, "drafts_accepted": 50}, {"projection": null, "time": 1791787415.97848, "duration_s": 1.5, "finish": "length", "prompt_tokens": 51, "reused": 0, "output_tokens": 100, "prompt_total": 51, "prompt_read": 51, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 417.2, "decode_ms": 1053.8, "decode_tok_s": 94.9, "hit_rate": 0.732, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 76, "drafts_accepted": 70}, {"projection": null, "time": 1791787414.1846852, "duration_s": 1.8, "finish": "length", "prompt_tokens": 46, "reused": 0, "output_tokens": 100, "prompt_total": 46, "prompt_read": 46, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 396.6, "decode_ms": 1390.8, "decode_tok_s": 71.9, "hit_rate": 0.766, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 70, "drafts_accepted": 50}, {"projection": null, "time": 1791787412.369226, "duration_s": 1.8, "finish": "length", "prompt_tokens": 43, "reused": 0, "output_tokens": 100, "prompt_total": 43, "prompt_read": 43, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 371.5, "decode_ms": 1436.9, "decode_tok_s": 69.6, "hit_rate": 0.787, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 81, "drafts_accepted": 48}, {"projection": null, "time": 1791787410.8280966, "duration_s": 1.5, "finish": "length", "prompt_tokens": 51, "reused": 0, "output_tokens": 100, "prompt_total": 51, "prompt_read": 51, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 424.0, "decode_ms": 1110.6, "decode_tok_s": 90.0, "hit_rate": 0.711, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 86, "drafts_accepted": 70}, {"projection": null, "time": 1791787408.6978765, "duration_s": 2.1, "finish": "length", "prompt_tokens": 1028, "reused": 0, "output_tokens": 8, "prompt_total": 1028, "prompt_read": 1028, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 8, "prompt_ms": 1984.3, "decode_ms": 139.5, "decode_tok_s": 57.3, "hit_rate": 0.699, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 7, "drafts_accepted": 3}, {"projection": null, "time": 1791787406.7341294, "duration_s": 2.0, "finish": "length", "prompt_tokens": 906, "reused": 0, "output_tokens": 1, "prompt_total": 906, "prompt_read": 906, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 1, "prompt_ms": 1938.2, "decode_ms": 18.7, "decode_tok_s": 53.5, "hit_rate": 0.769, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 0, "drafts_accepted": 0}, {"projection": null, "time": 1791787405.0124552, "duration_s": 1.7, "finish": "length", "prompt_tokens": 260, "reused": 0, "output_tokens": 1, "prompt_total": 260, "prompt_read": 260, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 1, "prompt_ms": 1694.1, "decode_ms": 19.0, "decode_tok_s": 52.6, "hit_rate": 0.774, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 0, "drafts_accepted": 0}, {"projection": null, "time": 1791787404.250783, "duration_s": 0.8, "finish": "length", "prompt_tokens": 34, "reused": 0, "output_tokens": 24, "prompt_total": 34, "prompt_read": 34, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 391.5, "decode_ms": 362.6, "decode_tok_s": 66.2, "hit_rate": 0.55, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 18, "drafts_accepted": 17}], "conversation_cache": {"enabled": false, "budget_mib": 0, "slots": 4, "parked": 0, "bytes": 0, "evictions": 0, "parks": 0, "restores": 0, "last_event": null, "last_tokens": null, "last_at": null, "requests": 40, "requests_reused": 15, "reused_tokens": 90134, "prompt_tokens": 291895, "last_reused": 0, "last_prompt": 32049}, "requests_kept": 40, "totals": {"since": 1791787339.2396867, "requests": 40, "prompt_tokens": 291895, "reused": 90134, "output_tokens": 1857, "prompt_ms": 181215.1, "decode_ms": 24287.800000000003, "drafts_offered": 1459, "drafts_accepted": 1048}, "hardware": {"gpus": [{"index": 0, "util": 0, "mem_used": 16836001792, "mem_total": 17103323136, "temp": 51, "power": 24.135}, {"index": 1, "util": 100, "mem_used": 12494372864, "mem_total": 12884901888, "temp": 55, "power": 67.196}], "gpu_util": 50.0, "gpu_mem_used": 29330374656, "gpu_mem_total": 29988225024, "gpu_temp": 55, "gpu_power": 91.331, "gpu_power_limit": 350.0, "gpu_pcie_gen": 5, "gpu_pcie_gen_max": 5, "gpu_pcie_width": 8, "gpu_pcie_rx_mb": 3297.955078125, "gpu_pcie_tx_mb": 378.87109375, "cpu": 4.2, "ram_used": 65279717376, "ram_total": 66472062976, "disk_read_mb": 0.24886409264777362, "disk_write_mb": 0.08656142352966038, "tok_s": 0.0, "tok_s_mean": 0, "prefill_tok_s_mean": 1900.9}, "hardware_static": {"gpu_name": "NVIDIA GeForce RTX 5060 Ti + NVIDIA GeForce RTX 3060", "gpu_count": 2, "gpu_note": null, "cpu_name": "AMD Ryzen 9 7900X 12-Core Processor", "cores": 12, "threads": 24, "psutil": true}, "history": {"gpu_util": [50.0, 50.0, 91.0, 48.0, 50.0, 67.0, 50.0, 46.0, 50.0, 50.0, 50.0, 100.0, 100.0, 50.0, 97.5, 99.0, 100.0, 50.0, 100.0, 99.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 99.5, 100.0, 50.0, 100.0, 99.0, 78.0, 50.0, 99.5, 100.0, 58.5, 50.0, 50.0, 50.0, 52.0, 50.0, 50.0, 50.0, 99.0, 99.5, 50.0, 100.0, 100.0, 99.5, 50.0, 100.0, 100.0, 50.0, 50.0, 50.0, 50.0, 50.0], "gpu_mem_used": [29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656], "gpu_temp": [64, 52, 54, 57, 63, 55, 53, 55, 65, 65, 67, 64, 66, 68, 65, 67, 68, 70, 70, 69, 68, 57, 57, 59, 54, 60, 67, 65, 68, 66, 67, 71, 69, 68, 69, 70, 68, 70, 72, 55, 56, 55, 57, 67, 68, 69, 62, 70, 69, 63, 69, 70, 71, 71, 68, 71, 58, 60, 60, 55], "gpu_power": [181.64, 140.06, 123.9, 88.25, 180.0, 177.5, 130.95, 87.06, 178.5, 196.06, 185.23, 178.23, 260.26, 256.22, 185.88, 249.31, 264.12, 246.8, 210.29, 252.96, 252.87, 172.77, 157.56, 143.19, 85.29, 122.54, 193.54, 186.71, 184.4, 214.46, 279.67, 207.06, 223.73, 263.74, 267.89, 207.95, 264.01, 261.99, 216.03, 130.17, 134.61, 127.48, 92.42, 177.59, 194.88, 186.24, 183.29, 261.86, 255.92, 191.92, 241.25, 263.37, 253.22, 209.17, 253.5, 254.34, 174.06, 157.54, 141.84, 91.33], "gpu_pcie_rx_mb": [6055.36, 3325.86, 3307.73, 29460.82, 29444.0, 3291.75, 3308.54, 62.36, 28573.81, 54.18, 29662.75, 19415.08, 24439.45, 455.07, 3360.43, 16708.96, 25213.78, 3954.16, 3364.08, 16.54, 16664.22, 3293.41, 3297.45, 3290.45, 3301.77, 1.05, 31043.86, 1.33, 3366.13, 3325.69, 16181.83, 29548.87, 26222.15, 21324.24, 8943.9, 3366.41, 3310.11, 64.71, 3292.2, 3313.45, 62.48, 3293.06, 253.09, 49.44, 2.21, 29616.0, 20686.93, 3347.81, 29492.57, 3303.69, 3359.54, 63.52, 1.93, 33332.15, 32957.79, 28972.5, 3294.69, 3306.88, 3294.04, 3297.96], "cpu": [4.3, 4.2, 4.2, 18.1, 4.2, 4.2, 4.9, 20.0, 6.8, 4.2, 4.2, 4.4, 4.2, 4.2, 4.3, 4.3, 4.2, 4.2, 4.3, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 22.3, 4.4, 7.5, 8.8, 9.0, 7.4, 4.2, 4.2, 4.3, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.3, 4.2, 18.1, 11.4, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2], "ram_used": [65470955520, 65470955520, 65569230848, 65568976896, 65568976896, 65568976896, 65541070848, 65541468160, 64969449472, 64969449472, 64969498624, 64969469952, 64969703424, 64969703424, 65071108096, 65071108096, 65083920384, 65083920384, 65083920384, 65084166144, 65084166144, 65084166144, 65084166144, 65182441472, 65182441472, 65006723072, 65005289472, 65014317056, 65032679424, 65039966208, 65042518016, 65048014848, 65146290176, 65147326464, 65156988928, 65155878912, 65155170304, 65153826816, 65153826816, 65153826816, 65153814528, 65250283520, 65250279424, 65060577280, 65060577280, 65060806656, 65060806656, 65061052416, 65061879808, 65160523776, 65160519680, 65183490048, 65183490048, 65183490048, 65183346688, 65183346688, 65183346688, 65183346688, 65279815680, 65279717376], "disk_read_mb": [0.17, 0.16, 0.14, 3.0, 0.14, 0.16, 0.18, 0.49, 5.94, 0.16, 9.22, 82.94, 0.16, 0.16, 29.68, 0.16, 0.93, 0.16, 0.16, 0.17, 0.16, 0.16, 0.16, 0.16, 0.16, 3.9, 94.65, 199.82, 13.29, 3.9, 1.54, 8.08, 0.16, 0.16, 0.16, 0.17, 0.16, 0.14, 0.19, 0.16, 0.16, 0.16, 0.26, 0.27, 0.16, 0.16, 0.27, 0.16, 0.16, 0.16, 0.14, 0.16, 0.16, 0.16, 1.85, 0.16, 0.14, 0.16, 0.16, 0.25], "tok_s": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 66.62, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 47.54, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "prefill_tok_s_mean": [null, null, 902.1, null, null, null, 889.8, 0.0, null, null, null, null, null, 1395.6, 1395.6, 1395.6, 1395.6, 1675.5, 1675.5, 1675.5, 1826.0, 1826.0, 1826.0, 1900.2, 1900.2, null, null, null, null, null, 1396.2, 1396.2, 1396.2, 1396.2, 1675.1, 1675.1, 1675.1, 1675.1, 1824.4, 1824.4, 1824.4, 1900.3, 0.0, null, null, null, null, null, 1394.9, 1394.9, 1394.9, 1394.9, 1675.5, 1675.5, 1675.5, 1825.8, 1825.8, 1825.8, 1900.9, 1900.9]}, "time": 1791787610.231949} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/C-after.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/C-after.json new file mode 100644 index 00000000000..572fa7a6757 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/C-after.json @@ -0,0 +1 @@ +{"engine": {"model": "qwen3.8-flash-next-iq3_s", "max_context": 524288, "images": true, "version": "0.1.43", "logit_bias": 1, "context": 524288, "kv": "int8", "kv_resident": 20480, "expert_slots": 8842, "expert_cache_mib": 17910, "expert_slots_primary": 4984, "expert_cache_primary_mib": 9474, "spec": 6, "mtp_max": 4, "lookup": 3, "vram_free_mib": 324, "cvec": 0, "arena_mib": 47962, "pool_workers": 11, "pcie_frac": "0.00", "spec_min_p": "0.70", "conversation_cache_mib": 0, "conversation_cache_slots": 4, "conversation_cache_min_free_mib": 2560, "conversation_cache_min_tokens": 0, "tail_role_token": -1, "vram_elastic": 0, "engine": "0.1.43"}, "live": {"state": "idle", "queued": 0, "phase": null, "prompt_tokens": null, "prompt_read": null, "prompt_total": null, "generated": null, "max_tokens": null, "elapsed_s": null, "tok_s": null, "tok_s_mean": null, "prefill_tok_s_mean": null, "tok_s_window_s": null}, "requests": [{"projection": null, "time": 1791787591.178125, "duration_s": 19.0, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 18593.1, "decode_ms": 411.4, "decode_tok_s": 58.3, "hit_rate": 0.953, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 19, "drafts_accepted": 10}, {"projection": null, "time": 1791787572.1203792, "duration_s": 19.0, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 18598.5, "decode_ms": 435.2, "decode_tok_s": 55.1, "hit_rate": 0.898, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 24, "drafts_accepted": 12}, {"projection": null, "time": 1791787553.131049, "duration_s": 19.0, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 18601.0, "decode_ms": 362.8, "decode_tok_s": 66.2, "hit_rate": 0.791, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 15, "drafts_accepted": 12}, {"projection": null, "time": 1791787548.5094302, "duration_s": 4.6, "finish": "stop", "prompt_tokens": 14460, "reused": 11614, "output_tokens": 24, "prompt_total": 14460, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 4221.6, "decode_ms": 376.6, "decode_tok_s": 63.7, "hit_rate": 0.925, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 19, "drafts_accepted": 10}, {"projection": null, "time": 1791787544.0395448, "duration_s": 4.5, "finish": "stop", "prompt_tokens": 11587, "reused": 8727, "output_tokens": 28, "prompt_total": 11587, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 28, "prompt_ms": 4189.1, "decode_ms": 268.1, "decode_tok_s": 104.4, "hit_rate": 0.945, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 19, "drafts_accepted": 19}, {"projection": null, "time": 1791787539.0323, "duration_s": 5.0, "finish": "stop", "prompt_tokens": 8687, "reused": 5306, "output_tokens": 41, "prompt_total": 8687, "prompt_read": 3381, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 41, "prompt_ms": 4503.9, "decode_ms": 491.9, "decode_tok_s": 83.4, "hit_rate": 0.916, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 27, "drafts_accepted": 23}, {"projection": null, "time": 1791787534.9737628, "duration_s": 4.1, "finish": "length", "prompt_tokens": 5243, "reused": 3774, "output_tokens": 64, "prompt_total": 5243, "prompt_read": 1469, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 3264.9, "decode_ms": 783.7, "decode_tok_s": 81.7, "hit_rate": 0.889, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 53, "drafts_accepted": 37}, {"projection": null, "time": 1791787529.9899266, "duration_s": 5.0, "finish": "length", "prompt_tokens": 3710, "reused": 630, "output_tokens": 64, "prompt_total": 3710, "prompt_read": 3080, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 4283.5, "decode_ms": 691.4, "decode_tok_s": 92.6, "hit_rate": 0.865, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 48, "drafts_accepted": 42}, {"projection": null, "time": 1791787527.716457, "duration_s": 2.3, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 19, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 19, "prompt_ms": 1999.2, "decode_ms": 266.8, "decode_tok_s": 71.2, "hit_rate": 0.792, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 20, "drafts_accepted": 13}, {"projection": null, "time": 1791787523.1084292, "duration_s": 4.6, "finish": "stop", "prompt_tokens": 14465, "reused": 11619, "output_tokens": 24, "prompt_total": 14465, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 4188.9, "decode_ms": 408.4, "decode_tok_s": 58.8, "hit_rate": 0.928, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 22, "drafts_accepted": 9}, {"projection": null, "time": 1791787518.6444337, "duration_s": 4.5, "finish": "stop", "prompt_tokens": 11594, "reused": 8734, "output_tokens": 26, "prompt_total": 11594, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 26, "prompt_ms": 4177.8, "decode_ms": 272.4, "decode_tok_s": 95.4, "hit_rate": 0.916, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 20, "drafts_accepted": 17}, {"projection": null, "time": 1791787513.5341477, "duration_s": 5.1, "finish": "stop", "prompt_tokens": 8687, "reused": 5307, "output_tokens": 48, "prompt_total": 8687, "prompt_read": 3380, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 48, "prompt_ms": 4532.3, "decode_ms": 565.8, "decode_tok_s": 84.8, "hit_rate": 0.919, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 37, "drafts_accepted": 28}], "conversation_cache": {"enabled": false, "budget_mib": 0, "slots": 4, "parked": 0, "bytes": 0, "evictions": 0, "parks": 0, "restores": 0, "last_event": null, "last_tokens": null, "last_at": null, "requests": 40, "requests_reused": 15, "reused_tokens": 90134, "prompt_tokens": 291895, "last_reused": 0, "last_prompt": 32049}, "requests_kept": 40, "totals": {"since": 1791787339.2396867, "requests": 40, "prompt_tokens": 291895, "reused": 90134, "output_tokens": 1857, "prompt_ms": 181215.1, "decode_ms": 24287.800000000003, "drafts_offered": 1459, "drafts_accepted": 1048}, "hardware": {"gpus": [{"index": 0, "util": 0, "mem_used": 16836001792, "mem_total": 17103323136, "temp": 51, "power": 24.135}, {"index": 1, "util": 100, "mem_used": 12494372864, "mem_total": 12884901888, "temp": 55, "power": 67.196}], "gpu_util": 50.0, "gpu_mem_used": 29330374656, "gpu_mem_total": 29988225024, "gpu_temp": 55, "gpu_power": 91.331, "gpu_power_limit": 350.0, "gpu_pcie_gen": 5, "gpu_pcie_gen_max": 5, "gpu_pcie_width": 8, "gpu_pcie_rx_mb": 3297.955078125, "gpu_pcie_tx_mb": 378.87109375, "cpu": 4.2, "ram_used": 65279717376, "ram_total": 66472062976, "disk_read_mb": 0.24886409264777362, "disk_write_mb": 0.08656142352966038, "tok_s": 0.0, "tok_s_mean": 0, "prefill_tok_s_mean": 1900.9}, "hardware_static": {"gpu_name": "NVIDIA GeForce RTX 5060 Ti + NVIDIA GeForce RTX 3060", "gpu_count": 2, "gpu_note": null, "cpu_name": "AMD Ryzen 9 7900X 12-Core Processor", "cores": 12, "threads": 24, "psutil": true}, "history": {"gpu_util": [50.0, 50.0, 91.0, 48.0, 50.0, 67.0, 50.0, 46.0, 50.0, 50.0, 50.0, 100.0, 100.0, 50.0, 97.5, 99.0, 100.0, 50.0, 100.0, 99.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 99.5, 100.0, 50.0, 100.0, 99.0, 78.0, 50.0, 99.5, 100.0, 58.5, 50.0, 50.0, 50.0, 52.0, 50.0, 50.0, 50.0, 99.0, 99.5, 50.0, 100.0, 100.0, 99.5, 50.0, 100.0, 100.0, 50.0, 50.0, 50.0, 50.0, 50.0], "gpu_mem_used": [29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29328277504, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656, 29330374656], "gpu_temp": [64, 52, 54, 57, 63, 55, 53, 55, 65, 65, 67, 64, 66, 68, 65, 67, 68, 70, 70, 69, 68, 57, 57, 59, 54, 60, 67, 65, 68, 66, 67, 71, 69, 68, 69, 70, 68, 70, 72, 55, 56, 55, 57, 67, 68, 69, 62, 70, 69, 63, 69, 70, 71, 71, 68, 71, 58, 60, 60, 55], "gpu_power": [181.64, 140.06, 123.9, 88.25, 180.0, 177.5, 130.95, 87.06, 178.5, 196.06, 185.23, 178.23, 260.26, 256.22, 185.88, 249.31, 264.12, 246.8, 210.29, 252.96, 252.87, 172.77, 157.56, 143.19, 85.29, 122.54, 193.54, 186.71, 184.4, 214.46, 279.67, 207.06, 223.73, 263.74, 267.89, 207.95, 264.01, 261.99, 216.03, 130.17, 134.61, 127.48, 92.42, 177.59, 194.88, 186.24, 183.29, 261.86, 255.92, 191.92, 241.25, 263.37, 253.22, 209.17, 253.5, 254.34, 174.06, 157.54, 141.84, 91.33], "gpu_pcie_rx_mb": [6055.36, 3325.86, 3307.73, 29460.82, 29444.0, 3291.75, 3308.54, 62.36, 28573.81, 54.18, 29662.75, 19415.08, 24439.45, 455.07, 3360.43, 16708.96, 25213.78, 3954.16, 3364.08, 16.54, 16664.22, 3293.41, 3297.45, 3290.45, 3301.77, 1.05, 31043.86, 1.33, 3366.13, 3325.69, 16181.83, 29548.87, 26222.15, 21324.24, 8943.9, 3366.41, 3310.11, 64.71, 3292.2, 3313.45, 62.48, 3293.06, 253.09, 49.44, 2.21, 29616.0, 20686.93, 3347.81, 29492.57, 3303.69, 3359.54, 63.52, 1.93, 33332.15, 32957.79, 28972.5, 3294.69, 3306.88, 3294.04, 3297.96], "cpu": [4.3, 4.2, 4.2, 18.1, 4.2, 4.2, 4.9, 20.0, 6.8, 4.2, 4.2, 4.4, 4.2, 4.2, 4.3, 4.3, 4.2, 4.2, 4.3, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 22.3, 4.4, 7.5, 8.8, 9.0, 7.4, 4.2, 4.2, 4.3, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.3, 4.2, 18.1, 11.4, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2], "ram_used": [65470955520, 65470955520, 65569230848, 65568976896, 65568976896, 65568976896, 65541070848, 65541468160, 64969449472, 64969449472, 64969498624, 64969469952, 64969703424, 64969703424, 65071108096, 65071108096, 65083920384, 65083920384, 65083920384, 65084166144, 65084166144, 65084166144, 65084166144, 65182441472, 65182441472, 65006723072, 65005289472, 65014317056, 65032679424, 65039966208, 65042518016, 65048014848, 65146290176, 65147326464, 65156988928, 65155878912, 65155170304, 65153826816, 65153826816, 65153826816, 65153814528, 65250283520, 65250279424, 65060577280, 65060577280, 65060806656, 65060806656, 65061052416, 65061879808, 65160523776, 65160519680, 65183490048, 65183490048, 65183490048, 65183346688, 65183346688, 65183346688, 65183346688, 65279815680, 65279717376], "disk_read_mb": [0.17, 0.16, 0.14, 3.0, 0.14, 0.16, 0.18, 0.49, 5.94, 0.16, 9.22, 82.94, 0.16, 0.16, 29.68, 0.16, 0.93, 0.16, 0.16, 0.17, 0.16, 0.16, 0.16, 0.16, 0.16, 3.9, 94.65, 199.82, 13.29, 3.9, 1.54, 8.08, 0.16, 0.16, 0.16, 0.17, 0.16, 0.14, 0.19, 0.16, 0.16, 0.16, 0.26, 0.27, 0.16, 0.16, 0.27, 0.16, 0.16, 0.16, 0.14, 0.16, 0.16, 0.16, 1.85, 0.16, 0.14, 0.16, 0.16, 0.25], "tok_s": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 66.62, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 47.54, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "prefill_tok_s_mean": [null, null, 902.1, null, null, null, 889.8, 0.0, null, null, null, null, null, 1395.6, 1395.6, 1395.6, 1395.6, 1675.5, 1675.5, 1675.5, 1826.0, 1826.0, 1826.0, 1900.2, 1900.2, null, null, null, null, null, 1396.2, 1396.2, 1396.2, 1396.2, 1675.1, 1675.1, 1675.1, 1675.1, 1824.4, 1824.4, 1824.4, 1900.3, 0.0, null, null, null, null, null, 1394.9, 1394.9, 1394.9, 1394.9, 1675.5, 1675.5, 1675.5, 1825.8, 1825.8, 1825.8, 1900.9, 1900.9]}, "time": 1791787610.2448392} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/C-before.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/C-before.json new file mode 100644 index 00000000000..ea02ee7de45 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/C-before.json @@ -0,0 +1 @@ +{"engine": {"model": "qwen3.8-flash-next-iq3_s", "max_context": 524288, "images": true, "version": "0.1.43", "logit_bias": 1, "context": 524288, "kv": "int8", "kv_resident": 20480, "expert_slots": 8842, "expert_cache_mib": 17910, "expert_slots_primary": 4984, "expert_cache_primary_mib": 9474, "spec": 6, "mtp_max": 4, "lookup": 3, "vram_free_mib": 324, "cvec": 0, "arena_mib": 47962, "pool_workers": 11, "pcie_frac": "0.00", "spec_min_p": "0.70", "conversation_cache_mib": 0, "conversation_cache_slots": 4, "conversation_cache_min_free_mib": 2560, "conversation_cache_min_tokens": 0, "tail_role_token": -1, "vram_elastic": 0, "engine": "0.1.43"}, "live": {"state": "idle", "queued": 0, "phase": null, "prompt_tokens": null, "prompt_read": null, "prompt_total": null, "generated": null, "max_tokens": null, "elapsed_s": null, "tok_s": null, "tok_s_mean": null, "prefill_tok_s_mean": null, "tok_s_window_s": null}, "requests": [], "conversation_cache": {"enabled": false, "budget_mib": 0, "slots": 4, "parked": 0, "bytes": 0, "evictions": 0, "parks": 0, "restores": 0, "last_event": null, "last_tokens": null, "last_at": null, "requests": 0, "requests_reused": 0, "reused_tokens": 0, "prompt_tokens": 0, "last_reused": null, "last_prompt": null}, "requests_kept": 0, "totals": {"since": 1791787339.2396867, "requests": 0, "prompt_tokens": 0, "reused": 0, "output_tokens": 0, "prompt_ms": 0.0, "decode_ms": 0.0, "drafts_offered": 0, "drafts_accepted": 0}, "hardware": {"gpus": [{"index": 0, "util": 0, "mem_used": 16762601472, "mem_total": 17103323136, "temp": 37, "power": 4.595}, {"index": 1, "util": 0, "mem_used": 12427264000, "mem_total": 12884901888, "temp": 44, "power": 7.773}], "gpu_util": 0.0, "gpu_mem_used": 29189865472, "gpu_mem_total": 29988225024, "gpu_temp": 44, "gpu_power": 12.367999999999999, "gpu_power_limit": 350.0, "gpu_pcie_gen": 1, "gpu_pcie_gen_max": 5, "gpu_pcie_width": 8, "gpu_pcie_rx_mb": 0.943359375, "gpu_pcie_tx_mb": 1.3251953125, "cpu": 0.0, "ram_used": 66141569024, "ram_total": 66472062976, "disk_read_mb": 0.0, "disk_write_mb": 0.0, "tok_s": 0.0, "tok_s_mean": 0, "prefill_tok_s_mean": 0.0}, "hardware_static": {"gpu_name": "NVIDIA GeForce RTX 5060 Ti + NVIDIA GeForce RTX 3060", "gpu_count": 2, "gpu_note": null, "cpu_name": "AMD Ryzen 9 7900X 12-Core Processor", "cores": 12, "threads": 24, "psutil": true}, "history": {"gpu_util": [2.5, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "gpu_mem_used": [29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472], "gpu_temp": [45, 45, 45, 45, 46, 45, 46, 46, 46, 46, 46, 46, 46, 46, 47, 47, 47, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 44, 44, 44, 44, 44, 44, 44, 44, 44, 44, 44, 44, 44, 44, 44, 44], "gpu_power": [57.04, 55.62, 54.8, 54.78, 54.78, 54.64, 54.71, 54.77, 54.72, 54.64, 54.67, 54.69, 54.75, 54.84, 54.83, 54.76, 46.91, 18.01, 14.92, 12.79, 12.47, 12.46, 12.54, 12.57, 12.46, 12.48, 12.51, 12.49, 12.48, 12.42, 12.43, 12.47, 12.42, 12.42, 12.46, 12.64, 12.5, 12.43, 12.42, 12.67, 12.54, 12.46, 12.44, 12.41, 12.55, 12.37], "gpu_pcie_rx_mb": [0.83, 0.88, 0.79, 0.8, 0.78, 0.8, 0.79, 0.87, 0.79, 0.78, 0.84, 0.81, 0.8, 0.87, 0.79, 0.99, 1.14, 1.01, 0.99, 0.95, 0.99, 0.99, 0.94, 0.97, 1.01, 0.94, 1.0, 0.96, 0.94, 0.94, 1.04, 1.01, 0.93, 1.1, 0.95, 0.96, 1.0, 0.95, 1.01, 1.01, 1.0, 1.0, 0.94, 0.94, 0.93, 0.94], "cpu": [0.0, 2.8, 0.2, 0.0, 0.1, 0.0, 0.1, 0.1, 0.0, 0.1, 0.1, 0.1, 0.0, 0.0, 0.1, 0.1, 0.0, 0.1, 0.1, 0.1, 0.2, 0.0, 0.1, 0.1, 0.0, 0.1, 0.1, 0.1, 0.0, 0.1, 0.1, 0.1, 0.1, 0.1, 0.0, 0.1, 0.1, 0.0, 0.1, 0.1, 0.1, 0.1, 0.1, 0.0, 0.2, 0.0], "ram_used": [66116673536, 66123706368, 66124840960, 66125254656, 66126217216, 66137378816, 66137653248, 66138316800, 66138177536, 66138103808, 66145427456, 66143367168, 66143518720, 66142314496, 66142203904, 66140897280, 66140897280, 66139533312, 66138337280, 66138337280, 66137292800, 66137550848, 66137550848, 66137550848, 66136866816, 66137124864, 66136326144, 66136555520, 66136408064, 66136895488, 66136895488, 66137124864, 66141319168, 66141319168, 66141319168, 66141319168, 66141319168, 66141319168, 66141573120, 66141573120, 66141573120, 66141569024, 66141569024, 66141569024, 66141569024, 66141569024], "disk_read_mb": [null, 1.0, 0.23, 0.19, 14.55, 66.51, 6.22, 0.21, 0.25, 0.17, 14.82, 0.2, 0.34, 0.17, 0.17, 0.17, 0.17, 0.29, 0.19, 0.17, 0.16, 0.18, 0.16, 0.17, 0.16, 0.14, 0.0, 0.02, 0.01, 0.01, 0.01, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.21, 0.11, 0.0, 0.0, 0.0, 0.0], "tok_s": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "prefill_tok_s_mean": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}, "time": 1791787388.9345667} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/README.md b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/README.md new file mode 100644 index 00000000000..f128466a15b --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/README.md @@ -0,0 +1,112 @@ +# Community benchmark C: `STRATA_ADAPT_OVERLAP=0` on a 2-GPU layer split (IQ3_S) + +Measured on 2026-10-11 by @mmbeanz. Targets the live upstream debate: +**#2010** (Linux CUDA, -21% decode), **#1975** (RTX 5090, IQ3_S -26%), **#1964** (4070 Ti, +IQ3_XXS) and **PR #2011** (make the 0.1.43 default opt-in everywhere). Every regression +reported so far is on a **single GPU**; this run measures the same flag on a **two-card +layer split with `--pcie-frac 0.00`**. + +**Headline.** On this topology the flag is **not a lever we can see**: the prompt path is +null to within run-to-run spread (**+0.14%** at 32K), and decode shows no significant +difference (permutation p = 0.39), which **rules out the -21%/-26% class of regressions +here** but cannot resolve effects smaller than roughly ±5-11% given per-request decode +variance. So the reported regressions do not generalize to unequal-card layer splits with +expert streaming disabled — the cost/benefit of `ADAPT_OVERLAP` is topology-dependent. + +## Setup: identical to run A except one environment variable + +Same daily-driver config as `../A/` (`--layer-split 40`, `--pcie-frac 0.00`, pinned +`data/expert-profile.bin`, no `--expert-profile-save`), same evening, cold boot, 0 requests +before the suite. Only difference: `STRATA_ADAPT_OVERLAP=0` (0.1.43's Linux CUDA default is +**on**). Engine facts confirm the layout is unchanged: `expert_slots 8842`, +`expert_slots_primary 4984`, `pcie_frac 0.00`, `pool_workers 11`, `vram_free_mib 324`. + +**Provenance caveat, stated plainly:** 0.1.43 does **not** log the overlap state — PR #2011 +adds logging "in both directions", which is exactly what would make this auditable. The env +var here is operator-asserted, not verifiable from the artifacts. Recommendation for future +runs (and worth saying in the thread): capture `env | grep STRATA_` at server start into the +results folder. + +Hardware/software/model/method are as hand-written in `../A/README.md` (7900X, +64 GB DDR5-6400, 5060 Ti PCIe 5.0 x8 @ 28.2-28.7 GB/s H2D + 3060 PCIe 3.0 x4 @ 3.15-3.2 +GB/s, nspawn on bare metal, Strata 0.1.43 from `7695859a` + `setup.py`-only local commit, +IQ3_S GSQ-RCO, 512K yarn 2.0). Bench driven remotely with +`strata_bench.py --url --long --runs 3`; `GET /metrics?requests=all` before/after +(`C-before.json`, `C-after-full.json`). Suite ~205 s. + +## Results — A (overlap ON, default) vs C (overlap OFF) + +| metric | A | C | delta | +| --- | ---: | ---: | ---: | +| prompt 32K tok/s (median of 3) | 1,720.8 (range 1720.5-1721.7) | 1,723.2 (1723.0-1723.7) | **+0.14%** | +| prompt 16K tok/s | 1,412.7 | 1,413.3 | +0.04% | +| prompt 4K tok/s | 868.1 | 867.8 | -0.03% | +| TTFT 32K (s) | 18.66 | 18.64 | -0.1% | +| short-chat decode (median, 9 runs) | 74.4 | 75.3 | +1.2% | +| agent decode (median, 15 turns) | 80.7 | 83.6 | +3.6% | +| expert hit_rate (median, 40 req) | 0.859 | 0.863 | +0.004 | +| draft acceptance | 68.9% | 71.8% | +2.9 pts | +| `pcie_share` / `ram_blobs` / `file_blobs` | 0 / 0 / 0 | 0 / 0 / 0 | identical | + +## Statistics (per-request values from `/metrics?requests=all`, n = 40 per run) + +Two-sample permutation tests, 6,000 shuffles, two-sided: + +| quantity | A | C | mean diff | p | +| --- | ---: | ---: | ---: | ---: | +| hit_rate | 0.8417 | 0.8394 | +0.0023 | **0.92** | +| decode tok/s (all requests) | 71.10 | 73.61 | -2.52 | **0.39** | +| decode at 32K prompt | 53.2 (med) | 58.3 (med) | -7.07 | 0.20 | +| decode at 16K prompt | 58.3 (med) | 56.4 (med) | +2.27 | 0.29 | +| decode at 4K prompt | 70.7 (med) | 71.6 (med) | -0.63 | 0.50 | +| prompt_ms at 32K | 18,624 ms | 18,599 ms | +25 ms | 0.10 | + +Nothing is significant. With 95% confidence intervals rather than p-values, the honest +resolution of this suite is: + +- **Prompt path: precisely null.** 32K prompt throughput differs by **+0.14%**, inside the + run-to-run spread of a single config (A's own 3 runs span 1720.5-1721.7). +- **Decode: imprecise.** mean difference -3.4% with 95% CI **[-11.2%, +4.4%]**. That is + enough to **rule out the -21% (#2010) and -26% (#1975) regressions on this topology**, + but not enough to claim the flag is exactly neutral. +- **Hit rate: null.** +0.0023, 95% CI [-0.039, +0.044]. + +Per-request decode varies enormously within a run (48-104 tok/s depending on context and +cache state), which is what limits the decode bound; a soak or a larger `--runs` would +tighten it. + +## Why a split might be immune (hypothesis, not measurement) + +Two structural differences from the single-GPU reporters: + +1. **Little to hide.** With `--pcie-frac 0.00` no expert is streamed on the miss path + (telemetry: `pcie_share = 0` on all 40 requests in both runs). The only large copies left + are the adaptive tier's own swaps: at the 0.1.43 defaults (96 swaps / 4 tokens, ~2 MB + blobs) that is on the order of ~48 MB per verify window — ~1.7 ms at 28 GB/s against a + ~43 ms decode window, i.e. ~4%. +2. **Already overlapped.** In a two-stage split, stage 0 works on the next window while + stage 1 finishes the current one, so some of the hiding that `ADAPT_OVERLAP` provides on + a single card is already happening structurally. + +Both are consistent with "no measurable effect either way here", but neither is proven by +this run. + +## What to post upstream + +A comment on **#2010** (per the contributing rule "add to a thread that already covers your +point"): *ADAPT_OVERLAP=0 vs default on a 5060 Ti + 3060 layer split, IQ3_S, +`--pcie-frac 0.00`: prompt path null to +0.14%, decode difference not significant +(p=0.39, 95% CI -11%..+4%), hit rate identical — the single-GPU regressions do not +reproduce on this topology.* Plus the small ask: 0.1.43 does not log the flag state, which +makes A/B runs unauditable; PR #2011's logging would fix that. + +## Limitations + +- Speed only; no correctness/needle check (needle suite: `../needles/`). +- Decode resolution ±~8% at 95% confidence (see CI above); prompt resolution ~0.2%. +- Env-var provenance not verifiable from artifacts (see caveat). +- One suite per config (3 runs), same evening as A; A is the same-day baseline. +- Permutation p-values treat requests as independent; the adaptive tier correlates + consecutive requests, so they are optimistic. +- Not tested: overlap on a split **with** `--pcie-frac > 0` (the case where copies are large + and the flag might matter), concurrency, vision, mmap tier, Windows/HIP. diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/config.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/config.json new file mode 100644 index 00000000000..37d4ef4599b --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/config.json @@ -0,0 +1,64 @@ +{ + "exe": "/home/llm/Strata/engine/strata", + "args": [ + "--pack", + "/home/llm/Strata-data/packs/iq3_s", + "--native", + "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf", + "--ple-gguf", + "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf", + "--expert-profile", + "/home/llm/Strata/data/expert-profile.bin", + "--expert-cache", + "auto", + "--prefill", + "auto", + "--spec", + "4", + "--mtp", + "/home/llm/Strata-data/mtp/rt", + "--max-context", + "524288", + "--rope-scaling", + "yarn", + "--rope-scale", + "2.0", + "--kv", + "int8", + "--kv-resident", + "20480", + "--vision", + "--remote-expert-opt", + "--trim-stage-weights", + "--pcie-frac", + "0.00", + "--spec-min-p", + "0.70" + ], + "cwd": "/home/llm/Strata", + "tokenizer": "/home/llm/Strata-data/packs/iq3_s/tokenizer", + "model_name": "qwen3.8-flash-next-iq3_s", + "log": "/home/llm/Strata/strata-iq3_s.log", + "lib_dirs": [ + "/opt/cuda/bin", + "/opt/cuda/lib64" + ], + "port": 8080, + "gpu": [ + 0, + 1 + ], + "gpus_asked": true, + "layer_split": "40", + "vision": { + "exe": "/home/llm/Strata/engine/strata-vision", + "mmproj": "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/mmproj-Qwen3.8-Flash-Next-BF16.gguf", + "model": "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf", + "gpu": true, + "max_tokens": 1024 + }, + "allowed_hosts": [ + "bertha" + ], + "draft_vocab": "en" +} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/results.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/results.json new file mode 100644 index 00000000000..4b61c1cd520 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/results.json @@ -0,0 +1,1018 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-11", + "params": { + "runs": 3, + "think": false, + "clients": 0, + "long": true, + "started_server": false, + "command": "python tools/strata_bench.py --long" + }, + "elapsed_s": 206.01516962051392, + "machine": { + "gpu": [ + "NVIDIA GeForce RTX 5060 Ti", + "NVIDIA GeForce RTX 3060" + ], + "gpu_vram_mib": 28599.0, + "gpu_driver": "615.71.09", + "cuda": "13.4 (nvcc V13.4.92)", + "cpu": "AMD Ryzen 9 7900X 12-Core Processor", + "cpu_cores": 12, + "cpu_threads": 24, + "ram_gib": 61.9, + "os": "Linux 7.2.9-1-cachyos", + "python": "3.14.7", + "strata_version": "0.1.43", + "strata_commit": "7695859a", + "engine_version": "0.1.43", + "model": "qwen3.8-flash-next-iq3_s", + "quant": "IQ3_S", + "context": 524288, + "parallel_slots": 1, + "config": { + "exe": "strata", + "args": [ + "--pack", + "iq3_s", + "--native", + "Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf", + "--ple-gguf", + "Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf", + "--expert-profile", + "expert-profile-live.bin", + "--expert-profile-save", + "expert-profile-live.bin", + "--expert-cache", + "auto", + "--prefill", + "auto", + "--spec", + "4", + "--mtp", + "rt", + "--max-context", + "524288", + "--rope-scaling", + "yarn", + "--rope-scale", + "2.0", + "--kv", + "int8", + "--kv-resident", + "20480", + "--vision", + "--remote-expert-opt", + "--trim-stage-weights", + "--pcie-frac", + "0.00", + "--spec-min-p", + "0.70" + ], + "model_name": "qwen3.8-flash-next-iq3_s", + "lib_dirs": [ + "/opt/cuda/bin", + "/opt/cuda/lib64" + ], + "port": 8080, + "gpu": [ + 0, + 1 + ], + "gpus_asked": true, + "layer_split": "40", + "vision": { + "exe": "/home//Strata/engine/strata-vision", + "mmproj": "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/mmproj-Qwen3.8-Flash-Next-BF16.gguf", + "model": "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf", + "gpu": true, + "max_tokens": 1024 + }, + "allowed_hosts": [ + "bertha" + ], + "draft_vocab": "en" + }, + "launch": "strata --pack iq3_s --native Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf --ple-gguf Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf --expert-profile expert-profile-live.bin --expert-profile-save expert-profile-live.bin --expert-cache auto --prefill auto --spec 4 --mtp rt --max-context 524288 --rope-scaling yarn --rope-scale 2.0 --kv int8 --kv-resident 20480 --vision --remote-expert-opt --trim-stage-weights --pcie-frac 0.00 --spec-min-p 0.70", + "engine_info": { + "model": "qwen3.8-flash-next-iq3_s", + "max_context": 524288, + "images": true, + "version": "0.1.43", + "logit_bias": 1, + "context": 524288, + "kv": "int8", + "kv_resident": 20480, + "expert_slots": 8842, + "expert_cache_mib": 17910, + "expert_slots_primary": 4984, + "expert_cache_primary_mib": 9474, + "spec": 6, + "mtp_max": 4, + "lookup": 3, + "vram_free_mib": 324, + "cvec": 0, + "arena_mib": 47962, + "pool_workers": 11, + "pcie_frac": "0.00", + "spec_min_p": "0.70", + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "conversation_cache_min_tokens": 0, + "tail_role_token": -1, + "vram_elastic": 0, + "engine": "0.1.43" + }, + "vram_after": { + "elastic": false + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.448, + "total_s": 1.541, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 120.3, + "prompt_tps_source": "engine", + "decode_tps": 90.0, + "decode_tps_source": "engine", + "prompt_ms": 424.0, + "decode_ms": 1110.6, + "drafts_offered": 86, + "drafts_accepted": 70, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.396, + "total_s": 1.815, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 115.7, + "prompt_tps_source": "engine", + "decode_tps": 69.6, + "decode_tps_source": "engine", + "prompt_ms": 371.5, + "decode_ms": 1436.9, + "drafts_offered": 81, + "drafts_accepted": 48, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.421, + "total_s": 1.794, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 116.0, + "prompt_tps_source": "engine", + "decode_tps": 71.9, + "decode_tps_source": "engine", + "prompt_ms": 396.6, + "decode_ms": 1390.8, + "drafts_offered": 70, + "drafts_accepted": 50, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.441, + "total_s": 1.477, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 122.2, + "prompt_tps_source": "engine", + "decode_tps": 94.9, + "decode_tps_source": "engine", + "prompt_ms": 417.2, + "decode_ms": 1053.8, + "drafts_offered": 76, + "drafts_accepted": 70, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.367, + "total_s": 1.709, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 125.3, + "prompt_tps_source": "engine", + "decode_tps": 73.5, + "decode_tps_source": "engine", + "prompt_ms": 343.1, + "decode_ms": 1360.2, + "drafts_offered": 79, + "drafts_accepted": 50, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.398, + "total_s": 1.708, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 122.8, + "prompt_tps_source": "engine", + "decode_tps": 75.3, + "decode_tps_source": "engine", + "prompt_ms": 374.6, + "decode_ms": 1327.5, + "drafts_offered": 70, + "drafts_accepted": 51, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.439, + "total_s": 1.534, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 122.8, + "prompt_tps_source": "engine", + "decode_tps": 89.9, + "decode_tps_source": "engine", + "prompt_ms": 415.4, + "decode_ms": 1112.1, + "drafts_offered": 81, + "drafts_accepted": 70, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.365, + "total_s": 1.62, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 126.0, + "prompt_tps_source": "engine", + "decode_tps": 78.6, + "decode_tps_source": "engine", + "prompt_ms": 341.4, + "decode_ms": 1272.3, + "drafts_offered": 74, + "drafts_accepted": 53, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.392, + "total_s": 1.772, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 125.0, + "prompt_tps_source": "engine", + "decode_tps": 71.6, + "decode_tps_source": "engine", + "prompt_ms": 367.9, + "decode_ms": 1397.5, + "drafts_offered": 80, + "drafts_accepted": 49, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 75.3, + "min": 69.6, + "max": 94.9, + "n": 9 + }, + "ttft_s": { + "median": 0.398, + "min": 0.365, + "max": 0.448, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 46, + "min": 43, + "max": 51, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 4.662, + "total_s": 4.982, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 867.0, + "prompt_tps_source": "engine", + "decode_tps": 70.9, + "decode_tps_source": "engine", + "prompt_ms": 4635.7, + "decode_ms": 338.3, + "drafts_offered": 14, + "drafts_accepted": 12, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.657, + "total_s": 4.974, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 867.8, + "prompt_tps_source": "engine", + "decode_tps": 71.6, + "decode_tps_source": "engine", + "prompt_ms": 4631.2, + "decode_ms": 335.4, + "drafts_offered": 14, + "drafts_accepted": 11, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.641, + "total_s": 4.947, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 871.0, + "prompt_tps_source": "engine", + "decode_tps": 73.8, + "decode_tps_source": "engine", + "prompt_ms": 4614.2, + "decode_ms": 325.0, + "drafts_offered": 13, + "drafts_accepted": 11, + "run": 2 + } + ], + "prompt_tps": { + "median": 867.8, + "min": 867.0, + "max": 871.0, + "n": 3 + }, + "ttft_s": { + "median": 4.657, + "min": 4.641, + "max": 4.662, + "n": 3 + }, + "prompt_tokens": { + "median": 4019, + "min": 4019, + "max": 4019, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 11.363, + "total_s": 11.77, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1417.1, + "prompt_tps_source": "engine", + "decode_tps": 56.4, + "decode_tps_source": "engine", + "prompt_ms": 11331.5, + "decode_ms": 425.2, + "drafts_offered": 25, + "drafts_accepted": 12, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 11.423, + "total_s": 11.819, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1409.6, + "prompt_tps_source": "engine", + "decode_tps": 58.0, + "decode_tps_source": "engine", + "prompt_ms": 11391.9, + "decode_ms": 414.1, + "drafts_offered": 24, + "drafts_accepted": 12, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 11.392, + "total_s": 11.82, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1413.3, + "prompt_tps_source": "engine", + "decode_tps": 53.8, + "decode_tps_source": "engine", + "prompt_ms": 11362.0, + "decode_ms": 445.9, + "drafts_offered": 22, + "drafts_accepted": 10, + "run": 2 + } + ], + "prompt_tps": { + "median": 1413.3, + "min": 1409.6, + "max": 1417.1, + "n": 3 + }, + "ttft_s": { + "median": 11.392, + "min": 11.363, + "max": 11.423, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "32K": { + "target_tokens": 32768, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 18.639, + "total_s": 18.983, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 1723.0, + "prompt_tps_source": "engine", + "decode_tps": 66.2, + "decode_tps_source": "engine", + "prompt_ms": 18601.0, + "decode_ms": 362.8, + "drafts_offered": 15, + "drafts_accepted": 12, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 18.637, + "total_s": 19.054, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 1723.2, + "prompt_tps_source": "engine", + "decode_tps": 55.1, + "decode_tps_source": "engine", + "prompt_ms": 18598.5, + "decode_ms": 435.2, + "drafts_offered": 24, + "drafts_accepted": 12, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 18.63, + "total_s": 19.023, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 1723.7, + "prompt_tps_source": "engine", + "decode_tps": 58.3, + "decode_tps_source": "engine", + "prompt_ms": 18593.1, + "decode_ms": 411.4, + "drafts_offered": 19, + "drafts_accepted": 10, + "run": 2 + } + ], + "prompt_tps": { + "median": 1723.2, + "min": 1723.0, + "max": 1723.7, + "n": 3 + }, + "ttft_s": { + "median": 18.637, + "min": 18.63, + "max": 18.639, + "n": 3 + }, + "prompt_tokens": { + "median": 32049, + "min": 32049, + "max": 32049, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 2.028, + "total_s": 2.283, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 19, + "prompt_tps": 304.2, + "prompt_tps_source": "engine", + "decode_tps": 69.6, + "decode_tps_source": "engine", + "prompt_ms": 2001.7, + "decode_ms": 272.8, + "drafts_offered": 19, + "drafts_accepted": 13, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.459, + "total_s": 5.179, + "prompt_tokens": 3710, + "reused_tokens": 630, + "read_tokens": 3080, + "generated_tokens": 64, + "prompt_tps": 695.6, + "prompt_tps_source": "engine", + "decode_tps": 86.6, + "decode_tps_source": "engine", + "prompt_ms": 4428.1, + "decode_ms": 739.1, + "drafts_offered": 48, + "drafts_accepted": 41, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.388, + "total_s": 4.304, + "prompt_tokens": 5243, + "reused_tokens": 3773, + "read_tokens": 1470, + "generated_tokens": 64, + "prompt_tps": 437.7, + "prompt_tps_source": "engine", + "decode_tps": 68.5, + "decode_tps_source": "engine", + "prompt_ms": 3358.4, + "decode_ms": 934.0, + "drafts_offered": 57, + "drafts_accepted": 32, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.719, + "total_s": 5.094, + "prompt_tokens": 8687, + "reused_tokens": 5307, + "read_tokens": 3380, + "generated_tokens": 29, + "prompt_tps": 720.9, + "prompt_tps_source": "engine", + "decode_tps": 73.8, + "decode_tps_source": "engine", + "prompt_ms": 4688.4, + "decode_ms": 393.2, + "drafts_offered": 19, + "drafts_accepted": 14, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.316, + "total_s": 4.544, + "prompt_tokens": 11575, + "reused_tokens": 8715, + "read_tokens": 2860, + "generated_tokens": 21, + "prompt_tps": 667.7, + "prompt_tps_source": "engine", + "decode_tps": 85.2, + "decode_tps_source": "engine", + "prompt_ms": 4283.5, + "decode_ms": 246.5, + "drafts_offered": 15, + "drafts_accepted": 12, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.305, + "total_s": 4.678, + "prompt_tokens": 14441, + "reused_tokens": 11595, + "read_tokens": 2846, + "generated_tokens": 25, + "prompt_tps": 666.3, + "prompt_tps_source": "engine", + "decode_tps": 63.9, + "decode_tps_source": "engine", + "prompt_ms": 4271.2, + "decode_ms": 391.1, + "drafts_offered": 22, + "drafts_accepted": 11, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 2.019, + "total_s": 2.269, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 19, + "prompt_tps": 305.3, + "prompt_tps_source": "engine", + "decode_tps": 71.1, + "decode_tps_source": "engine", + "prompt_ms": 1994.6, + "decode_ms": 267.1, + "drafts_offered": 21, + "drafts_accepted": 13, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.326, + "total_s": 4.966, + "prompt_tokens": 3710, + "reused_tokens": 630, + "read_tokens": 3080, + "generated_tokens": 64, + "prompt_tps": 716.4, + "prompt_tps_source": "engine", + "decode_tps": 97.2, + "decode_tps_source": "engine", + "prompt_ms": 4299.5, + "decode_ms": 658.4, + "drafts_offered": 47, + "drafts_accepted": 42, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.337, + "total_s": 4.084, + "prompt_tokens": 5243, + "reused_tokens": 3773, + "read_tokens": 1470, + "generated_tokens": 64, + "prompt_tps": 444.2, + "prompt_tps_source": "engine", + "decode_tps": 83.6, + "decode_tps_source": "engine", + "prompt_ms": 3309.6, + "decode_ms": 765.7, + "drafts_offered": 54, + "drafts_accepted": 39, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.561, + "total_s": 5.109, + "prompt_tokens": 8687, + "reused_tokens": 5307, + "read_tokens": 3380, + "generated_tokens": 48, + "prompt_tps": 745.8, + "prompt_tps_source": "engine", + "decode_tps": 84.8, + "decode_tps_source": "engine", + "prompt_ms": 4532.3, + "decode_ms": 565.8, + "drafts_offered": 37, + "drafts_accepted": 28, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.209, + "total_s": 4.463, + "prompt_tokens": 11594, + "reused_tokens": 8734, + "read_tokens": 2860, + "generated_tokens": 26, + "prompt_tps": 684.6, + "prompt_tps_source": "engine", + "decode_tps": 95.4, + "decode_tps_source": "engine", + "prompt_ms": 4177.8, + "decode_ms": 272.4, + "drafts_offered": 20, + "drafts_accepted": 17, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.221, + "total_s": 4.611, + "prompt_tokens": 14465, + "reused_tokens": 11619, + "read_tokens": 2846, + "generated_tokens": 24, + "prompt_tps": 679.4, + "prompt_tps_source": "engine", + "decode_tps": 58.8, + "decode_tps_source": "engine", + "prompt_ms": 4188.9, + "decode_ms": 408.4, + "drafts_offered": 22, + "drafts_accepted": 9, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 2.023, + "total_s": 2.272, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 19, + "prompt_tps": 304.6, + "prompt_tps_source": "engine", + "decode_tps": 71.2, + "decode_tps_source": "engine", + "prompt_ms": 1999.2, + "decode_ms": 266.8, + "drafts_offered": 20, + "drafts_accepted": 13, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.309, + "total_s": 4.983, + "prompt_tokens": 3710, + "reused_tokens": 630, + "read_tokens": 3080, + "generated_tokens": 64, + "prompt_tps": 719.0, + "prompt_tps_source": "engine", + "decode_tps": 92.6, + "decode_tps_source": "engine", + "prompt_ms": 4283.5, + "decode_ms": 691.4, + "drafts_offered": 48, + "drafts_accepted": 42, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.292, + "total_s": 4.058, + "prompt_tokens": 5243, + "reused_tokens": 3774, + "read_tokens": 1469, + "generated_tokens": 64, + "prompt_tps": 449.9, + "prompt_tps_source": "engine", + "decode_tps": 81.7, + "decode_tps_source": "engine", + "prompt_ms": 3264.9, + "decode_ms": 783.7, + "drafts_offered": 53, + "drafts_accepted": 37, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.532, + "total_s": 5.006, + "prompt_tokens": 8687, + "reused_tokens": 5306, + "read_tokens": 3381, + "generated_tokens": 41, + "prompt_tps": 750.7, + "prompt_tps_source": "engine", + "decode_tps": 83.4, + "decode_tps_source": "engine", + "prompt_ms": 4503.9, + "decode_ms": 491.9, + "drafts_offered": 27, + "drafts_accepted": 23, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.219, + "total_s": 4.469, + "prompt_tokens": 11587, + "reused_tokens": 8727, + "read_tokens": 2860, + "generated_tokens": 28, + "prompt_tps": 682.7, + "prompt_tps_source": "engine", + "decode_tps": 104.4, + "decode_tps_source": "engine", + "prompt_ms": 4189.1, + "decode_ms": 268.1, + "drafts_offered": 19, + "drafts_accepted": 19, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.253, + "total_s": 4.611, + "prompt_tokens": 14460, + "reused_tokens": 11614, + "read_tokens": 2846, + "generated_tokens": 24, + "prompt_tps": 674.2, + "prompt_tps_source": "engine", + "decode_tps": 63.7, + "decode_tps_source": "engine", + "prompt_ms": 4221.6, + "decode_ms": 376.6, + "drafts_offered": 19, + "drafts_accepted": 10, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 68.787, + "min": 68.765, + "max": 68.796, + "n": 3 + }, + "reread_tokens": { + "median": 3, + "min": 3, + "max": 3, + "n": 3 + }, + "new_tokens": { + "median": 13633, + "min": 13633, + "max": 13633, + "n": 3 + }, + "turn_ttft_s": { + "median": 4.305, + "min": 3.292, + "max": 4.719, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 14460, + "min": 14441, + "max": 14465, + "n": 3 + }, + "turn_decode_tps": { + "median": 83.6, + "min": 58.8, + "max": 104.4, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 2.023, + "min": 2.019, + "max": 2.028, + "n": 3 + } + }, + "clients": {}, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/D-after-full.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/D-after-full.json new file mode 100644 index 00000000000..b88ddff469b --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/D-after-full.json @@ -0,0 +1 @@ +{"engine": {"model": "qwen3.8-flash-next-iq3_s", "max_context": 524288, "images": true, "version": "0.1.43", "logit_bias": 1, "context": 524288, "kv": "int8", "kv_resident": 20480, "expert_slots": 8842, "expert_cache_mib": 17910, "expert_slots_primary": 4984, "expert_cache_primary_mib": 9474, "spec": 6, "mtp_max": 4, "lookup": 3, "vram_free_mib": 324, "cvec": 0, "arena_mib": 47962, "pool_workers": 11, "pcie_frac": "0.00", "spec_min_p": "0.70", "conversation_cache_mib": 0, "conversation_cache_slots": 4, "conversation_cache_min_free_mib": 2560, "conversation_cache_min_tokens": 0, "tail_role_token": -1, "vram_elastic": 0, "engine": "0.1.43"}, "live": {"state": "idle", "queued": 0, "phase": null, "prompt_tokens": null, "prompt_read": null, "prompt_total": null, "generated": null, "max_tokens": null, "elapsed_s": null, "tok_s": null, "tok_s_mean": null, "prefill_tok_s_mean": null, "tok_s_window_s": null}, "requests": [{"projection": null, "time": 1791788712.3789334, "duration_s": 19.1, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 18641.5, "decode_ms": 416.8, "decode_tok_s": 57.6, "hit_rate": 0.927, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 20, "drafts_accepted": 12}, {"projection": null, "time": 1791788693.2908297, "duration_s": 19.1, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 18603.6, "decode_ms": 459.3, "decode_tok_s": 52.3, "hit_rate": 0.863, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 21, "drafts_accepted": 10}, {"projection": null, "time": 1791788674.1490877, "duration_s": 19.1, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 18589.0, "decode_ms": 527.1, "decode_tok_s": 45.5, "hit_rate": 0.738, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 23, "drafts_accepted": 10}, {"projection": null, "time": 1791788669.542707, "duration_s": 4.6, "finish": "stop", "prompt_tokens": 14451, "reused": 11605, "output_tokens": 21, "prompt_total": 14451, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 21, "prompt_ms": 4236.6, "decode_ms": 345.6, "decode_tok_s": 60.8, "hit_rate": 0.856, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 20, "drafts_accepted": 9}, {"projection": null, "time": 1791788665.0265403, "duration_s": 4.5, "finish": "stop", "prompt_tokens": 11584, "reused": 8724, "output_tokens": 22, "prompt_total": 11584, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 22, "prompt_ms": 4238.5, "decode_ms": 264.1, "decode_tok_s": 83.3, "hit_rate": 0.91, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 18, "drafts_accepted": 13}, {"projection": null, "time": 1791788659.9198847, "duration_s": 5.1, "finish": "stop", "prompt_tokens": 8686, "reused": 5306, "output_tokens": 39, "prompt_total": 8686, "prompt_read": 3380, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 39, "prompt_ms": 4570.3, "decode_ms": 524.3, "decode_tok_s": 74.4, "hit_rate": 0.87, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 33, "drafts_accepted": 21}, {"projection": null, "time": 1791788655.7222373, "duration_s": 4.2, "finish": "length", "prompt_tokens": 5242, "reused": 3772, "output_tokens": 64, "prompt_total": 5242, "prompt_read": 1470, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 3299.7, "decode_ms": 887.6, "decode_tok_s": 72.1, "hit_rate": 0.827, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 61, "drafts_accepted": 37}, {"projection": null, "time": 1791788650.7081132, "duration_s": 5.0, "finish": "length", "prompt_tokens": 3709, "reused": 628, "output_tokens": 64, "prompt_total": 3709, "prompt_read": 3081, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 4263.1, "decode_ms": 742.8, "decode_tok_s": 86.2, "hit_rate": 0.851, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 50, "drafts_accepted": 40}, {"projection": null, "time": 1791788648.3910105, "duration_s": 2.3, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 18, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 18, "prompt_ms": 2012.6, "decode_ms": 297.0, "decode_tok_s": 60.6, "hit_rate": 0.737, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 22, "drafts_accepted": 11}, {"projection": null, "time": 1791788643.8934813, "duration_s": 4.5, "finish": "stop", "prompt_tokens": 14469, "reused": 11623, "output_tokens": 15, "prompt_total": 14469, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 15, "prompt_ms": 4230.1, "decode_ms": 256.2, "decode_tok_s": 58.5, "hit_rate": 0.875, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 14, "drafts_accepted": 6}, {"projection": null, "time": 1791788639.380019, "duration_s": 4.5, "finish": "stop", "prompt_tokens": 11598, "reused": 8738, "output_tokens": 26, "prompt_total": 11598, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 26, "prompt_ms": 4224.0, "decode_ms": 274.7, "decode_tok_s": 94.6, "hit_rate": 0.898, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 18, "drafts_accepted": 17}, {"projection": null, "time": 1791788634.0151885, "duration_s": 5.4, "finish": "stop", "prompt_tokens": 8692, "reused": 5241, "output_tokens": 47, "prompt_total": 8692, "prompt_read": 3451, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 47, "prompt_ms": 4629.0, "decode_ms": 723.7, "decode_tok_s": 64.9, "hit_rate": 0.874, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 44, "drafts_accepted": 22}, {"projection": null, "time": 1791788629.881107, "duration_s": 4.1, "finish": "length", "prompt_tokens": 5248, "reused": 3778, "output_tokens": 64, "prompt_total": 5248, "prompt_read": 1470, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 3281.3, "decode_ms": 842.2, "decode_tok_s": 76.0, "hit_rate": 0.825, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 55, "drafts_accepted": 39}, {"projection": null, "time": 1791788624.861889, "duration_s": 5.0, "finish": "length", "prompt_tokens": 3715, "reused": 635, "output_tokens": 64, "prompt_total": 3715, "prompt_read": 3080, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 4305.3, "decode_ms": 705.4, "decode_tok_s": 90.7, "hit_rate": 0.847, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 47, "drafts_accepted": 41}, {"projection": null, "time": 1791788622.5321174, "duration_s": 2.3, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 24, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 2004.5, "decode_ms": 317.7, "decode_tok_s": 75.5, "hit_rate": 0.762, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 22, "drafts_accepted": 17}, {"projection": null, "time": 1791788617.779921, "duration_s": 4.7, "finish": "stop", "prompt_tokens": 14468, "reused": 11622, "output_tokens": 33, "prompt_total": 14468, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 33, "prompt_ms": 4235.0, "decode_ms": 506.3, "decode_tok_s": 65.2, "hit_rate": 0.865, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 36, "drafts_accepted": 17}, {"projection": null, "time": 1791788613.2631962, "duration_s": 4.5, "finish": "stop", "prompt_tokens": 11602, "reused": 8742, "output_tokens": 21, "prompt_total": 11602, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 21, "prompt_ms": 4260.8, "decode_ms": 238.6, "decode_tok_s": 88.0, "hit_rate": 0.904, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 20, "drafts_accepted": 14}, {"projection": null, "time": 1791788607.8756454, "duration_s": 5.4, "finish": "stop", "prompt_tokens": 8689, "reused": 5308, "output_tokens": 54, "prompt_total": 8689, "prompt_read": 3381, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 54, "prompt_ms": 4642.0, "decode_ms": 732.6, "decode_tok_s": 73.7, "hit_rate": 0.86, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 40, "drafts_accepted": 28}, {"projection": null, "time": 1791788603.6801581, "duration_s": 4.2, "finish": "length", "prompt_tokens": 5245, "reused": 3775, "output_tokens": 64, "prompt_total": 5245, "prompt_read": 1470, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 3341.1, "decode_ms": 842.5, "decode_tok_s": 76.0, "hit_rate": 0.806, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 51, "drafts_accepted": 36}, {"projection": null, "time": 1791788598.4303672, "duration_s": 5.2, "finish": "length", "prompt_tokens": 3712, "reused": 631, "output_tokens": 64, "prompt_total": 3712, "prompt_read": 3081, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 4427.1, "decode_ms": 811.9, "decode_tok_s": 78.8, "hit_rate": 0.663, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 47, "drafts_accepted": 41}, {"projection": null, "time": 1791788596.077813, "duration_s": 2.3, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 21, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 21, "prompt_ms": 1993.7, "decode_ms": 348.2, "decode_tok_s": 60.3, "hit_rate": 0.701, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 24, "drafts_accepted": 13}, {"projection": null, "time": 1791788584.2396493, "duration_s": 11.8, "finish": "length", "prompt_tokens": 16058, "reused": 0, "output_tokens": 24, "prompt_total": 16058, "prompt_read": 16058, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 11396.2, "decode_ms": 429.7, "decode_tok_s": 55.9, "hit_rate": 0.937, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 19, "drafts_accepted": 9}, {"projection": null, "time": 1791788572.395241, "duration_s": 11.8, "finish": "length", "prompt_tokens": 16058, "reused": 0, "output_tokens": 24, "prompt_total": 16058, "prompt_read": 16058, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 11380.0, "decode_ms": 448.7, "decode_tok_s": 53.5, "hit_rate": 0.947, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 23, "drafts_accepted": 10}, {"projection": null, "time": 1791788560.564103, "duration_s": 11.8, "finish": "length", "prompt_tokens": 16058, "reused": 0, "output_tokens": 24, "prompt_total": 16058, "prompt_read": 16058, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 11348.1, "decode_ms": 467.8, "decode_tok_s": 51.3, "hit_rate": 0.905, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 23, "drafts_accepted": 10}, {"projection": null, "time": 1791788555.5166695, "duration_s": 5.0, "finish": "length", "prompt_tokens": 4019, "reused": 0, "output_tokens": 24, "prompt_total": 4019, "prompt_read": 4019, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 4607.9, "decode_ms": 424.4, "decode_tok_s": 56.6, "hit_rate": 0.828, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 23, "drafts_accepted": 12}, {"projection": null, "time": 1791788550.5395792, "duration_s": 5.0, "finish": "length", "prompt_tokens": 4019, "reused": 0, "output_tokens": 24, "prompt_total": 4019, "prompt_read": 4019, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 4613.4, "decode_ms": 354.7, "decode_tok_s": 67.7, "hit_rate": 0.789, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 14, "drafts_accepted": 11}, {"projection": null, "time": 1791788545.487269, "duration_s": 5.0, "finish": "length", "prompt_tokens": 4019, "reused": 0, "output_tokens": 24, "prompt_total": 4019, "prompt_read": 4019, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 4657.1, "decode_ms": 386.4, "decode_tok_s": 62.1, "hit_rate": 0.639, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 13, "drafts_accepted": 11}, {"projection": null, "time": 1791788543.5769916, "duration_s": 1.9, "finish": "length", "prompt_tokens": 46, "reused": 0, "output_tokens": 100, "prompt_total": 46, "prompt_read": 46, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 411.6, "decode_ms": 1490.4, "decode_tok_s": 67.1, "hit_rate": 0.769, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 75, "drafts_accepted": 49}, {"projection": null, "time": 1791788541.8149881, "duration_s": 1.8, "finish": "length", "prompt_tokens": 43, "reused": 0, "output_tokens": 100, "prompt_total": 43, "prompt_read": 43, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 375.8, "decode_ms": 1380.0, "decode_tok_s": 72.5, "hit_rate": 0.79, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 75, "drafts_accepted": 53}, {"projection": null, "time": 1791788540.1167061, "duration_s": 1.7, "finish": "length", "prompt_tokens": 51, "reused": 0, "output_tokens": 100, "prompt_total": 51, "prompt_read": 51, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 463.3, "decode_ms": 1228.8, "decode_tok_s": 81.4, "hit_rate": 0.644, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 79, "drafts_accepted": 70}, {"projection": null, "time": 1791788538.3008997, "duration_s": 1.8, "finish": "length", "prompt_tokens": 46, "reused": 0, "output_tokens": 100, "prompt_total": 46, "prompt_read": 46, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 420.8, "decode_ms": 1388.8, "decode_tok_s": 72.0, "hit_rate": 0.747, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 75, "drafts_accepted": 54}, {"projection": null, "time": 1791788536.3733196, "duration_s": 1.9, "finish": "length", "prompt_tokens": 43, "reused": 0, "output_tokens": 100, "prompt_total": 43, "prompt_read": 43, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 382.4, "decode_ms": 1539.1, "decode_tok_s": 65.0, "hit_rate": 0.807, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 77, "drafts_accepted": 45}, {"projection": null, "time": 1791788534.69093, "duration_s": 1.7, "finish": "length", "prompt_tokens": 51, "reused": 0, "output_tokens": 100, "prompt_total": 51, "prompt_read": 51, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 461.2, "decode_ms": 1214.8, "decode_tok_s": 82.3, "hit_rate": 0.648, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 77, "drafts_accepted": 70}, {"projection": null, "time": 1791788532.7720346, "duration_s": 1.9, "finish": "length", "prompt_tokens": 46, "reused": 0, "output_tokens": 100, "prompt_total": 46, "prompt_read": 46, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 442.0, "decode_ms": 1470.7, "decode_tok_s": 68.0, "hit_rate": 0.694, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 75, "drafts_accepted": 53}, {"projection": null, "time": 1791788530.7873933, "duration_s": 2.0, "finish": "length", "prompt_tokens": 43, "reused": 0, "output_tokens": 100, "prompt_total": 43, "prompt_read": 43, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 418.9, "decode_ms": 1559.3, "decode_tok_s": 64.1, "hit_rate": 0.709, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 84, "drafts_accepted": 50}, {"projection": null, "time": 1791788529.0386772, "duration_s": 1.7, "finish": "length", "prompt_tokens": 51, "reused": 0, "output_tokens": 100, "prompt_total": 51, "prompt_read": 51, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 100, "prompt_ms": 474.0, "decode_ms": 1268.2, "decode_tok_s": 78.9, "hit_rate": 0.627, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 83, "drafts_accepted": 70}, {"projection": null, "time": 1791788526.8912263, "duration_s": 2.1, "finish": "length", "prompt_tokens": 1028, "reused": 0, "output_tokens": 8, "prompt_total": 1028, "prompt_read": 1028, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 8, "prompt_ms": 1987.5, "decode_ms": 153.2, "decode_tok_s": 52.2, "hit_rate": 0.635, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 7, "drafts_accepted": 3}, {"projection": null, "time": 1791788524.9199712, "duration_s": 2.0, "finish": "length", "prompt_tokens": 906, "reused": 0, "output_tokens": 1, "prompt_total": 906, "prompt_read": 906, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 1, "prompt_ms": 1944.9, "decode_ms": 19.7, "decode_tok_s": 50.8, "hit_rate": 0.673, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 0, "drafts_accepted": 0}, {"projection": null, "time": 1791788523.5194087, "duration_s": 1.4, "finish": "length", "prompt_tokens": 260, "reused": 0, "output_tokens": 1, "prompt_total": 260, "prompt_read": 260, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 1, "prompt_ms": 1373.6, "decode_ms": 19.9, "decode_tok_s": 50.3, "hit_rate": 0.683, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 0, "drafts_accepted": 0}, {"projection": null, "time": 1791788522.656761, "duration_s": 0.9, "finish": "length", "prompt_tokens": 34, "reused": 0, "output_tokens": 24, "prompt_total": 34, "prompt_read": 34, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 431.2, "decode_ms": 424.6, "decode_tok_s": 56.5, "hit_rate": 0.48, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 18, "drafts_accepted": 17}], "conversation_cache": {"enabled": false, "budget_mib": 0, "slots": 4, "parked": 0, "bytes": 0, "evictions": 0, "parks": 0, "restores": 0, "last_event": null, "last_tokens": null, "last_at": null, "requests": 40, "requests_reused": 15, "reused_tokens": 90128, "prompt_tokens": 291963, "last_reused": 0, "last_prompt": 32049}, "requests_kept": 40, "totals": {"since": 1791788499.317125, "requests": 40, "prompt_tokens": 291963, "reused": 90128, "output_tokens": 1875, "prompt_ms": 181618.70000000004, "decode_ms": 26733.799999999992, "drafts_offered": 1526, "drafts_accepted": 1051}, "hardware": {"gpus": [{"index": 0, "util": 0, "mem_used": 16836001792, "mem_total": 17103323136, "temp": 50, "power": 25.689}, {"index": 1, "util": 100, "mem_used": 12498567168, "mem_total": 12884901888, "temp": 55, "power": 46.777}], "gpu_util": 50.0, "gpu_mem_used": 29334568960, "gpu_mem_total": 29988225024, "gpu_temp": 55, "gpu_power": 72.46600000000001, "gpu_power_limit": 350.0, "gpu_pcie_gen": 5, "gpu_pcie_gen_max": 5, "gpu_pcie_width": 8, "gpu_pcie_rx_mb": 3296.087890625, "gpu_pcie_tx_mb": 377.41015625, "cpu": 4.2, "ram_used": 65821134848, "ram_total": 66472062976, "disk_read_mb": 0.17310337465208495, "disk_write_mb": 0.3462067493041699, "tok_s": 0.0, "tok_s_mean": 0, "prefill_tok_s_mean": 1895.6}, "hardware_static": {"gpu_name": "NVIDIA GeForce RTX 5060 Ti + NVIDIA GeForce RTX 3060", "gpu_count": 2, "gpu_note": null, "cpu_name": "AMD Ryzen 9 7900X 12-Core Processor", "cores": 12, "threads": 24, "psutil": true}, "history": {"gpu_util": [50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 51.5, 50.0, 50.0, 50.0, 99.5, 99.5, 50.0, 99.0, 99.0, 50.0, 100.0, 99.5, 100.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 100.0, 100.0, 100.0, 50.0, 100.0, 98.5, 50.0, 99.5, 100.0, 100.0, 50.0, 50.0, 50.0, 50.0, 55.0, 50.0, 50.0, 50.0, 99.5, 99.5, 50.0, 100.0, 99.0, 100.0, 50.0, 100.0, 100.0, 50.0, 50.0, 50.0, 65.0, 50.0], "gpu_mem_used": [29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960], "gpu_temp": [64, 56, 53, 63, 64, 55, 53, 54, 64, 67, 67, 67, 68, 68, 65, 68, 69, 70, 68, 68, 70, 57, 57, 54, 53, 62, 66, 66, 67, 69, 70, 69, 69, 68, 69, 68, 70, 69, 70, 59, 60, 55, 55, 66, 68, 69, 69, 69, 69, 68, 68, 68, 71, 72, 71, 70, 57, 57, 56, 55], "gpu_power": [191.81, 126.33, 117.88, 159.27, 190.66, 128.41, 119.38, 105.66, 177.54, 184.29, 182.9, 219.57, 267.98, 238.14, 195.88, 258.61, 257.6, 185.15, 245.55, 262.58, 251.37, 144.68, 142.33, 120.99, 72.39, 130.59, 192.0, 184.95, 181.49, 270.77, 276.52, 208.72, 219.72, 264.33, 250.04, 209.05, 265.26, 262.33, 217.45, 156.22, 142.1, 88.03, 92.6, 174.41, 193.64, 184.13, 212.25, 271.09, 256.54, 191.13, 253.42, 262.89, 209.89, 230.22, 257.04, 255.74, 201.09, 153.29, 128.65, 72.47], "gpu_pcie_rx_mb": [3305.35, 257.88, 3292.96, 18893.09, 26970.17, 3313.07, 3292.38, 29784.47, 9.92, 1.76, 3329.47, 3308.21, 31368.28, 2860.18, 3316.38, 3307.34, 41.64, 4652.46, 28997.04, 23321.12, 3406.87, 3314.79, 62.82, 3291.67, 214.25, 4416.06, 1.31, 17834.11, 3316.01, 27260.92, 26588.14, 4797.63, 3332.33, 62.52, 2.34, 3312.65, 22083.04, 29740.8, 3311.93, 3319.45, 31320.76, 3295.91, 29920.97, 27746.8, 11003.02, 29857.36, 3315.46, 63.2, 23877.75, 14541.87, 34661.71, 21404.31, 3324.21, 24615.74, 108.05, 40.0, 3292.46, 3180.99, 3289.33, 3296.09], "cpu": [4.2, 4.2, 4.5, 18.1, 4.2, 4.2, 5.2, 21.3, 4.2, 4.2, 4.2, 4.4, 4.2, 4.2, 4.3, 4.3, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.3, 4.2, 15.8, 18.1, 4.2, 4.3, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.3, 4.2, 4.1, 4.3, 4.2, 4.2, 4.2, 4.3, 26.5, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2], "ram_used": [66000932864, 66001178624, 66090168320, 66096078848, 66096496640, 66096754688, 66070831104, 65502691328, 65503133696, 65503133696, 65503133696, 65511768064, 65511768064, 65511768064, 65616625664, 65616625664, 65639755776, 65640267776, 65641074688, 65641074688, 65641562112, 65641562112, 65641562112, 65738031104, 65738031104, 65561264128, 65561677824, 65565036544, 65565036544, 65565282304, 65565282304, 65672269824, 65672269824, 65691914240, 65692565504, 65692491776, 65692491776, 65692385280, 65690836992, 65688686592, 65785413632, 65785634816, 65609682944, 65610424320, 65610424320, 65614798848, 65615044608, 65615044608, 65615044608, 65711935488, 65709961216, 65720979456, 65721380864, 65721380864, 65724284928, 65724149760, 65724149760, 65724149760, 65820618752, 65821134848], "disk_read_mb": [0.16, 0.14, 0.16, 15.12, 0.16, 0.17, 0.16, 7.07, 0.16, 0.16, 0.16, 82.94, 0.16, 0.14, 29.67, 0.16, 7.85, 0.16, 0.16, 0.16, 0.15, 0.16, 0.16, 0.16, 0.16, 0.79, 11.51, 10.02, 0.16, 0.16, 0.16, 48.36, 0.16, 5.86, 0.16, 0.18, 0.16, 0.15, 0.18, 8.32, 0.26, 0.16, 0.5, 0.16, 0.16, 0.16, 0.17, 0.16, 0.14, 0.16, 5.07, 0.16, 0.17, 0.16, 0.19, 0.23, 0.16, 0.16, 0.16, 0.17], "tok_s": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 36.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "prefill_tok_s_mean": [null, null, 894.8, null, null, null, 890.8, null, null, null, null, null, null, 1394.2, 1394.2, 1394.2, 1675.9, 1675.9, 1675.9, 1675.9, 1828.0, 1828.0, 1828.0, 1902.2, 0.0, null, null, null, null, null, 1395.0, 1395.0, 1395.0, 1395.0, 1674.6, 1674.6, 1674.6, 1826.0, 1826.0, 1826.0, 1900.0, 1900.0, null, null, null, null, null, null, 1393.9, 1393.9, 1393.9, 1673.7, 1673.7, 1673.7, 1673.7, 1823.1, 1823.1, 1823.1, 1895.6, 1895.6]}, "time": 1791788731.4873133} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/D-after.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/D-after.json new file mode 100644 index 00000000000..9ab474e5ff9 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/D-after.json @@ -0,0 +1 @@ +{"engine": {"model": "qwen3.8-flash-next-iq3_s", "max_context": 524288, "images": true, "version": "0.1.43", "logit_bias": 1, "context": 524288, "kv": "int8", "kv_resident": 20480, "expert_slots": 8842, "expert_cache_mib": 17910, "expert_slots_primary": 4984, "expert_cache_primary_mib": 9474, "spec": 6, "mtp_max": 4, "lookup": 3, "vram_free_mib": 324, "cvec": 0, "arena_mib": 47962, "pool_workers": 11, "pcie_frac": "0.00", "spec_min_p": "0.70", "conversation_cache_mib": 0, "conversation_cache_slots": 4, "conversation_cache_min_free_mib": 2560, "conversation_cache_min_tokens": 0, "tail_role_token": -1, "vram_elastic": 0, "engine": "0.1.43"}, "live": {"state": "idle", "queued": 0, "phase": null, "prompt_tokens": null, "prompt_read": null, "prompt_total": null, "generated": null, "max_tokens": null, "elapsed_s": null, "tok_s": null, "tok_s_mean": null, "prefill_tok_s_mean": null, "tok_s_window_s": null}, "requests": [{"projection": null, "time": 1791788712.3789334, "duration_s": 19.1, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 18641.5, "decode_ms": 416.8, "decode_tok_s": 57.6, "hit_rate": 0.927, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 20, "drafts_accepted": 12}, {"projection": null, "time": 1791788693.2908297, "duration_s": 19.1, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 18603.6, "decode_ms": 459.3, "decode_tok_s": 52.3, "hit_rate": 0.863, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 21, "drafts_accepted": 10}, {"projection": null, "time": 1791788674.1490877, "duration_s": 19.1, "finish": "length", "prompt_tokens": 32049, "reused": 0, "output_tokens": 24, "prompt_total": 32049, "prompt_read": 32049, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 24, "prompt_ms": 18589.0, "decode_ms": 527.1, "decode_tok_s": 45.5, "hit_rate": 0.738, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 23, "drafts_accepted": 10}, {"projection": null, "time": 1791788669.542707, "duration_s": 4.6, "finish": "stop", "prompt_tokens": 14451, "reused": 11605, "output_tokens": 21, "prompt_total": 14451, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 21, "prompt_ms": 4236.6, "decode_ms": 345.6, "decode_tok_s": 60.8, "hit_rate": 0.856, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 20, "drafts_accepted": 9}, {"projection": null, "time": 1791788665.0265403, "duration_s": 4.5, "finish": "stop", "prompt_tokens": 11584, "reused": 8724, "output_tokens": 22, "prompt_total": 11584, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 22, "prompt_ms": 4238.5, "decode_ms": 264.1, "decode_tok_s": 83.3, "hit_rate": 0.91, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 18, "drafts_accepted": 13}, {"projection": null, "time": 1791788659.9198847, "duration_s": 5.1, "finish": "stop", "prompt_tokens": 8686, "reused": 5306, "output_tokens": 39, "prompt_total": 8686, "prompt_read": 3380, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 39, "prompt_ms": 4570.3, "decode_ms": 524.3, "decode_tok_s": 74.4, "hit_rate": 0.87, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 33, "drafts_accepted": 21}, {"projection": null, "time": 1791788655.7222373, "duration_s": 4.2, "finish": "length", "prompt_tokens": 5242, "reused": 3772, "output_tokens": 64, "prompt_total": 5242, "prompt_read": 1470, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 3299.7, "decode_ms": 887.6, "decode_tok_s": 72.1, "hit_rate": 0.827, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 61, "drafts_accepted": 37}, {"projection": null, "time": 1791788650.7081132, "duration_s": 5.0, "finish": "length", "prompt_tokens": 3709, "reused": 628, "output_tokens": 64, "prompt_total": 3709, "prompt_read": 3081, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 64, "prompt_ms": 4263.1, "decode_ms": 742.8, "decode_tok_s": 86.2, "hit_rate": 0.851, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 50, "drafts_accepted": 40}, {"projection": null, "time": 1791788648.3910105, "duration_s": 2.3, "finish": "stop", "prompt_tokens": 609, "reused": 0, "output_tokens": 18, "prompt_total": 609, "prompt_read": 609, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 18, "prompt_ms": 2012.6, "decode_ms": 297.0, "decode_tok_s": 60.6, "hit_rate": 0.737, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 22, "drafts_accepted": 11}, {"projection": null, "time": 1791788643.8934813, "duration_s": 4.5, "finish": "stop", "prompt_tokens": 14469, "reused": 11623, "output_tokens": 15, "prompt_total": 14469, "prompt_read": 2846, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 15, "prompt_ms": 4230.1, "decode_ms": 256.2, "decode_tok_s": 58.5, "hit_rate": 0.875, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 14, "drafts_accepted": 6}, {"projection": null, "time": 1791788639.380019, "duration_s": 4.5, "finish": "stop", "prompt_tokens": 11598, "reused": 8738, "output_tokens": 26, "prompt_total": 11598, "prompt_read": 2860, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 26, "prompt_ms": 4224.0, "decode_ms": 274.7, "decode_tok_s": 94.6, "hit_rate": 0.898, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 18, "drafts_accepted": 17}, {"projection": null, "time": 1791788634.0151885, "duration_s": 5.4, "finish": "stop", "prompt_tokens": 8692, "reused": 5241, "output_tokens": 47, "prompt_total": 8692, "prompt_read": 3451, "reasoning_recoveries": 0, "reasoning_repeat_coverage": 0.0, "engine_generated": 47, "prompt_ms": 4629.0, "decode_ms": 723.7, "decode_tok_s": 64.9, "hit_rate": 0.874, "pcie_share": 0.0, "ram_blobs": 0, "file_blobs": 0, "file_mb": 0, "drafts_offered": 44, "drafts_accepted": 22}], "conversation_cache": {"enabled": false, "budget_mib": 0, "slots": 4, "parked": 0, "bytes": 0, "evictions": 0, "parks": 0, "restores": 0, "last_event": null, "last_tokens": null, "last_at": null, "requests": 40, "requests_reused": 15, "reused_tokens": 90128, "prompt_tokens": 291963, "last_reused": 0, "last_prompt": 32049}, "requests_kept": 40, "totals": {"since": 1791788499.317125, "requests": 40, "prompt_tokens": 291963, "reused": 90128, "output_tokens": 1875, "prompt_ms": 181618.70000000004, "decode_ms": 26733.799999999992, "drafts_offered": 1526, "drafts_accepted": 1051}, "hardware": {"gpus": [{"index": 0, "util": 0, "mem_used": 16836001792, "mem_total": 17103323136, "temp": 50, "power": 25.689}, {"index": 1, "util": 100, "mem_used": 12498567168, "mem_total": 12884901888, "temp": 55, "power": 46.777}], "gpu_util": 50.0, "gpu_mem_used": 29334568960, "gpu_mem_total": 29988225024, "gpu_temp": 55, "gpu_power": 72.46600000000001, "gpu_power_limit": 350.0, "gpu_pcie_gen": 5, "gpu_pcie_gen_max": 5, "gpu_pcie_width": 8, "gpu_pcie_rx_mb": 3296.087890625, "gpu_pcie_tx_mb": 377.41015625, "cpu": 4.2, "ram_used": 65821134848, "ram_total": 66472062976, "disk_read_mb": 0.17310337465208495, "disk_write_mb": 0.3462067493041699, "tok_s": 0.0, "tok_s_mean": 0, "prefill_tok_s_mean": 1895.6}, "hardware_static": {"gpu_name": "NVIDIA GeForce RTX 5060 Ti + NVIDIA GeForce RTX 3060", "gpu_count": 2, "gpu_note": null, "cpu_name": "AMD Ryzen 9 7900X 12-Core Processor", "cores": 12, "threads": 24, "psutil": true}, "history": {"gpu_util": [50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 51.5, 50.0, 50.0, 50.0, 99.5, 99.5, 50.0, 99.0, 99.0, 50.0, 100.0, 99.5, 100.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 100.0, 100.0, 100.0, 50.0, 100.0, 98.5, 50.0, 99.5, 100.0, 100.0, 50.0, 50.0, 50.0, 50.0, 55.0, 50.0, 50.0, 50.0, 99.5, 99.5, 50.0, 100.0, 99.0, 100.0, 50.0, 100.0, 100.0, 50.0, 50.0, 50.0, 65.0, 50.0], "gpu_mem_used": [29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29332471808, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960, 29334568960], "gpu_temp": [64, 56, 53, 63, 64, 55, 53, 54, 64, 67, 67, 67, 68, 68, 65, 68, 69, 70, 68, 68, 70, 57, 57, 54, 53, 62, 66, 66, 67, 69, 70, 69, 69, 68, 69, 68, 70, 69, 70, 59, 60, 55, 55, 66, 68, 69, 69, 69, 69, 68, 68, 68, 71, 72, 71, 70, 57, 57, 56, 55], "gpu_power": [191.81, 126.33, 117.88, 159.27, 190.66, 128.41, 119.38, 105.66, 177.54, 184.29, 182.9, 219.57, 267.98, 238.14, 195.88, 258.61, 257.6, 185.15, 245.55, 262.58, 251.37, 144.68, 142.33, 120.99, 72.39, 130.59, 192.0, 184.95, 181.49, 270.77, 276.52, 208.72, 219.72, 264.33, 250.04, 209.05, 265.26, 262.33, 217.45, 156.22, 142.1, 88.03, 92.6, 174.41, 193.64, 184.13, 212.25, 271.09, 256.54, 191.13, 253.42, 262.89, 209.89, 230.22, 257.04, 255.74, 201.09, 153.29, 128.65, 72.47], "gpu_pcie_rx_mb": [3305.35, 257.88, 3292.96, 18893.09, 26970.17, 3313.07, 3292.38, 29784.47, 9.92, 1.76, 3329.47, 3308.21, 31368.28, 2860.18, 3316.38, 3307.34, 41.64, 4652.46, 28997.04, 23321.12, 3406.87, 3314.79, 62.82, 3291.67, 214.25, 4416.06, 1.31, 17834.11, 3316.01, 27260.92, 26588.14, 4797.63, 3332.33, 62.52, 2.34, 3312.65, 22083.04, 29740.8, 3311.93, 3319.45, 31320.76, 3295.91, 29920.97, 27746.8, 11003.02, 29857.36, 3315.46, 63.2, 23877.75, 14541.87, 34661.71, 21404.31, 3324.21, 24615.74, 108.05, 40.0, 3292.46, 3180.99, 3289.33, 3296.09], "cpu": [4.2, 4.2, 4.5, 18.1, 4.2, 4.2, 5.2, 21.3, 4.2, 4.2, 4.2, 4.4, 4.2, 4.2, 4.3, 4.3, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.3, 4.2, 15.8, 18.1, 4.2, 4.3, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.3, 4.2, 4.1, 4.3, 4.2, 4.2, 4.2, 4.3, 26.5, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2, 4.3, 4.2, 4.2, 4.2, 4.2, 4.2, 4.2], "ram_used": [66000932864, 66001178624, 66090168320, 66096078848, 66096496640, 66096754688, 66070831104, 65502691328, 65503133696, 65503133696, 65503133696, 65511768064, 65511768064, 65511768064, 65616625664, 65616625664, 65639755776, 65640267776, 65641074688, 65641074688, 65641562112, 65641562112, 65641562112, 65738031104, 65738031104, 65561264128, 65561677824, 65565036544, 65565036544, 65565282304, 65565282304, 65672269824, 65672269824, 65691914240, 65692565504, 65692491776, 65692491776, 65692385280, 65690836992, 65688686592, 65785413632, 65785634816, 65609682944, 65610424320, 65610424320, 65614798848, 65615044608, 65615044608, 65615044608, 65711935488, 65709961216, 65720979456, 65721380864, 65721380864, 65724284928, 65724149760, 65724149760, 65724149760, 65820618752, 65821134848], "disk_read_mb": [0.16, 0.14, 0.16, 15.12, 0.16, 0.17, 0.16, 7.07, 0.16, 0.16, 0.16, 82.94, 0.16, 0.14, 29.67, 0.16, 7.85, 0.16, 0.16, 0.16, 0.15, 0.16, 0.16, 0.16, 0.16, 0.79, 11.51, 10.02, 0.16, 0.16, 0.16, 48.36, 0.16, 5.86, 0.16, 0.18, 0.16, 0.15, 0.18, 8.32, 0.26, 0.16, 0.5, 0.16, 0.16, 0.16, 0.17, 0.16, 0.14, 0.16, 5.07, 0.16, 0.17, 0.16, 0.19, 0.23, 0.16, 0.16, 0.16, 0.17], "tok_s": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 36.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "prefill_tok_s_mean": [null, null, 894.8, null, null, null, 890.8, null, null, null, null, null, null, 1394.2, 1394.2, 1394.2, 1675.9, 1675.9, 1675.9, 1675.9, 1828.0, 1828.0, 1828.0, 1902.2, 0.0, null, null, null, null, null, 1395.0, 1395.0, 1395.0, 1395.0, 1674.6, 1674.6, 1674.6, 1826.0, 1826.0, 1826.0, 1900.0, 1900.0, null, null, null, null, null, null, 1393.9, 1393.9, 1393.9, 1673.7, 1673.7, 1673.7, 1673.7, 1823.1, 1823.1, 1823.1, 1895.6, 1895.6]}, "time": 1791788731.5003824} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/D-before.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/D-before.json new file mode 100644 index 00000000000..0e4a105337d --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/D-before.json @@ -0,0 +1 @@ +{"engine": {"model": "qwen3.8-flash-next-iq3_s", "max_context": 524288, "images": true, "version": "0.1.43", "logit_bias": 1, "context": 524288, "kv": "int8", "kv_resident": 20480, "expert_slots": 8842, "expert_cache_mib": 17910, "expert_slots_primary": 4984, "expert_cache_primary_mib": 9474, "spec": 6, "mtp_max": 4, "lookup": 3, "vram_free_mib": 324, "cvec": 0, "arena_mib": 47962, "pool_workers": 11, "pcie_frac": "0.00", "spec_min_p": "0.70", "conversation_cache_mib": 0, "conversation_cache_slots": 4, "conversation_cache_min_free_mib": 2560, "conversation_cache_min_tokens": 0, "tail_role_token": -1, "vram_elastic": 0, "engine": "0.1.43"}, "live": {"state": "idle", "queued": 0, "phase": null, "prompt_tokens": null, "prompt_read": null, "prompt_total": null, "generated": null, "max_tokens": null, "elapsed_s": null, "tok_s": null, "tok_s_mean": null, "prefill_tok_s_mean": null, "tok_s_window_s": null}, "requests": [], "conversation_cache": {"enabled": false, "budget_mib": 0, "slots": 4, "parked": 0, "bytes": 0, "evictions": 0, "parks": 0, "restores": 0, "last_event": null, "last_tokens": null, "last_at": null, "requests": 0, "requests_reused": 0, "reused_tokens": 0, "prompt_tokens": 0, "last_reused": null, "last_prompt": null}, "requests_kept": 0, "totals": {"since": 1791788499.317125, "requests": 0, "prompt_tokens": 0, "reused": 0, "output_tokens": 0, "prompt_ms": 0.0, "decode_ms": 0.0, "drafts_offered": 0, "drafts_accepted": 0}, "hardware": {"gpus": [{"index": 0, "util": 0, "mem_used": 16762601472, "mem_total": 17103323136, "temp": 38, "power": 18.561}, {"index": 1, "util": 0, "mem_used": 12427264000, "mem_total": 12884901888, "temp": 43, "power": 35.25}], "gpu_util": 0.0, "gpu_mem_used": 29189865472, "gpu_mem_total": 29988225024, "gpu_temp": 43, "gpu_power": 53.811, "gpu_power_limit": 350.0, "gpu_pcie_gen": 5, "gpu_pcie_gen_max": 5, "gpu_pcie_width": 8, "gpu_pcie_rx_mb": 0.802734375, "gpu_pcie_tx_mb": 1.072265625, "cpu": 0.1, "ram_used": 66181529600, "ram_total": 66472062976, "disk_read_mb": 0.18013172572218156, "disk_write_mb": 0.0, "tok_s": 0.0, "tok_s_mean": 0, "prefill_tok_s_mean": 0.0}, "hardware_static": {"gpu_name": "NVIDIA GeForce RTX 5060 Ti + NVIDIA GeForce RTX 3060", "gpu_count": 2, "gpu_note": null, "cpu_name": "AMD Ryzen 9 7900X 12-Core Processor", "cores": 12, "threads": 24, "psutil": true}, "history": {"gpu_util": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "gpu_mem_used": [29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472, 29189865472], "gpu_temp": [42, 42, 42, 42, 42, 42, 42, 42, 43, 43, 43, 43, 43, 43, 43, 43], "gpu_power": [57.82, 56.54, 53.53, 53.64, 53.56, 53.52, 53.69, 53.72, 53.64, 53.67, 53.74, 53.77, 53.77, 53.81, 53.79, 53.81], "gpu_pcie_rx_mb": [0.85, 0.89, 0.8, 0.8, 0.78, 0.83, 0.78, 0.88, 0.79, 0.8, 0.79, 0.79, 0.77, 0.88, 0.79, 0.8], "cpu": [0.0, 1.8, 0.1, 0.0, 0.0, 0.1, 0.0, 0.1, 0.1, 0.0, 0.0, 0.0, 0.1, 0.1, 0.0, 0.1], "ram_used": [66170847232, 66167623680, 66170122240, 66170380288, 66178867200, 66180952064, 66181464064, 66181939200, 66181939200, 66181939200, 66181881856, 66181881856, 66182139904, 66182590464, 66182590464, 66181529600], "disk_read_mb": [null, 0.22, 1.26, 0.35, 0.19, 1.28, 0.18, 0.19, 0.28, 0.18, 0.17, 0.18, 0.16, 0.21, 0.18, 0.18], "tok_s": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "prefill_tok_s_mean": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}, "time": 1791788516.510336} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/README.md b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/README.md new file mode 100644 index 00000000000..ce28361fba4 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/README.md @@ -0,0 +1,144 @@ +# Community benchmark D: `STRATA_ROUTE_TAIL_SKIP=0` on a 2-GPU layer split (IQ3_S) + +Measured on 2026-10-12 by @mmbeanz. Targets the 0.1.42 default +change, whose own log line claims *"+10..20% decode on 12-16 GB cards that miss experts"*. +Those figures were measured on single CUDA cards and HIP splits; there was no NVIDIA-split +IQ3_S number. + +**Headline.** On this box the default is **worth keeping but the claim does not hold**: +tail-skip ON is **+5.2% aggregate decode** vs OFF (95% CI -2.9%..+13.3%, p = 0.20), and the +gain is concentrated at short context (**+9.3% at 4K, +7.4% at 609, +4.8% at 16K, +1.0% at +32K**) with **no prompt-path effect**. Separately, a **measurement trap**: the engine skipped +**98,217 of 996,599 decode lookups (9.9%)** in run A, and depending on how the `hit_rate` +denominator treats them the metric moves by up to **+9.0 points** — so `hit_rate` is **not +comparable across this flag**, which matters for anyone reading coverage numbers off the +monitor. + +## Setup: identical to run A except one environment variable + +Same daily-driver config as `../A/` (`--layer-split 40`, `--pcie-frac 0.00`, pinned +`data/expert-profile.bin`), cold boot, 0 requests before the suite, engine facts identical +(`expert_slots 8842`, `expert_slots_primary 4984`, `pcie_frac 0.00`, `pool_workers 11`, +`vram_free_mib 324`). Only difference: `STRATA_ROUTE_TAIL_SKIP=0`. + +**Unlike the overlap run (`../C/`), this A/B is auditable from artifacts**: the engine +prints the state, and run A's log contains + +``` +strata serve: STRATA_ROUTE_TAIL_SKIP=7 is ON (default): ... (+10..20% decode on 12-16 GB + cards that miss experts, answers differ slightly from 0.1.41); STRATA_ROUTE_TAIL_SKIP=0 turns it off +``` + +and D's log shows the off state. Hardware/software/model/method as hand-written in +`../A/README.md`. Bench driven remotely (`--url`, LAN RTT 0.25 ms), +`GET /metrics?requests=all` before/after. Suite ~205 s. + +## Results — A (skip ON, default) vs D (skip OFF) + +| metric | A | D | delta | +| --- | ---: | ---: | ---: | +| short-chat decode (median, 9 runs) | 74.4 | 72.0 | -3.2% | +| agent decode (median, 15 turns) | 80.7 | 76.0 | -5.8% | +| prompt 32K tok/s | 1,720.8 | 1,722.7 | +0.1% | +| prompt 16K tok/s | 1,412.7 | 1,411.1 | -0.1% | +| prompt 4K tok/s | 868.1 | 871.2 | +0.4% | +| TTFT 32K (s) | 18.66 | 18.64 | -0.1% | +| draft acceptance | 68.9% | 68.9% | 0.0 pts | +| expert hit_rate (median, 40 req) | 0.859 | 0.806 | -5.3 pts *(see trap)* | + +## Statistics (per-request `/metrics?requests=all`, n = 40 per run) + +Permutation tests, 6,000 shuffles, two-sided; CI on the mean difference: + +| quantity | A (ON) | D (OFF) | skip-ON effect | p | 95% CI | +| --- | ---: | ---: | ---: | ---: | --- | +| **decode tok/s (all requests)** | 71.10 | 67.58 | **+5.2%** | 0.20 | **-2.9% .. +13.3%** | +| decode at 4K prompt | 70.7 (med) | 62.1 (med) | +9.3 | 0.10 | [+2.9, +15.8] | +| decode at 609 prompt | 71.6 (med) | 60.6 (med) | +7.4 | 0.30 | [-3.9, +18.8] | +| decode at 16K prompt | 58.3 (med) | 53.5 (med) | +4.8 | 0.10 | [+1.4, +8.2] | +| decode at 32K prompt | 53.2 (med) | 52.3 (med) | +1.0 | 0.80 | [-7.4, +9.4] | +| prompt_ms at 32K | 18,624 | 18,604 | +11 ms (0.06%) | 0.61 | [-21, +43] | +| hit_rate | 0.8417 | 0.786 | +5.6 pts | 0.016 | [+1.2, +10.1] | + +Reading: the direction is consistent (skip ON never slower, usually faster), the point +estimate is **+5.2% aggregate decode**, and the confidence interval **excludes the "+10..20%" +claim only at its upper end** — i.e. we cannot confirm the advertised range on this rig, and +the most likely true effect is single-digit. The prompt path is unaffected (as expected: +tail-skip is a decode-window mechanism). Draft acceptance is identical, so unlike the V100 +observation in #1884 (70.1% -> 68.2%) there is **no acceptance cost here**. + +## The hit_rate trap (important for anyone reading the monitor) + +Run A's log carries a cumulative counter: + +``` +strata serve: route tail skip: 98217 missed experts skipped (105328 entries) since the start +``` + +against `996,599` decode lookups summed over the same run's 40 hit-rate lines — i.e. **9.9% +of decode lookups were skipped rather than counted as misses**. Recomputing A's aggregate +hit rate both ways: + +| denominator treatment | A aggregate hit_rate | +| --- | ---: | +| skipped misses **included** as misses | 0.8259 | +| skipped misses **excluded** from lookups | 0.9162 (**+9.0 points**) | + +The measured A-vs-D hit_rate difference was +5.6 points, so **it is plausibly entirely a +definition artifact** rather than a coverage change. Practical guidance: do not compare +`hit_rate` across `STRATA_ROUTE_TAIL_SKIP` on/off, and if the docs or the Monitor present +hit rate as a coverage measure, the flag's effect on the denominator should be stated. + +**Confirmed by the cross-run arithmetic (both logs now in this folder):** + +| | A (skip ON) | D (skip OFF) | +| --- | ---: | ---: | +| output tokens | 1,766 | 1,875 | +| lookups per output token, as reported | 564.3 | **611.8** | +| skipped per output token (log counter) | 55.6 | 0 | +| lookups per token including skipped | **619.9** | 611.8 | +| `hit_rate` as reported | 0.8259 | 0.7746 | +| `hit_rate` on a common denominator | **0.7518** | 0.7746 | + +A's total routed lookups per token (619.9) match D's (611.8) to within 1.3%, and A's +*reported* lookups fall short by exactly the skipped count — so **the denominator excludes +tail-skipped misses**. On a like-for-like denominator the direction **reverses**: skip ON +measures 0.752 vs skip OFF 0.775, because the skipped experts are precisely misses the +engine chose not to compute. The Monitor's "expert cache hit" and `/metrics` `hit_rate` +therefore silently change meaning with this flag; the `route tail skip: N missed experts +skipped` log counter is the only way to correct it. + +## Why the gain is modest here (mechanism, consistent with the data) + +The flag avoids work on ~10% of decode expert lookups. Whether that translates to speed +depends on how much of the decode window is expert work: + +- **Short context (609-4K):** experts dominate the window -> +7..9%. +- **16K:** +4.8%. +- **32K:** attention/KV dominate -> +1.0% (indistinguishable from zero). + +That gradient is the clearest signal in this run, and it is the opposite of what a single +"10-20%" figure implies: the benefit shrinks as context grows, which is precisely the regime +where agent traffic spends most of its time. + +## Recommendation for this box + +**Keep the default (skip ON).** It is never slower in these measurements, worth ~5% decode +at short context, and costs nothing measurable in draft acceptance. The policy question the +engine itself flags — *answers differ slightly from 0.1.41* — is a quality decision, not a +speed one, and was not measured here (the needle suite (`../needles/`) is the place to do it). + +## Limitations + +- Speed only; no output-quality or needle comparison of skip ON vs OFF (that is the real + open question the log line's "answers differ slightly" raises). +- Decode resolution: CI spans -2.9%..+13.3%, so "+5%" is a point estimate, not an + established number; a soak or `--runs 10` would tighten it. +- The hit_rate denominator treatment is **confirmed by the two logs** (see the trap section); + what is still missing is documentation of it. Note also that **run D's log contains no + tail-skip lines at all** — when the flag is off there is no "is OFF" line and no counter, + so the off state is only inferable from the absence of the ON line. PR #2011's + "log the state in both directions" would fix that for overlap too. +- Per-request decode is highly variable (48-104 tok/s); permutation p-values assume + independence, which the adaptive tier violates, so they are optimistic. +- A vs D is one variable; both differ from B1/B1c/B1d in split policy. diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/config.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/config.json new file mode 100644 index 00000000000..37d4ef4599b --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/config.json @@ -0,0 +1,64 @@ +{ + "exe": "/home/llm/Strata/engine/strata", + "args": [ + "--pack", + "/home/llm/Strata-data/packs/iq3_s", + "--native", + "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf", + "--ple-gguf", + "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf", + "--expert-profile", + "/home/llm/Strata/data/expert-profile.bin", + "--expert-cache", + "auto", + "--prefill", + "auto", + "--spec", + "4", + "--mtp", + "/home/llm/Strata-data/mtp/rt", + "--max-context", + "524288", + "--rope-scaling", + "yarn", + "--rope-scale", + "2.0", + "--kv", + "int8", + "--kv-resident", + "20480", + "--vision", + "--remote-expert-opt", + "--trim-stage-weights", + "--pcie-frac", + "0.00", + "--spec-min-p", + "0.70" + ], + "cwd": "/home/llm/Strata", + "tokenizer": "/home/llm/Strata-data/packs/iq3_s/tokenizer", + "model_name": "qwen3.8-flash-next-iq3_s", + "log": "/home/llm/Strata/strata-iq3_s.log", + "lib_dirs": [ + "/opt/cuda/bin", + "/opt/cuda/lib64" + ], + "port": 8080, + "gpu": [ + 0, + 1 + ], + "gpus_asked": true, + "layer_split": "40", + "vision": { + "exe": "/home/llm/Strata/engine/strata-vision", + "mmproj": "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/mmproj-Qwen3.8-Flash-Next-BF16.gguf", + "model": "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf", + "gpu": true, + "max_tokens": 1024 + }, + "allowed_hosts": [ + "bertha" + ], + "draft_vocab": "en" +} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/results.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/results.json new file mode 100644 index 00000000000..dbc41d5ee11 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/D/results.json @@ -0,0 +1,1018 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-12", + "params": { + "runs": 3, + "think": false, + "clients": 0, + "long": true, + "started_server": false, + "command": "python tools/strata_bench.py --long" + }, + "elapsed_s": 208.82759809494019, + "machine": { + "gpu": [ + "NVIDIA GeForce RTX 5060 Ti", + "NVIDIA GeForce RTX 3060" + ], + "gpu_vram_mib": 28599.0, + "gpu_driver": "615.71.09", + "cuda": "13.4 (nvcc V13.4.92)", + "cpu": "AMD Ryzen 9 7900X 12-Core Processor", + "cpu_cores": 12, + "cpu_threads": 24, + "ram_gib": 61.9, + "os": "Linux 7.2.9-1-cachyos", + "python": "3.14.7", + "strata_version": "0.1.43", + "strata_commit": "7695859a", + "engine_version": "0.1.43", + "model": "qwen3.8-flash-next-iq3_s", + "quant": "IQ3_S", + "context": 524288, + "parallel_slots": 1, + "config": { + "exe": "strata", + "args": [ + "--pack", + "iq3_s", + "--native", + "Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf", + "--ple-gguf", + "Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf", + "--expert-profile", + "expert-profile-live.bin", + "--expert-profile-save", + "expert-profile-live.bin", + "--expert-cache", + "auto", + "--prefill", + "auto", + "--spec", + "4", + "--mtp", + "rt", + "--max-context", + "524288", + "--rope-scaling", + "yarn", + "--rope-scale", + "2.0", + "--kv", + "int8", + "--kv-resident", + "20480", + "--vision", + "--remote-expert-opt", + "--trim-stage-weights", + "--pcie-frac", + "0.00", + "--spec-min-p", + "0.70" + ], + "model_name": "qwen3.8-flash-next-iq3_s", + "lib_dirs": [ + "/opt/cuda/bin", + "/opt/cuda/lib64" + ], + "port": 8080, + "gpu": [ + 0, + 1 + ], + "gpus_asked": true, + "layer_split": "40", + "vision": { + "exe": "/home//Strata/engine/strata-vision", + "mmproj": "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/mmproj-Qwen3.8-Flash-Next-BF16.gguf", + "model": "/llm/models/qwen3.8-Flash-Next-GSQ-RCO/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf", + "gpu": true, + "max_tokens": 1024 + }, + "allowed_hosts": [ + "bertha" + ], + "draft_vocab": "en" + }, + "launch": "strata --pack iq3_s --native Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf --ple-gguf Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf --expert-profile expert-profile-live.bin --expert-profile-save expert-profile-live.bin --expert-cache auto --prefill auto --spec 4 --mtp rt --max-context 524288 --rope-scaling yarn --rope-scale 2.0 --kv int8 --kv-resident 20480 --vision --remote-expert-opt --trim-stage-weights --pcie-frac 0.00 --spec-min-p 0.70", + "engine_info": { + "model": "qwen3.8-flash-next-iq3_s", + "max_context": 524288, + "images": true, + "version": "0.1.43", + "logit_bias": 1, + "context": 524288, + "kv": "int8", + "kv_resident": 20480, + "expert_slots": 8842, + "expert_cache_mib": 17910, + "expert_slots_primary": 4984, + "expert_cache_primary_mib": 9474, + "spec": 6, + "mtp_max": 4, + "lookup": 3, + "vram_free_mib": 324, + "cvec": 0, + "arena_mib": 47962, + "pool_workers": 11, + "pcie_frac": "0.00", + "spec_min_p": "0.70", + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "conversation_cache_min_tokens": 0, + "tail_role_token": -1, + "vram_elastic": 0, + "engine": "0.1.43" + }, + "vram_after": { + "elastic": false + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.498, + "total_s": 1.749, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 107.6, + "prompt_tps_source": "engine", + "decode_tps": 78.9, + "decode_tps_source": "engine", + "prompt_ms": 474.0, + "decode_ms": 1268.2, + "drafts_offered": 83, + "drafts_accepted": 70, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.443, + "total_s": 1.984, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 102.6, + "prompt_tps_source": "engine", + "decode_tps": 64.1, + "decode_tps_source": "engine", + "prompt_ms": 418.9, + "decode_ms": 1559.3, + "drafts_offered": 84, + "drafts_accepted": 50, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.467, + "total_s": 1.919, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 104.1, + "prompt_tps_source": "engine", + "decode_tps": 68.0, + "decode_tps_source": "engine", + "prompt_ms": 442.0, + "decode_ms": 1470.7, + "drafts_offered": 75, + "drafts_accepted": 53, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.485, + "total_s": 1.682, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 110.6, + "prompt_tps_source": "engine", + "decode_tps": 82.3, + "decode_tps_source": "engine", + "prompt_ms": 461.2, + "decode_ms": 1214.8, + "drafts_offered": 77, + "drafts_accepted": 70, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.406, + "total_s": 1.928, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 112.4, + "prompt_tps_source": "engine", + "decode_tps": 65.0, + "decode_tps_source": "engine", + "prompt_ms": 382.4, + "decode_ms": 1539.1, + "drafts_offered": 77, + "drafts_accepted": 45, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.445, + "total_s": 1.816, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 109.3, + "prompt_tps_source": "engine", + "decode_tps": 72.0, + "decode_tps_source": "engine", + "prompt_ms": 420.8, + "decode_ms": 1388.8, + "drafts_offered": 75, + "drafts_accepted": 54, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.487, + "total_s": 1.698, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 110.1, + "prompt_tps_source": "engine", + "decode_tps": 81.4, + "decode_tps_source": "engine", + "prompt_ms": 463.3, + "decode_ms": 1228.8, + "drafts_offered": 79, + "drafts_accepted": 70, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.4, + "total_s": 1.762, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 114.4, + "prompt_tps_source": "engine", + "decode_tps": 72.5, + "decode_tps_source": "engine", + "prompt_ms": 375.8, + "decode_ms": 1380.0, + "drafts_offered": 75, + "drafts_accepted": 53, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.436, + "total_s": 1.908, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 111.8, + "prompt_tps_source": "engine", + "decode_tps": 67.1, + "decode_tps_source": "engine", + "prompt_ms": 411.6, + "decode_ms": 1490.4, + "drafts_offered": 75, + "drafts_accepted": 49, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 72.0, + "min": 64.1, + "max": 82.3, + "n": 9 + }, + "ttft_s": { + "median": 0.445, + "min": 0.4, + "max": 0.498, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 46, + "min": 43, + "max": 51, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 4.684, + "total_s": 5.051, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 863.0, + "prompt_tps_source": "engine", + "decode_tps": 62.1, + "decode_tps_source": "engine", + "prompt_ms": 4657.1, + "decode_ms": 386.4, + "drafts_offered": 13, + "drafts_accepted": 11, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.64, + "total_s": 4.976, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 871.2, + "prompt_tps_source": "engine", + "decode_tps": 67.7, + "decode_tps_source": "engine", + "prompt_ms": 4613.4, + "decode_ms": 354.7, + "drafts_offered": 14, + "drafts_accepted": 11, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.634, + "total_s": 5.04, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 872.2, + "prompt_tps_source": "engine", + "decode_tps": 56.6, + "decode_tps_source": "engine", + "prompt_ms": 4607.9, + "decode_ms": 424.4, + "drafts_offered": 23, + "drafts_accepted": 12, + "run": 2 + } + ], + "prompt_tps": { + "median": 871.2, + "min": 863.0, + "max": 872.2, + "n": 3 + }, + "ttft_s": { + "median": 4.64, + "min": 4.634, + "max": 4.684, + "n": 3 + }, + "prompt_tokens": { + "median": 4019, + "min": 4019, + "max": 4019, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 11.38, + "total_s": 11.829, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1415.0, + "prompt_tps_source": "engine", + "decode_tps": 51.3, + "decode_tps_source": "engine", + "prompt_ms": 11348.1, + "decode_ms": 467.8, + "drafts_offered": 23, + "drafts_accepted": 10, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 11.411, + "total_s": 11.841, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1411.1, + "prompt_tps_source": "engine", + "decode_tps": 53.5, + "decode_tps_source": "engine", + "prompt_ms": 11380.0, + "decode_ms": 448.7, + "drafts_offered": 23, + "drafts_accepted": 10, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 11.428, + "total_s": 11.839, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1409.1, + "prompt_tps_source": "engine", + "decode_tps": 55.9, + "decode_tps_source": "engine", + "prompt_ms": 11396.2, + "decode_ms": 429.7, + "drafts_offered": 19, + "drafts_accepted": 9, + "run": 2 + } + ], + "prompt_tps": { + "median": 1411.1, + "min": 1409.1, + "max": 1415.0, + "n": 3 + }, + "ttft_s": { + "median": 11.411, + "min": 11.38, + "max": 11.428, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "32K": { + "target_tokens": 32768, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 18.628, + "total_s": 19.136, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 1724.1, + "prompt_tps_source": "engine", + "decode_tps": 45.5, + "decode_tps_source": "engine", + "prompt_ms": 18589.0, + "decode_ms": 527.1, + "drafts_offered": 23, + "drafts_accepted": 10, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 18.642, + "total_s": 19.083, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 1722.7, + "prompt_tps_source": "engine", + "decode_tps": 52.3, + "decode_tps_source": "engine", + "prompt_ms": 18603.6, + "decode_ms": 459.3, + "drafts_offered": 21, + "drafts_accepted": 10, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 18.68, + "total_s": 19.078, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 1719.2, + "prompt_tps_source": "engine", + "decode_tps": 57.6, + "decode_tps_source": "engine", + "prompt_ms": 18641.5, + "decode_ms": 416.8, + "drafts_offered": 20, + "drafts_accepted": 12, + "run": 2 + } + ], + "prompt_tps": { + "median": 1722.7, + "min": 1719.2, + "max": 1724.1, + "n": 3 + }, + "ttft_s": { + "median": 18.642, + "min": 18.628, + "max": 18.68, + "n": 3 + }, + "prompt_tokens": { + "median": 32049, + "min": 32049, + "max": 32049, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 2.019, + "total_s": 2.349, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 21, + "prompt_tps": 305.5, + "prompt_tps_source": "engine", + "decode_tps": 60.3, + "decode_tps_source": "engine", + "prompt_ms": 1993.7, + "decode_ms": 348.2, + "drafts_offered": 24, + "drafts_accepted": 13, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.456, + "total_s": 5.25, + "prompt_tokens": 3712, + "reused_tokens": 631, + "read_tokens": 3081, + "generated_tokens": 64, + "prompt_tps": 695.9, + "prompt_tps_source": "engine", + "decode_tps": 78.8, + "decode_tps_source": "engine", + "prompt_ms": 4427.1, + "decode_ms": 811.9, + "drafts_offered": 47, + "drafts_accepted": 41, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.37, + "total_s": 4.194, + "prompt_tokens": 5245, + "reused_tokens": 3775, + "read_tokens": 1470, + "generated_tokens": 64, + "prompt_tps": 440.0, + "prompt_tps_source": "engine", + "decode_tps": 76.0, + "decode_tps_source": "engine", + "prompt_ms": 3341.1, + "decode_ms": 842.5, + "drafts_offered": 51, + "drafts_accepted": 36, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.672, + "total_s": 5.387, + "prompt_tokens": 8689, + "reused_tokens": 5308, + "read_tokens": 3381, + "generated_tokens": 54, + "prompt_tps": 728.3, + "prompt_tps_source": "engine", + "decode_tps": 73.7, + "decode_tps_source": "engine", + "prompt_ms": 4642.0, + "decode_ms": 732.6, + "drafts_offered": 40, + "drafts_accepted": 28, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.295, + "total_s": 4.515, + "prompt_tokens": 11602, + "reused_tokens": 8742, + "read_tokens": 2860, + "generated_tokens": 21, + "prompt_tps": 671.2, + "prompt_tps_source": "engine", + "decode_tps": 88.0, + "decode_tps_source": "engine", + "prompt_ms": 4260.8, + "decode_ms": 238.6, + "drafts_offered": 20, + "drafts_accepted": 14, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.268, + "total_s": 4.756, + "prompt_tokens": 14468, + "reused_tokens": 11622, + "read_tokens": 2846, + "generated_tokens": 33, + "prompt_tps": 672.0, + "prompt_tps_source": "engine", + "decode_tps": 65.2, + "decode_tps_source": "engine", + "prompt_ms": 4235.0, + "decode_ms": 506.3, + "drafts_offered": 36, + "drafts_accepted": 17, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 2.029, + "total_s": 2.329, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 24, + "prompt_tps": 303.8, + "prompt_tps_source": "engine", + "decode_tps": 75.5, + "decode_tps_source": "engine", + "prompt_ms": 2004.5, + "decode_ms": 317.7, + "drafts_offered": 22, + "drafts_accepted": 17, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.331, + "total_s": 5.018, + "prompt_tokens": 3715, + "reused_tokens": 635, + "read_tokens": 3080, + "generated_tokens": 64, + "prompt_tps": 715.4, + "prompt_tps_source": "engine", + "decode_tps": 90.7, + "decode_tps_source": "engine", + "prompt_ms": 4305.3, + "decode_ms": 705.4, + "drafts_offered": 47, + "drafts_accepted": 41, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.308, + "total_s": 4.132, + "prompt_tokens": 5248, + "reused_tokens": 3778, + "read_tokens": 1470, + "generated_tokens": 64, + "prompt_tps": 448.0, + "prompt_tps_source": "engine", + "decode_tps": 76.0, + "decode_tps_source": "engine", + "prompt_ms": 3281.3, + "decode_ms": 842.2, + "drafts_offered": 55, + "drafts_accepted": 39, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.658, + "total_s": 5.364, + "prompt_tokens": 8692, + "reused_tokens": 5241, + "read_tokens": 3451, + "generated_tokens": 47, + "prompt_tps": 745.5, + "prompt_tps_source": "engine", + "decode_tps": 64.9, + "decode_tps_source": "engine", + "prompt_ms": 4629.0, + "decode_ms": 723.7, + "drafts_offered": 44, + "drafts_accepted": 22, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.255, + "total_s": 4.511, + "prompt_tokens": 11598, + "reused_tokens": 8738, + "read_tokens": 2860, + "generated_tokens": 26, + "prompt_tps": 677.1, + "prompt_tps_source": "engine", + "decode_tps": 94.6, + "decode_tps_source": "engine", + "prompt_ms": 4224.0, + "decode_ms": 274.7, + "drafts_offered": 18, + "drafts_accepted": 17, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.263, + "total_s": 4.501, + "prompt_tokens": 14469, + "reused_tokens": 11623, + "read_tokens": 2846, + "generated_tokens": 15, + "prompt_tps": 672.8, + "prompt_tps_source": "engine", + "decode_tps": 58.5, + "decode_tps_source": "engine", + "prompt_ms": 4230.1, + "decode_ms": 256.2, + "drafts_offered": 14, + "drafts_accepted": 6, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 2.037, + "total_s": 2.316, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 18, + "prompt_tps": 302.6, + "prompt_tps_source": "engine", + "decode_tps": 60.6, + "decode_tps_source": "engine", + "prompt_ms": 2012.6, + "decode_ms": 297.0, + "drafts_offered": 22, + "drafts_accepted": 11, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.289, + "total_s": 5.014, + "prompt_tokens": 3709, + "reused_tokens": 628, + "read_tokens": 3081, + "generated_tokens": 64, + "prompt_tps": 722.7, + "prompt_tps_source": "engine", + "decode_tps": 86.2, + "decode_tps_source": "engine", + "prompt_ms": 4263.1, + "decode_ms": 742.8, + "drafts_offered": 50, + "drafts_accepted": 40, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.326, + "total_s": 4.196, + "prompt_tokens": 5242, + "reused_tokens": 3772, + "read_tokens": 1470, + "generated_tokens": 64, + "prompt_tps": 445.5, + "prompt_tps_source": "engine", + "decode_tps": 72.1, + "decode_tps_source": "engine", + "prompt_ms": 3299.7, + "decode_ms": 887.6, + "drafts_offered": 61, + "drafts_accepted": 37, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.599, + "total_s": 5.105, + "prompt_tokens": 8686, + "reused_tokens": 5306, + "read_tokens": 3380, + "generated_tokens": 39, + "prompt_tps": 739.6, + "prompt_tps_source": "engine", + "decode_tps": 74.4, + "decode_tps_source": "engine", + "prompt_ms": 4570.3, + "decode_ms": 524.3, + "drafts_offered": 33, + "drafts_accepted": 21, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.269, + "total_s": 4.515, + "prompt_tokens": 11584, + "reused_tokens": 8724, + "read_tokens": 2860, + "generated_tokens": 22, + "prompt_tps": 674.8, + "prompt_tps_source": "engine", + "decode_tps": 83.3, + "decode_tps_source": "engine", + "prompt_ms": 4238.5, + "decode_ms": 264.1, + "drafts_offered": 18, + "drafts_accepted": 13, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.269, + "total_s": 4.596, + "prompt_tokens": 14451, + "reused_tokens": 11605, + "read_tokens": 2846, + "generated_tokens": 21, + "prompt_tps": 671.8, + "prompt_tps_source": "engine", + "decode_tps": 60.8, + "decode_tps_source": "engine", + "prompt_ms": 4236.6, + "decode_ms": 345.6, + "drafts_offered": 20, + "drafts_accepted": 9, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 68.774, + "min": 68.65, + "max": 68.803, + "n": 3 + }, + "reread_tokens": { + "median": 4, + "min": 3, + "max": 74, + "n": 3 + }, + "new_tokens": { + "median": 13634, + "min": 13633, + "max": 13634, + "n": 3 + }, + "turn_ttft_s": { + "median": 4.269, + "min": 3.308, + "max": 4.672, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 14468, + "min": 14451, + "max": 14469, + "n": 3 + }, + "turn_decode_tps": { + "median": 76.0, + "min": 58.5, + "max": 94.6, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 2.029, + "min": 2.019, + "max": 2.037, + "n": 3 + } + }, + "clients": {}, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/NOTES.md b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/NOTES.md new file mode 100644 index 00000000000..6a32e0ff26c --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/NOTES.md @@ -0,0 +1,19 @@ +# Notes on these artifacts (what we changed and why) + +1. **`machine` blocks in `results.json` were hand-corrected.** Run with + `--url` (the bench client on a second machine), `collect_machine()` blends + client and server fields: the originals recorded `ram_gib: 3.8` (the client + VM), `gpu_driver: null`, and one combined `gpu` string. Corrected to the + host's measured facts: `gpu` split into the two cards, `gpu_driver + 615.71.09`, `cuda 13.4`, `ram_gib 61.9` (visible to the nspawn container; + 64 GB installed). Reported upstream in #2062. +2. **Environment variables were not captured per run.** 0.1.43 does not log + `STRATA_*` state, so the C run's `STRATA_ADAPT_OVERLAP=0` is attested by our + run record, not provable from these artifacts. The D run's + `STRATA_ROUTE_TAIL_SKIP=0` **is** auditable from the logs: A's log carries + `route tail skip: N missed experts skipped` lines (last: 98,217), D's log + carries none. PR #2011 adds the logging. +3. **One measurement day:** all six speed runs and the needle suite ran in one + continuous session on one machine, 2026-10-11 22:39 to 2026-10-12 00:46 + local (A at 22:39, D and the needles after midnight). No reboots, no other + workloads, same config tree, same pinned profile. diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/README.md b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/README.md new file mode 100644 index 00000000000..b1ece8d8248 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/README.md @@ -0,0 +1,145 @@ +# Community benchmark: RTX 5060 Ti 16 GB + RTX 3060 12 GB, unequal layer split + +Measured on 2026-10-11/12 (one continuous evening session, 22:39-00:46) by +@mmbeanz. An unequal 16+12 GB NVIDIA pair with **unequal weak PCIe links** +(gen5 x8 + gen3 x4, measured under load), answering `docs/TEST_REQUESTS.md` +row 1 (two NVIDIA cards of different sizes, `--layer-split auto`), plus A/B +runs of the 0.1.43 `STRATA_ADAPT_OVERLAP` and `STRATA_ROUTE_TAIL_SKIP` +defaults on a **split** (the open threads measured them single-GPU), and a +needle suite at 32K/128K/256K. Main limitation: one rig, no soak, no +batching, no Windows/HIP. + +Sub-reports per run: [`A/`](A/README.md) (baseline), [`B1/`](B1/README.md) +(`auto`), [`B1c/`](B1c/README.md) (explicit `6`), [`B1d/`](B1d/README.md) +(explicit `42`), [`C/`](C/README.md) (overlap off), [`D/`](D/README.md) +(tail-skip off), [`needles/`](needles/README.md). What we edited in the raw +artifacts and why: [`NOTES.md`](NOTES.md). Artifact hashes: +[`sha256sums.txt`](sha256sums.txt). + +## Hardware and software + +Hand-written from `nvidia-smi`, `lspci -vv` **under load**, `dmidecode` and +`nvcc` on the host - do not use the auto-generated `machine` block (see +#2062 and NOTES.md). + +- GPUs: **RTX 5060 Ti 16,311 MiB** on PCIe **5.0 x8** (measured H2D 28.2-28.7 + GB/s) + **RTX 3060 12,288 MiB** on PCIe **3.0 x4** (measured H2D 3.15-3.2 + GB/s), the 3060 behind the 600-series chipset; the board cannot negotiate + better widths. Idle `lspci` shows gen1 (ASPM), so only loaded readings are + meaningful. +- CPU: AMD Ryzen 9 7900X (12 cores / 24 threads). RAM: 64 GB DDR5-6400 + dual-channel (61.9 GiB visible to the container). Storage: Samsung 990 EVO + Plus 4 TB NVMe, DRAM-less (HMB); serial redacted. +- OS: Linux 7.2.9-1-cachyos, bare metal; the engine runs in a + **systemd-nspawn** container (shared host kernel: real links, real CPU + topology). GPU driver 615.71.09; CUDA 13.4 (nvcc V13.4.92). +- Strata 0.1.43, engine built from upstream `7695859a` (tag v0.1.43); the + install carries one local commit changing **only `setup.py` (pip -> uv)** - + no engine code changes. +- Background workloads: none (dedicated inference box); no power-limit + changes. Bench client on a second machine over LAN (RTT 0.25 ms; the 32K + prompt number matched an on-box run to 0.06%). + +## Model and configuration + +- Model: Qwen3.8-Flash-Next **IQ3_S**, ISTA-DASLab GSQ-RCO pack (sha256 in + `sha256sums.txt`), 512K context via yarn factor 2.0, `--vision`. +- Baseline (run A): `--layer-split 40` + `--trim-stage-weights`, `--pcie-frac + 0.00`, `--kv int8 --kv-resident 20480`, `--spec 4 --spec-min-p 0.70`, + `--mtp rt`, draft vocab `en`, pinned `data/expert-profile.bin` (24,576 + ranked pairs) with **no** `--expert-profile-save` - the profile never + mutates between runs. Exact launch lines are in each run's + `strata-iq3_s.log` and `config.json`. +- Boot facts (`GET /metrics`): `expert_slots 8842` (17,910 MiB) across both + cards, `arena_mib 47962`, `pool_workers 11`, `vram_free_mib 324`. +- `--remote-expert-opt` is present but **inert** with a layer split (it feeds + the helper-cache path only, #578). + +## Method + +`tools/strata_bench.py --long --runs 3` against our own server (we own the +lifecycle), thinking off, fixed output caps, 0 failures in all six runs. +`/metrics?requests=all` snapshotted before and after each run for the +per-request `hit_rate`, `pcie_share`, `decode_tok_s` (n=40 per run). Between +configs: full engine restart, same pinned profile, warm expert cache from the +same boot profile. One continuous session; see NOTES.md. + +## Results + +Medians with min-max over 3 runs; prompt tok/s and decode tok/s are the +engine's own timings, TTFT is the client's. + +| run | change vs A | prompt 4K | prompt 16K | prompt 32K | TTFT 32K | chat decode | agent decode | hit_rate (engine, decode) | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| A | baseline, K=40 | 868 (866-875) | 1,413 (1,412-1,415) | 1,721 (1,720-1,722) | 18.7 s | 74.4 (73.2-92.9) | 80.7 | 0.859 | +| B1 | `--layer-split auto` (chose K=3) | 836 | 1,227 | **1,291 (-25.0%)** | 24.9 s (+33%) | 66.6 (-10%) | 73.3 | 0.865 | +| B1c | explicit `--layer-split 6` | 264 | 346 | 408 (-76%) | 78.6 s | 41.4 | 38.6 | 0.669 | +| B1d | explicit `--layer-split 42` | 889 | 1,397 | 1,683 (-2.2%) | 19.1 s | 76.2 | 80.3 | 0.818 | +| C | `STRATA_ADAPT_OVERLAP=0` | 868 | 1,413 | 1,723 (+0.1%) | 18.6 s | 75.3 | 83.6 | 0.839 (mean) | +| D | `STRATA_ROUTE_TAIL_SKIP=0` | 871 | 1,411 | 1,723 (+0.1%) | 18.6 s | 72.0 | 76.0 | 0.785 (see finding 4) | + +Memory: boot snapshots only (`vram_free_mib 324` on the 3060 with everything +loaded, arena 46.8 GiB of 61.9 GiB); peak RSS/VRAM during inference not +measured - a soak with sampling is planned as a follow-up. + +## Findings + +1. **`auto` on an unequal pair with unequal links (B1).** The probes are + right (3.2 / 28.7 GB/s -> miss cost x6.26 / x1.00, matching our + `h2d_bench`), but the choice costs 25% of 32K prompt throughput: K=3 + strands 3,548 MiB on the 3060 and puts 45 layers on the 5060 Ti, which + clamps the prompt chunk to 3,328 (#1616). The engine's own caveat ("prices + decode, not the prompt") was exactly right. Related: #1238, #880, #1616. +2. **The suggested split is printed in the wrong frame (B1c vs B1d).** `auto` + reorders cards so the faster one is last (#1352), but an explicit + `--layer-split N` does not reorder - so following the engine's printed + `--layer-split 6` literally put 42 layers on the slow 3060: -76% prompt, + hit_rate 0.669, ~7 GiB stranded on the 16 GB card. The as-given-frame + equivalent (`42`, run B1d) is a wash on speed with a significantly worse + hit rate (0.818 vs 0.859, permutation p=0.024). **Question: should the + suggestion be printed in the as-given frame, or should explicit splits + inherit the reorder?** +3. **Hit rate is per-layer coverage, not held count.** A (8,842 experts held) + and B1 (5,935) are statistically indistinguishable in hit rate (p=0.93) + despite a 33% difference in held experts; no count-based coverage curve + (old or new `H(f)`) survives this. What tracks is each stage caching its + own layers' experts. Details and permutation tests in A/B1/B1c/B1d. +4. **`hit_rate` silently changes meaning with `STRATA_ROUTE_TAIL_SKIP` + (D).** From our two logs: A's reported lookups/token (564.3) **exclude** + the 98,217 tail-skipped misses; on a common denominator skip ON measures + **0.752 vs 0.775** - the sign reverses. Worth a doc line for `/metrics` + and Monitor consumers; and when the flag is off the log prints nothing at + all, so the off state is only inferable from absence. +5. **Tail-skip on a split: ~+5%, not +10-20%, and it decays with context + (D).** +9.3% at 4K, +4.8% at 16K, +1.0% at 32K (decode mean 71.10 vs + 67.58, p=0.20); prompt unaffected; draft acceptance identical. Keep the + default ON here. The unpriced cost is quality ("answers differ slightly") + - needles pass 9/9 with it on. Related: #2007 (a T10 split measured -1.4% + for `=7`), #1964. +6. **`STRATA_ADAPT_OVERLAP` is null on this split (C).** 32K prompt +0.14%, + decode inside run-to-run spread (p=0.39), hit_rate identical (p=0.92): + the -21%/-26% single-GPU regressions (#2010, #1975) do not reproduce on + an unequal two-card split with `--pcie-frac 0.00`. The flag's cost is + topology-dependent - input for #2011's opt-in-everywhere proposal. +7. **Tool findings (measurements of the tool, not the engine):** #2062 + confirmed and extended - `gpu_slug()` drops the second card + (`2x-rtx-5060-ti`), the remote `machine` block blends client fields (3.8 + GiB RAM, null driver), and the generated README cites + `engine-log-excerpt.txt` in `--url` mode where no such file is written. + Separately: `--expert-profile-save` writes on server QUIT, and + `strata_bench.py` starts **and stops** its own server - a bench run + silently overwrites the live profile with what it learned (we hit this; + pin the profile and drop `--save`). + +## Correctness and limitations + +Needles: **9/9 found** at 32K/128K/256K x depths 10/50/90 on the daily +driver (cold boot), `needles/`. Not tested: long soak (planned follow-up), +batching/concurrency, vision latency, the mmap file tier, Windows, HIP, +single-card configs on this box, and the answer-quality cost of tail-skip +beyond needles. `hit_rate` is the engine's decode-phase expert-cache metric; +prefill routing is not covered. Permutation p-values treat requests as +independent, which the adaptive tier correlates - they are optimistic. + + +*Drafted with AI assistance; I reviewed every line and ran the measurements myself.* diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/needles/README.md b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/needles/README.md new file mode 100644 index 00000000000..3af0d96794e --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/needles/README.md @@ -0,0 +1,78 @@ +# Community benchmark: needle-in-a-haystack recall (daily driver, 0.1.43 defaults) + +Measured on 2026-10-12 by @mmbeanz. The correctness counterweight +to the speed suite, on the **daily driver with defaults** (`--layer-split 40`, +`--pcie-frac 0.00`, route tail-skip ON, ADAPT_OVERLAP ON), cold boot, 0 requests before the +run. Tool: `tools/needle_bench.py --lengths 32k,128k,256k --depths 10,50,90 --timeout 900` +(haystack from the tool's `third_party/` text, code word hidden at a depth, greedy, thinking +off, normal `/v1/chat/completions` path). + +**Headline: 9 of 9 found** — perfect recall at 32K, 128K and 256K, at depths 10/50/90%, on a +16+12 GB unequal GPU pair with a 12 GB card carrying 8 layers. This is the quality anchor for +the speed findings: the tuned split (and the tail-skip default that is on for these runs) +does not cost needle recall at any tested length. + +## Results (`needles-final.json`) + +| length | depth | prompt tokens | seconds | tok/s incl. generation | result | +| --- | ---: | ---: | ---: | ---: | --- | +| 32k | 10% | 32,804 | 21.7 | 1,512 | **FOUND** | +| 32k | 50% | 32,804 | 19.8 | 1,657 | **FOUND** | +| 32k | 90% | 32,805 | 19.8 | 1,657 | **FOUND** | +| 128k | 10% | 124,963 | 66.6 | 1,876 | **FOUND** | +| 128k | 50% | 124,961 | 66.6 | 1,876 | **FOUND** | +| 128k | 90% | 124,961 | 59.2 | 2,111 | **FOUND** | +| 256k | 10% | 253,177 | 129.9 | 1,949 | **FOUND** | +| 256k | 50% | 253,177 | 130.2 | 1,945 | **FOUND** | +| 256k | 90% | 253,178 | 130.1 | 1,946 | **FOUND** | + +No misses, no hallucinated or near-miss answers, no position dependence (depth 10 vs 90 +identical within run-to-run noise). + +### Two side observations from the timings + +1. **Prompt throughput rises with length, then plateaus:** ~1,600 tok/s at 32K, ~1,950 at + 128K and 256K (these include a short generation, so they slightly understate the engine's + own prompt figure; the suite's 32K number was 1,720.8 tok/s). +2. **Cold vs warm matters at 128K.** An earlier pass with warm expert cache and OS page + cache read the same 128K prompt in **50.1 s (2,494 tok/s)**; here, on a cold engine, the + same size takes **66.6 s (1,876 tok/s)** — roughly a third slower cold. Worth stating + whenever quoting prompt tok/s. + +## Scope note: 256K is *within* the trained window, not beyond it + +The model's native context is **262,144**; our config runs 524,288 via yarn factor 2.0. A +"256k" probe is **253,177 tokens ≈ 0.97x the trained window**, so these results show recall +holds *up to* the trained length — they do **not** test yarn extrapolation. Testing beyond it +needs ~393K-480K probes (the tool skips a length if `tokens + 200 > ctx`, and the answer +needs room too, so ~480K is about the practical ceiling at 524,288 context). That is a +follow-up row, not claimed here. + +## Tool finding: the summary hides failures + +`needle_bench.py`'s summary counts only rows carrying a `found` field +(`done = [r for r in rows if "found" in r]`), so errored probes vanish from the headline: +an earlier pass on this box printed **`4 of 4 found`** while 5 of 9 requests had actually +errored. For a correctness check that is the dangerous kind of silent truncation. +Suggested fix: print `completed / attempted`, or exit non-zero when any probe errored. The +published `needles-final.json` has no error rows: 9 attempted, 9 found. + +Also noted: our config has the **conversation cache disabled** (`/metrics`: +`conversation_cache.enabled=false, budget_mib=0`), so the tool reusing one haystack across +the three depths of a length buys nothing — every probe is a cold read. + +## Follow-ups suggested + +1. **Beyond trained context:** ~393K and ~480K probes, to actually price yarn factor 2.0 + (the deep-dive warning is that static yarn can cost quality at long range). +2. **Tail-skip quality:** the same 9 probes with `STRATA_ROUTE_TAIL_SKIP=0`, to price the + engine's own caveat *"answers differ slightly from 0.1.41"*. Our speed run (../D/) + measured ~+5% decode for the default; this would measure what it costs. + +## Limitations + +- One needle per cell, single sample, no repeats: a strong smoke test, not a recall + benchmark with error bars. +- Haystack is the tool's `third_party/` text, so domain-specific; greedy + thinking off. +- 256K is within the trained window (see scope note); nothing beyond it tested yet. +- Timings include a short generation, so they understate prompt-only throughput. diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/needles/needles-final.json b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/needles/needles-final.json new file mode 100644 index 00000000000..5b6dd7cf87d --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/needles/needles-final.json @@ -0,0 +1,83 @@ +[ + { + "length": "32k", + "depth": 10, + "prompt_tokens": 32804, + "seconds": 21.7, + "found": true, + "word": "meadow-copper-504", + "answer": "meadow-copper-504" + }, + { + "length": "32k", + "depth": 50, + "prompt_tokens": 32804, + "seconds": 19.8, + "found": true, + "word": "falcon-quartz-940", + "answer": "falcon-quartz-940" + }, + { + "length": "32k", + "depth": 90, + "prompt_tokens": 32805, + "seconds": 19.8, + "found": true, + "word": "willow-cobalt-696", + "answer": "willow-cobalt-696" + }, + { + "length": "128k", + "depth": 10, + "prompt_tokens": 124963, + "seconds": 66.6, + "found": true, + "word": "falcon-saffron-138", + "answer": "falcon-saffron-138" + }, + { + "length": "128k", + "depth": 50, + "prompt_tokens": 124961, + "seconds": 66.6, + "found": true, + "word": "quartz-tundra-528", + "answer": "quartz-tundra-528" + }, + { + "length": "128k", + "depth": 90, + "prompt_tokens": 124961, + "seconds": 59.2, + "found": true, + "word": "quartz-glacier-192", + "answer": "quartz-glacier-192" + }, + { + "length": "256k", + "depth": 10, + "prompt_tokens": 253177, + "seconds": 129.9, + "found": true, + "word": "tundra-falcon-946", + "answer": "tundra-falcon-946" + }, + { + "length": "256k", + "depth": 50, + "prompt_tokens": 253177, + "seconds": 130.2, + "found": true, + "word": "willow-glacier-745", + "answer": "willow-glacier-745" + }, + { + "length": "256k", + "depth": 90, + "prompt_tokens": 253178, + "seconds": 130.1, + "found": true, + "word": "falcon-juniper-150", + "answer": "falcon-juniper-150" + } +] \ No newline at end of file diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/sha256sums.txt b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/sha256sums.txt new file mode 100644 index 00000000000..990129dd5e9 --- /dev/null +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/sha256sums.txt @@ -0,0 +1,5 @@ +# sha256sum of the model artifacts and pinned expert profile used by every run +# measured on 2026-10-12 on the bench host (sha256sum -t) +4c1eb2ceb4915e1192f4f386021897bde56a97f40a0bb78bb86465e0f7d2aca3 Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf +316b46f3a2dbd68c900f43136ab9449f9dcc3725dfd8c794847c204bc161e113 Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf +8f59b4aa8873209dff11c11e37bcda9529a1335b724a1afeea37bf6388975baf data/expert-profile.bin From 79cc2366737ddb4188a09541f5e9c04ff95c6661 Mon Sep 17 00:00:00 2001 From: beanz Date: Mon, 12 Oct 2026 16:30:20 -0700 Subject: [PATCH 2/2] bench 2026-10-11-community-rtx-5060ti-3060-split: correct the C row after review in #2010 - on a layer split the engine disables STRATA_ADAPT_OVERLAP before consulting the flag (generate.cpp 'not on a layer split'; --remote-expert-opt a second gate), so the A/B ran the same configuration twice; the null result stands as evidence of configuration equality, the 'does not reproduce on a split' headline is retracted; thanks @Krypto-Whitehat for the code check --- .../C/README.md | 28 +++++++++++++------ .../NOTES.md | 7 +++-- .../README.md | 15 ++++++---- 3 files changed, 33 insertions(+), 17 deletions(-) diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/README.md b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/README.md index f128466a15b..ce9335bb5d2 100644 --- a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/README.md +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/C/README.md @@ -3,15 +3,25 @@ Measured on 2026-10-11 by @mmbeanz. Targets the live upstream debate: **#2010** (Linux CUDA, -21% decode), **#1975** (RTX 5090, IQ3_S -26%), **#1964** (4070 Ti, IQ3_XXS) and **PR #2011** (make the 0.1.43 default opt-in everywhere). Every regression -reported so far is on a **single GPU**; this run measures the same flag on a **two-card -layer split with `--pcie-frac 0.00`**. - -**Headline.** On this topology the flag is **not a lever we can see**: the prompt path is -null to within run-to-run spread (**+0.14%** at 32K), and decode shows no significant -difference (permutation p = 0.39), which **rules out the -21%/-26% class of regressions -here** but cannot resolve effects smaller than roughly ±5-11% given per-request decode -variance. So the reported regressions do not generalize to unequal-card layer splits with -expert streaming disabled — the cost/benefit of `ADAPT_OVERLAP` is topology-dependent. +reported so far is on a **single GPU**; this run set out to measure the same flag on a +**two-card layer split with `--pcie-frac 0.00`**. + +**Headline, corrected 2026-10-12 after review in #2010: this A/B did not test the flag.** +On a layer split the engine **disables adaptive overlap before consulting +`STRATA_ADAPT_OVERLAP`** (`src/program/generate.cpp`, v0.1.43: `why = multi_gpu || +!stages.empty() ? "not on a layer split" : ...`; our config also carries a second gate, +`--remote-expert-opt`). Both arms therefore ran the *same configuration* - overlap off - +and the null result (prompt +0.14%, decode p=0.39, hit_rate p=0.92, `pcie_share` 0 on all +40 requests in both arms) is the fingerprint of configuration equality, not a topology +finding. Credit: @Krypto-Whitehat caught this in #2010 with the code check; our original +headline ("the regressions do not reproduce on a split") is retracted. + +What the run *does* establish: (a) on a split the flag cannot cost anything, because the +feature never runs there - the -21%/-26% class of regressions is structurally impossible +on this topology; (b) 0.1.43 logs nothing about the flag in either state on a split (the +banner prints only when the env var is set *and* the gate passes), which is why this run's +engine log carries no `ADAPT_OVERLAP` lines at all - fixed by #2011's logging, shipped in +0.1.43.1. ## Setup: identical to run A except one environment variable diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/NOTES.md b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/NOTES.md index 6a32e0ff26c..74e317cacf6 100644 --- a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/NOTES.md +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/NOTES.md @@ -8,11 +8,12 @@ 615.71.09`, `cuda 13.4`, `ram_gib 61.9` (visible to the nspawn container; 64 GB installed). Reported upstream in #2062. 2. **Environment variables were not captured per run.** 0.1.43 does not log - `STRATA_*` state, so the C run's `STRATA_ADAPT_OVERLAP=0` is attested by our - run record, not provable from these artifacts. The D run's + `STRATA_*` state. For the C run this turned out to be moot: on a layer split the + engine disables the overlap before consulting the flag (see #2010 and + `C/README.md`), so both C arms ran the same configuration. The D run's `STRATA_ROUTE_TAIL_SKIP=0` **is** auditable from the logs: A's log carries `route tail skip: N missed experts skipped` lines (last: 98,217), D's log - carries none. PR #2011 adds the logging. + carries none. PR #2011's state logging shipped in 0.1.43.1. 3. **One measurement day:** all six speed runs and the needle suite ran in one continuous session on one machine, 2026-10-11 22:39 to 2026-10-12 00:46 local (A at 22:39, D and the needles after midnight). No reboots, no other diff --git a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/README.md b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/README.md index b1ece8d8248..ea3763ec933 100644 --- a/bench/results/2026-10-11-community-rtx-5060ti-3060-split/README.md +++ b/bench/results/2026-10-11-community-rtx-5060ti-3060-split/README.md @@ -116,11 +116,16 @@ measured - a soak with sampling is planned as a follow-up. default ON here. The unpriced cost is quality ("answers differ slightly") - needles pass 9/9 with it on. Related: #2007 (a T10 split measured -1.4% for `=7`), #1964. -6. **`STRATA_ADAPT_OVERLAP` is null on this split (C).** 32K prompt +0.14%, - decode inside run-to-run spread (p=0.39), hit_rate identical (p=0.92): - the -21%/-26% single-GPU regressions (#2010, #1975) do not reproduce on - an unequal two-card split with `--pcie-frac 0.00`. The flag's cost is - topology-dependent - input for #2011's opt-in-everywhere proposal. +6. **`STRATA_ADAPT_OVERLAP` is inert on layer splits (C, corrected after review in + #2010).** The engine disables the overlap before consulting the flag + (`generate.cpp`: `why = multi_gpu || !stages.empty() ? "not on a layer split"`; + `--remote-expert-opt` is a second gate), so our A/B ran the same configuration + twice - the null result (prompt +0.14%, decode p=0.39, hit_rate p=0.92, + `pcie_share` 0 on all 40) is the signature of that, not a topology finding, and + our original "does not reproduce on a split" headline is retracted. What stands: + the -21%/-26% single-GPU regressions (#2010, #1975) are structurally impossible + on a split, and 0.1.43 logs nothing about the flag in either arm (fixed by + #2011, shipped in 0.1.43.1). Thanks to @Krypto-Whitehat for the code check. 7. **Tool findings (measurements of the tool, not the engine):** #2062 confirmed and extended - `gpu_slug()` drops the second card (`2x-rtx-5060-ti`), the remote `machine` block blends client fields (3.8