Repository navigation
bench: community report, Tesla V100 32 GB + P100 16 GB expert-helper (mixed Volta/Pascal), Flash-Next IQ3_XXS - #1452
Closed
aleesposito85 wants to merge 2 commits into
Closed
aleesposito85 wants to merge 2 commits into
aleesposito85 wants to merge 2 commits into
Conversation
…(mixed Volta/Pascal), Flash-Next IQ3_XXS (Niko1221#1079)
Owner
|
Thanks for the pull request. We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected. To reopen it takes about two minutes:
The rules are in CONTRIBUTING.md. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #1079 (data report; the issue itself was closed as answered - this keeps the numbers with the other community reports).
Summary
Community benchmark report: Tesla V100-PCIE-32GB (sm_70) + Tesla P100-PCIE-16GB (sm_60) as the expert-helper cache (
--expert-cache-device1 auto --remote-expert-opt), Qwen3.8-Flash-Next IQ3_XXS, 262,144-token context, engine 0.1.40.2 built from source (CUDA 12.8, sm_70+sm_60). This is the first mixed Volta/Pascal report: the maintainer's reply on #1079 asked for a bench PR with the data.What changed
Adds
bench/results/2026-10-07-community-v100-p100-helper/:README.md- report in the format of docs/COMMUNITY_BENCHMARKS.md (hardware, software, model hashes, exact launch config, method, results table, correctness checks, limitations).results.json- per-run engine timing blocks for all measured runs.engine-timings.txt- the engine's own timing lines for the battery.needles.json-tools/needle_bench.pyoutput: 6 of 6 found (32k x3 depths, 128k x3 depths).strata-iq3_xxs.json- the complete server config.model-sha256.txt- model, mmproj, MTP and expert-profile hashes.bench_report.py- the measurement script (runs unchanged against any Strata server).Key numbers (warm, single-sequence, 256-token cap, greedy, reasoning off): prompt 1,436.9 tok/s median at 17,971 tokens; 1,446.4 at 44,011; 1,410.5 single run at 119,731. Decode ~73 tok/s median. Two configuration notes measured for this rig:
--prefill auto:32768(the default 8192 chunk cap costs 50-100% prompt speed on helper-cache rigs - each >=1024-token chunk streams every non-resident expert) andSTRATA_PREFILL_CPU_SHARE=auto(-23% TTFT on sub-1024-token prompts, output-identical in a fixed-prompt check).No engine changes in this PR - results only.
Extra Notes
The rig has no P2P between the cards (different CPU sockets); decode includes the helper round-trip. The 0.1.39-to-0.1.40 decode delta and the helper-cache A/B were reported earlier in #1079.