Skip to content

bench: community report, Tesla V100-PCIE-32GB + Tesla PG500-216 (2 x sm_70, layer split), Flash-Next GSQ-RCO IQ3_XXS at 256K: --pipeline-windows 2 on two stages, decode +5.3% to +17.9%, prompt unchanged, replies bit-identical - #2395

Closed
Warabii1248 wants to merge 1 commit into
Niko1221:mainfrom
Warabii1248:community-v100-pg500-pipeline-windows

Conversation

@Warabii1248

Copy link
Copy Markdown
Contributor

Results-only submission, no engine changes. Adds bench/results/2026-10-14-community-v100-pg500-pipeline-windows/ (README, three CSVs, the engine's timing lines, the two probe scripts) and one index line in docs/COMMUNITY_BENCHMARKS.md, following its report template.

--pipeline-windows 2 against off on a two-stage layer split of two sm_70 cards; the flag is the only difference between the arms. docs/MULTI_GPU.md turns it on by default for two or more AMD gfx12 stages (two stages since 0.1.44, with the draft layer moved to the first card) and keeps NVIDIA builds opt-in; on CUDA builds mtp_first_default_on() in src/program/generate.cpp is false, so the draft layer stays on the last card unless STRATA_MTP_STAGE0=1. Here, on the current main with the draft layer on the last card, decode rose 5.3% to 17.9% (engine timings, medians), prompt reads stayed within +0.0% to +0.6%, and the replies were bit-identical in all 72 requests. With the flag on, STRATA_MTP_STAGE0=1 (the draft layer on the first card, as on gfx12) was slower here: decode -2.9% to -6.7% against the draft layer on the last card, same replies.

Other two-stage numbers in the repository were taken under other conditions: docs/MULTI_GPU.md has RTX 4060 Ti + RTX 5080, decode +16% (no faster with --mmap-experts on that 32 GB PC), and a 2x RTX 3090 table (#1447) whose --pipeline-windows 2 row (decode 136.7, prefill 2,097) used layer split 29 and no --remote-expert-opt, against setup's automatic split in the baseline row (137.0 / 136.9, 1,798). Here every expert is resident in VRAM on both cards. We did not investigate the reasons for the differences. The open PR #1747 (engine 0.1.41 with patches, four GP100 stages, +14.6% to +28.8%) points the same way.

🤖 Generated with Claude Code

…peline-windows 2 on two stages

bench/results/2026-10-14-community-v100-pg500-pipeline-windows: --pipeline-windows 2 against off on a
two-stage layer split of two sm_70 cards, main at 296b59d (0.1.44.1), Flash-Next GSQ-RCO IQ3_XXS,
262,144-token context, ABBA at 170 W and 250 W. Decode +5.3% to +17.9%, prompt +0.0% to +0.6%,
replies bit-identical in all 72 requests. docs/COMMUNITY_BENCHMARKS.md: one index line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Niko1221

Copy link
Copy Markdown
Owner

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

  1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
  2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
  3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
  4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

@Niko1221 Niko1221 closed this Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants