Repository navigation
bench: community report, Tesla V100-PCIE-32GB + Tesla PG500-216 (2 x sm_70, layer split), Flash-Next GSQ-RCO IQ3_XXS at 256K: --pipeline-windows 2 on two stages, decode +5.3% to +17.9%, prompt unchanged, replies bit-identical - #2395
Closed
Warabii1248 wants to merge 1 commit into
Closed
Warabii1248 wants to merge 1 commit into
Warabii1248 wants to merge 1 commit into
Conversation
…peline-windows 2 on two stages bench/results/2026-10-14-community-v100-pg500-pipeline-windows: --pipeline-windows 2 against off on a two-stage layer split of two sm_70 cards, main at 296b59d (0.1.44.1), Flash-Next GSQ-RCO IQ3_XXS, 262,144-token context, ABBA at 170 W and 250 W. Decode +5.3% to +17.9%, prompt +0.0% to +0.6%, replies bit-identical in all 72 requests. docs/COMMUNITY_BENCHMARKS.md: one index line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
|
Thanks for the pull request. We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected. To reopen it takes about two minutes:
The rules are in CONTRIBUTING.md. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Results-only submission, no engine changes. Adds
bench/results/2026-10-14-community-v100-pg500-pipeline-windows/(README, three CSVs, the engine's timing lines, the two probe scripts) and one index line in docs/COMMUNITY_BENCHMARKS.md, following its report template.--pipeline-windows 2against off on a two-stage layer split of two sm_70 cards; the flag is the only difference between the arms. docs/MULTI_GPU.md turns it on by default for two or more AMD gfx12 stages (two stages since 0.1.44, with the draft layer moved to the first card) and keeps NVIDIA builds opt-in; on CUDA buildsmtp_first_default_on()in src/program/generate.cpp is false, so the draft layer stays on the last card unlessSTRATA_MTP_STAGE0=1. Here, on the current main with the draft layer on the last card, decode rose 5.3% to 17.9% (engine timings, medians), prompt reads stayed within +0.0% to +0.6%, and the replies were bit-identical in all 72 requests. With the flag on,STRATA_MTP_STAGE0=1(the draft layer on the first card, as on gfx12) was slower here: decode -2.9% to -6.7% against the draft layer on the last card, same replies.Other two-stage numbers in the repository were taken under other conditions: docs/MULTI_GPU.md has RTX 4060 Ti + RTX 5080, decode +16% (no faster with
--mmap-expertson that 32 GB PC), and a 2x RTX 3090 table (#1447) whose--pipeline-windows 2row (decode 136.7, prefill 2,097) used layer split 29 and no--remote-expert-opt, against setup's automatic split in the baseline row (137.0 / 136.9, 1,798). Here every expert is resident in VRAM on both cards. We did not investigate the reasons for the differences. The open PR #1747 (engine 0.1.41 with patches, four GP100 stages, +14.6% to +28.8%) points the same way.🤖 Generated with Claude Code