[https://nvbugs/6278337][fix] Unwaive DeepSeekV3Lite bf16 PP4 tests (#20020) by YihuiLu512 · Pull Request #20585 · NVIDIA/TensorRT-LLM

YihuiLu512 · 2026-10-19T07:32:03Z

Dev Engineer Review

tests/integration/test_lists/waives.txt removes three DeepSeekV3Lite BF16 PP4 waivers for NVBug 6278337. The waiver for torch_compile=True with CUDA graphs disabled remains.
This changes test selection only. No implementation or API changes are reported. Current review finding counts are unavailable.

QA Engineer Review

One test-list file changed: tests/integration/test_lists/waives.txt. Three PP4 test configurations are no longer skipped; the remaining PP4 waiver is unchanged.
The change makes those configurations eligible to run. This result does not establish that they pass. Test execution results are unavailable; coverage verdict: needs follow-up.

Per-File QA Perspective

tests/integration/test_lists/waives.txt: Removes three waivers for DeepSeekV3Lite BF16 PP4 configurations. The change affects test selection, not test code; CI or manual-QA list placement is not established by the available evidence.

Description

Cherry-pick from #20020.

This removes the three NVBug 6278337 waives for TestDeepSeekV3Lite::test_bfloat16_4gpus[pp4-mtp_nextn=0-attention_dp=False-...-overlap_scheduler=False-...] on release/1.3, where the bug is committed for v1.3.0. The fix, #19643, is on this branch through #19947. #20258 (cherry-pick of #20030) is also on this branch; it adds a native MPI progress thread on PP ranks on top of #19643, as on main since 2026-10-15.

The cherry-pick had a context conflict: on release/1.3, an unrelated waive line sits between the three lines. That line is left unchanged; only the three 6278337 lines are removed.

Test Coverage

The three unwaived tests are in these release/1.3 lists:

cuda_graph=True, torch_compile=True and cuda_graph=True, torch_compile=False: l0_dgx_h200.yml, post-merge only.
cuda_graph=False, torch_compile=False: l0_dgx_b200.yml, pre-merge and post-merge.
All three: qa/llm_function_core.txt.

Before merge, the H200 post-merge stage and the B200 4-GPU stages will be run with --extra-stage.

On main after #20030 merged (2026-10-15 to 2026-10-19), L0 runs on commits that contain both #19643 and #20030 passed 8/8, 8/8 and 35/35 times respectively, with no failures. The GB300 validation of #19643 for these tests, on the platform where the hang was reported, is in the description of #20020.

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

…VIDIA#20020) Signed-off-by: Yihui Lu <269394165+YihuiLu512@users.noreply.github.com>

coderabbitai · 2026-10-19T07:33:50Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: NVIDIA/TensorRT-LLM/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: bb11ddad-8162-4eea-ac06-9583d1e5ae20

📥 Commits

Reviewing files that changed from the base of the PR and between 4317a84 and 1ad7139.

📒 Files selected for processing: 1

tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes: 1

tests/integration/test_lists/waives.txt

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The DeepSeekV3Lite pp4 waiver list removes three skip entries. It retains the entry for torch_compile=True with CUDA graphs disabled.

Changes

Integration test waivers

Layer / File(s)	Summary
Update pp4 waiver entries `tests/integration/test_lists/waives.txt`	The list retains the case with `torch_compile=True` and CUDA graphs disabled. It removes the other three combinations.

Priority: ➖ Normal

Estimated code review effort: 1 (Trivial) | ~3 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 1ad71

No actionable issue is identified with the waiver update; it is mergeable after the planned test runs.

Pre-merge checks: 5

Check name	Status	Explanation
Title check		The title clearly identifies the NVBug, fix type, and primary change: removing DeepSeekV3Lite BF16 PP4 test waives.
Description check		The description explains the cherry-pick, affected waives, branch context, conflict handling, test coverage, validation results, and checklist status.
Docstring Coverage		No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-10-19T07:35:24Z

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

YihuiLu512 · 2026-10-19T07:41:52Z

/bot run --disable-fail-fast --extra-stage "DGX_H200-4_GPUs-PyTorch-Post-Merge-1, DGX_B200-4_GPUs-PyTorch-1, DGX_B200-4_GPUs-PyTorch-2, DGX_B200-4_GPUs-PyTorch-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-1, DGX_B200-4_GPUs-PyTorch-Post-Merge-2, DGX_B200-4_GPUs-PyTorch-Post-Merge-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-4"

tensorrt-cicd · 2026-10-19T07:49:42Z

PR_Github #78844 [ run ] triggered by Bot. Commit: 1ad7139 Link to invocation

YihuiLu512 · 2026-10-19T12:19:58Z

/bot run --disable-fail-fast --extra-stage "DGX_H200-4_GPUs-PyTorch-Post-Merge-1, DGX_B200-4_GPUs-PyTorch-1, DGX_B200-4_GPUs-PyTorch-2, DGX_B200-4_GPUs-PyTorch-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-1, DGX_B200-4_GPUs-PyTorch-Post-Merge-2, DGX_B200-4_GPUs-PyTorch-Post-Merge-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-4"

tensorrt-cicd · 2026-10-19T12:27:58Z

PR_Github #78910 [ run ] triggered by Bot. Commit: 1ad7139 Link to invocation

tensorrt-cicd · 2026-10-19T12:30:36Z

PR_Github #78844 [ run ] completed with state ABORTED. Commit: 1ad7139

Link to invocation

tensorrt-cicd · 2026-10-19T17:09:57Z

PR_Github #78910 [ run ] completed with state SUCCESS. Commit: 1ad7139
/LLM/release-1.3/L0_MergeRequest_PR pipeline #350 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

YihuiLu512 · 2026-10-19T17:20:22Z

/bot run --disable-fail-fast --extra-stage "DGX_H200-4_GPUs-PyTorch-Post-Merge-1, DGX_B200-4_GPUs-PyTorch-1, DGX_B200-4_GPUs-PyTorch-2, DGX_B200-4_GPUs-PyTorch-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-1, DGX_B200-4_GPUs-PyTorch-Post-Merge-2, DGX_B200-4_GPUs-PyTorch-Post-Merge-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-4"

tensorrt-cicd · 2026-10-19T17:27:04Z

PR_Github #78994 [ run ] triggered by Bot. Commit: 1ad7139 Link to invocation

tensorrt-cicd · 2026-10-19T20:00:52Z

PR_Github #78994 [ run ] completed with state SUCCESS. Commit: 1ad7139
/LLM/release-1.3/L0_MergeRequest_PR pipeline #362 completed with status: 'SUCCESS'

CI Report

Link to invocation

Remove 7 waive entries on release/1.3 whose NVBugs are closed and whose tests passed on the waived platform in pre-merge CI (L0_MergeRequest_PR NVIDIA#342 with post-merge extra stages): 6481323, 6428057, 6893730, 6762288 (x2), 6700265, 6075431. Waives for the other closed bugs (6435097, 6695515, 6672360, 6812288, 6432818, 6428092, 6432826) are kept: their tests did not run on the waived platform in that CI run, so the unwaive is not verified yet. 6278337 is handled separately by NVIDIA#20585. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

Remove 5 waive entries on release/1.3 whose NVBugs are closed and whose tests passed on the waived platform in pre-merge CI (L0_MergeRequest_PR NVIDIA#342 and NVIDIA#375, with post-merge extra stages): 6481323, 6428057, 6762288 (triton variant), 6700265, 6075431. Handled by other release/1.3 PRs: 6893730 (NVIDIA#20592), the 6762288 cutlass variant and 6672360 (NVIDIA#20632), 6278337 (NVIDIA#20585). Waives for the other closed bugs (6435097, 6695515, 6812288, 6432818, 6428092, 6432826) are kept: their tests did not run on the waived platform in that CI run, so the unwaive is not verified yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

Remove 5 waive entries on release/1.3 whose NVBugs are closed and whose tests passed on the waived platform in pre-merge CI (L0_MergeRequest_PR 6481323, 6428057, 6762288 (triton variant), 6700265, 6075431. Handled by other release/1.3 PRs: 6893730 (NVIDIA#20592), the 6762288 cutlass variant and 6672360 (NVIDIA#20632), 6278337 (NVIDIA#20585). Waives for the other closed bugs (6435097, 6695515, 6812288, 6432818, 6428092, 6432826) are kept: their tests did not run on the waived platform in that CI run, so the unwaive is not verified yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

Remove the remaining 6 waive entries on release/1.3 whose NVBugs are closed. PRs NVIDIA#20585, NVIDIA#20590, NVIDIA#20592 and NVIDIA#20632 handle the other closed-bug entries. - 6428092, 6432826: pipeline-parallel TorchSampler IndexError, fixed by NVIDIA#16163, which is on release/1.3. - 6435097, 6432818: closed as fixed by NVIDIA#16163, but their B200 / GB300 waivers were added later from runs on those GPUs; only QA lists run these tests there. - 6695515, 6812288: closed as not reproducible. The 6428092 entry waives the whole `-m "gpu2"` selection of test_llm_multi_gpu_pytorch.py, which also holds test_llm_rpc_tp2, test_llm_rpc_streaming_tp2 and test_llm_return_logprobs_streaming_tp2. main waives those three under 6859934 because it does not have the fix in NVIDIA#19847 yet; release/1.3 has it. Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

Remove 5 waive entries on release/1.3 whose NVBugs are closed and whose tests passed on the waived platform in pre-merge CI (L0_MergeRequest_PR 6481323, 6428057, 6762288 (triton variant), 6700265, 6075431. Handled by other release/1.3 PRs: 6893730 (NVIDIA#20592), the 6762288 cutlass variant and 6672360 (NVIDIA#20632), 6278337 (NVIDIA#20585). Waives for the other closed bugs (6435097, 6695515, 6812288, 6432818, 6428092, 6432826) are kept: their tests did not run on the waived platform in that CI run, so the unwaive is not verified yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

[https://nvbugs/6278337][fix] Unwaive DeepSeekV3Lite bf16 PP4 tests (N…

1ad7139

…VIDIA#20020) Signed-off-by: Yihui Lu <269394165+YihuiLu512@users.noreply.github.com>

github-actions Bot assigned YihuiLu512 Oct 19, 2026

lori-ren approved these changes Oct 19, 2026

View reviewed changes

trtllm-agent added the ci: full pre-merge approved label Oct 19, 2026

QiJune mentioned this pull request Oct 19, 2026

[None][test] Remove closed-bug waive entries for release/1.3 #20590

Merged

1 task done

YihuiLu512 merged commit d40f019 into NVIDIA:release/1.3 Oct 19, 2026

QiJune mentioned this pull request Oct 20, 2026

[None][test] Unwaive test_llm_multi_gpu_pytorch gpu2 tests for release/1.3 #20716

Merged

1 task done

Navigation Menu

Repository navigation

[https://nvbugs/6278337][fix] Unwaive DeepSeekV3Lite bf16 PP4 tests (#20020) - #20585

[https://nvbugs/6278337][fix] Unwaive DeepSeekV3Lite bf16 PP4 tests (#20020)#20585
YihuiLu512 merged 1 commit into
NVIDIA:release/1.3from
YihuiLu512:bug-fix-J13270-N6278337-rel1.3

YihuiLu512 commented Oct 19, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Oct 19, 2026

Uh oh!

github-actions Bot commented Oct 19, 2026

Uh oh!

YihuiLu512 commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

YihuiLu512 commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

YihuiLu512 commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

YihuiLu512 commented Oct 19, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Per-File QA Perspective

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

coderabbitai Bot commented Oct 19, 2026

Walkthrough

Changes

Uh oh!

github-actions Bot commented Oct 19, 2026

Uh oh!

YihuiLu512 commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

YihuiLu512 commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

YihuiLu512 commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

YihuiLu512 commented Oct 19, 2026 •

edited by coderabbitai Bot

Loading