[https://nvbugs/6278337][fix] Unwaive DeepSeekV3Lite bf16 PP4 tests

[YihuiLu512]

Dev Engineer Review

The change removes three pp4, mtp_nextn=0 waiver entries from tests/integration/test_lists/waives.txt. The PR objectives attribute the deadlock to stalled MPI progress and report that the progress pump resolves it. No source code or other configuration changes are reported. Current review finding counts are unavailable.

QA Engineer Review

The waiver removal makes the three pp4 cases eligible to run instead of being skipped. The cases remain listed in the GB300 CI test list and the core manual-QA list. The PR objectives report 30/30 passes with the progress pump under UCX TCP conditions and 9/9 with the default network; these are reported validation results, not independently verified here. Coverage verdict: needs follow-up, because the change removes waivers based on validation reported for a specific revision and hardware setup.

Per-File QA Perspective

• tests/integration/test_lists/waives.txt: Removed three waiver entries for TestDeepSeekV3Lite::test_bfloat16_4gpus with pp4 and mtp_nextn=0. Confirm that the formerly skipped cases run and pass in the target CI and manual-QA environments.

Description

The three TestDeepSeekV3Lite::test_bfloat16_4gpus[pp4-mtp_nextn=0-attention_dp=False-...-overlap_scheduler=False-...] cases were waived for a PP4 deadlock. A rank's executor thread blocks on its own GPU (sampler event sync) while no thread on that rank drives MPI progress. The next rank then cannot finish receiving the pickle5 schedule message, so the ring stalls until HangDetector fires.

#19643 added an MPI progress pump on the sample-state relay thread (TLLM_PP_MPI_PROGRESS_POLL_MS, default 5 ms). That keeps the rendezvous moving and breaks the loop. This PR removes the three waive entries.

Test Coverage

The three unwaived tests:

• cuda_graph=True, torch_compile=True
• cuda_graph=True, torch_compile=False
• cuda_graph=False, torch_compile=False

Validated on main b27b1e9c90 (includes #19643), 4x GB300, with the UCX TCP conditions of the platform where the hang was reported (the schedule message takes the rendezvous path):

• Default config (pump on): 30/30 passed (10 per case), 0 hangs, GSM8K 63.65–65.39. Also 9/9 passed with the default network configuration.
• Attribution (same build, only TLLM_PP_MPI_PROGRESS_POLL_MS differs), cuda_graph=True, torch_compile=True: pump on 0/24 hangs vs pump off 14/24 hangs. Fisher two-sided p = 8e-6.
  • 12 of the 14 pump-off hangs show the same deadlock stacks: rank 2 in the sampler event sync with an idle relay thread, rank 3 in pkl5 _recv_raw.
  • In the other 2, rank 2's executor thread is in KvCache::close() instead. The rank 3 side is identical.

The commits on main since b27b1e9c90 that touch py_executor.py or sampler.py (#19896, #19484, #19791) do not change the PP relay, progress pump, or HangDetector paths.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

🤖 Generated with Claude Code

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/TensorRT-LLM/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 03a4c081-f473-4f22-9cc9-ec99fd4ccf4d

📥 Commits

Reviewing files that changed from the base of the PR and between 348192a and a71bfe6.

📒 Files selected for processing (1)

• tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

• tests/integration/test_lists/waives.txt

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

Three skip entries for TestDeepSeekV3Lite::test_bfloat16_4gpus were removed from the integration test waiver list. The entries covered pp4 with mtp_nextn=0 and different CUDA graph and torch compile combinations.

Changes

Integration test waiver updates

Layer / File(s)	Summary
Remove DeepSeek skip entries tests/integration/test_lists/waives.txt	Removed three skip entries for test_bfloat16_4gpus, covering CUDA graphs disabled, enabled without torch compile, and enabled with torch compile.

Priority: ⬇️ Low

Estimated code review effort: 1 (Trivial) | ~5 minutes

Change: Other

Suggested reviewers: bowenfu

Merge Risk: ⚪ Minimal · up to a71bf

The affected tests can run without these waivers; no actionable issue has been identified that would prevent merging.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage		No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Title check		The title follows the required NVBugs and type format and clearly describes removing the DeepSeekV3Lite BF16 PP4 test waivers.
Description check		The description explains the deadlock, identifies the MPI progress pump as the fix, lists the three affected tests, and provides detailed validation results. The checklist is also completed.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[YihuiLu512]

/bot run --disable-fail-fast --extra-stage "DGX_H200-4_GPUs-PyTorch-Post-Merge-1, DGX_B200-4_GPUs-PyTorch-1, DGX_B200-4_GPUs-PyTorch-2, DGX_B200-4_GPUs-PyTorch-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-1, DGX_B200-4_GPUs-PyTorch-Post-Merge-2, DGX_B200-4_GPUs-PyTorch-Post-Merge-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-4"

[YihuiLu512]

/bot run --disable-fail-fast --extra-stage "DGX_H200-4_GPUs-PyTorch-Post-Merge-1, DGX_B200-4_GPUs-PyTorch-1, DGX_B200-4_GPUs-PyTorch-2, DGX_B200-4_GPUs-PyTorch-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-1, DGX_B200-4_GPUs-PyTorch-Post-Merge-2, DGX_B200-4_GPUs-PyTorch-Post-Merge-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-4"

[tensorrt-cicd]

PR_Github #76887 [ run ] triggered by Bot. Commit: a71bfe6 Link to invocation

[github-actions]

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

[tensorrt-cicd]

PR_Github #76887 [ run ] completed with state FAILURE. Commit: a71bfe6
/LLM/main/L0_MergeRequest_PR pipeline #63390 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[YihuiLu512]

/bot run --disable-fail-fast --extra-stage "DGX_H200-4_GPUs-PyTorch-Post-Merge-1, DGX_B200-4_GPUs-PyTorch-1, DGX_B200-4_GPUs-PyTorch-2, DGX_B200-4_GPUs-PyTorch-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-1, DGX_B200-4_GPUs-PyTorch-Post-Merge-2, DGX_B200-4_GPUs-PyTorch-Post-Merge-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-4"

[tensorrt-cicd]

PR_Github #76936 [ run ] triggered by Bot. Commit: a71bfe6 Link to invocation

[tensorrt-cicd]

PR_Github #76936 [ run ] completed with state SUCCESS. Commit: a71bfe6
/LLM/main/L0_MergeRequest_PR pipeline #63427 completed with status: 'SUCCESS'

CI Report

Link to invocation

[YihuiLu512]

/bot run --disable-fail-fast --disable-reuse-test --stage-list "DGX_H200-4_GPUs-PyTorch-Post-Merge-1, DGX_B200-4_GPUs-PyTorch-1, DGX_B200-4_GPUs-PyTorch-Post-Merge-1"

[tensorrt-cicd]

PR_Github #76958 [ run ] triggered by Bot. Commit: a71bfe6 Link to invocation

[tensorrt-cicd]

PR_Github #76958 [ run ] completed with state SUCCESS. Commit: a71bfe6
/LLM/main/L0_MergeRequest_PR pipeline #63444 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[YihuiLu512]

/bot run --disable-fail-fast --disable-reuse-test --stage-list "DGX_H200-4_GPUs-PyTorch-Post-Merge-1"

[tensorrt-cicd]

PR_Github #77001 [ run ] triggered by Bot. Commit: a71bfe6 Link to invocation

[tensorrt-cicd]

PR_Github #77001 [ run ] completed with state SUCCESS. Commit: a71bfe6
/LLM/main/L0_MergeRequest_PR pipeline #63480 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[YihuiLu512]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #77027 [ run ] triggered by Bot. Commit: a71bfe6 Link to invocation

[tensorrt-cicd]

PR_Github #77027 [ run ] completed with state SUCCESS. Commit: a71bfe6
/LLM/main/L0_MergeRequest_PR pipeline #63501 completed with status: 'SUCCESS'

CI Report

Link to invocation