[reasonsolo]

Description

test_e2e.py::test_openai_disagg_multi_nodes_completion* still fail on H100 after the UCX/NIXL fix (#19544). Since #18654, these tests run Qwen3.5-4B with the default max_batch_size (2048) and max_seq_len (262144). For this hybrid model, the V2 cache manager reserves recurrent state up front for every resident sequence (max_batch_size * pp_size), which is about 54 GB per GPU. With free_gpu_memory_fraction=0.8 that doesn't fit on an 80 GB H100, so the ctx/gen servers exit during startup:

• tp1pp2: ValueError: The V2 Mamba GPU cache quota is too small for live recurrent states and attention pages: got 51936015974 bytes, need at least 53969223680 bytes.
• tp1pp1: RuntimeError: Executor creation failed due to insufficient GPU memory.

This PR caps max_batch_size=128 and max_seq_len=8192 in both multi-node test workers (the tests only send 10-token completions). It also removes the two H100 waivers for this bug. The GB300 waivers stay untouched.

Test Coverage

Built release/1.3 (2bafd95) and ran the test's trtllm-serve config on H100 PCIe 80 GB:

Server	Default config	With this change
tp1pp2	exits (V2 Mamba GPU cache quota is too small)	healthy, completion OK
tp1pp1	exits (insufficient GPU memory)	healthy, completion OK

With the change, the fixed reservation drops from 53.97 GB to 3.4 GB.

Bug: https://nvbugspro.nvidia.com/bug/6812588

🤖 Generated with Claude Code

Dev Engineer Review

Both multi-node worker fixtures now set max_batch_size to 128 and max_seq_len to 8192. This reduces the Qwen3.5 recurrent-state reservation for the 80 GB GPU test configuration. The waives.txt change removes the H100 waivers for the two completion cases. The GB300 waivers remain unchanged. No test execution results are available here.

QA Engineer Review

The change modifies two test-code files and tests/integration/test_lists/waives.txt. It does not add or modify test functions or IDs. Both completion cases remain in the manual-QA list; membership in the CI test-db/ lists is not established. Removing the H100 waivers re-enables those cases. Coverage verdict: needs follow-up, because no current run results confirm the tests pass without the waivers.

Per-File QA Perspective

• tests/unittest/llmapi/apps/_test_disagg_serving_multi_nodes.py: Covers multi-node disaggregated completion. The worker configuration now limits batch size and sequence length; the test is listed for manual QA. CI-list membership is not established.
• tests/unittest/llmapi/apps/_test_disagg_serving_multi_nodes_service_discovery.py: Covers completion through multi-node service discovery. The worker configuration now limits batch size and sequence length; the test is listed for manual QA. CI-list membership is not established.
• tests/integration/test_lists/waives.txt: Removes the H100 waiver entries for the service-discovery completion case and the ctx_tp1pp2-gen_tp1pp2 completion case. The GB300 waiver entries remain.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/TensorRT-LLM/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: e0dd1ded-edbf-4b1c-b7dc-deb62a4dd1d8

📥 Commits

Reviewing files that changed from the base of the PR and between 78f58bb and 641fd2f.

📒 Files selected for processing (1)

• tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

• tests/integration/test_lists/waives.txt

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

The multi-node disaggregated-serving worker configurations now set batch and sequence limits. The waiver list no longer skips two multi-node tests.

Changes

Multi-node disaggregated serving tests

Layer / File(s)	Summary
Set multi-node worker limits tests/unittest/llmapi/apps/_test_disagg_serving_multi_nodes.py, tests/unittest/llmapi/apps/_test_disagg_serving_multi_nodes_service_discovery.py	Both worker configurations set max_batch_size to 128 and max_seq_len to 8192. Comments describe Qwen3.5 recurrent-state memory reservation and GPU fit constraints.
Remove multi-node test waivers tests/integration/test_lists/waives.txt	The waiver entries for the multi-node completion and service-discovery tests were removed.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: bowenfu

Merge Risk: ⚪ Minimal · up to 641fd

The two unwaived tests use the capped worker configurations. No actionable merge risk remains beyond normal test validation.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check		The title clearly identifies the bug fix and the specific Qwen3.5 configuration limits applied to multi-node disaggregated tests. It follows the required ticket and type format.
Description check		The description explains the failure cause, the configuration changes, the affected hardware, the retained waivers, and the test results. It includes the required Description and Test Coverage section…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

[reasonsolo]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #76888 [ run ] triggered by Bot. Commit: 641fd2f Link to invocation

[tensorrt-cicd]

PR_Github #76888 [ run ] completed with state SUCCESS. Commit: 641fd2f
/LLM/main/L0_MergeRequest_PR pipeline #63392 completed with status: 'SUCCESS'

CI Report

Link to invocation