Skip to content

perf: Enable CUDA graphs when attention DP is used and active requests on different GPUs are uneven - #3010

Merged
jinyangyuan-nvidia merged 2 commits into
NVIDIA:mainfrom
jinyangyuan-nvidia:dev/perf_dp_cuda_graph
Mar 26, 2025
Merged

perf: Enable CUDA graphs when attention DP is used and active requests on different GPUs are uneven#3010
jinyangyuan-nvidia merged 2 commits into
NVIDIA:mainfrom
jinyangyuan-nvidia:dev/perf_dp_cuda_graph

Conversation

@jinyangyuan-nvidia

@jinyangyuan-nvidia jinyangyuan-nvidia commented Mar 24, 2025 •
edited
Loading

Copy link
Copy Markdown
Collaborator

This PR modifies the code related to dummy requests to allow the use of CUDA graphs when attention DP is used and active requests on different GPUs are uneven.

@jinyangyuan-nvidia jinyangyuan-nvidia changed the title perf: Enable CUDA graphs when attention DP is used and the number of new requests is not divisible by DP size perf: Enable CUDA graphs when attention DP is used and the number of active real requests is not divisible by DP size Mar 24, 2025
@jinyangyuan-nvidia

jinyangyuan-nvidia commented Mar 24, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@niukuo

niukuo commented Mar 24, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #266 [ run ] triggered by Bot

@niukuo

niukuo commented Mar 24, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #266 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #256 completed with status: 'FAILURE'

@jinyangyuan-nvidia

jinyangyuan-nvidia commented Mar 24, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@niukuo

niukuo commented Mar 24, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #322 [ run ] triggered by Bot

@niukuo

niukuo commented Mar 24, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #322 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #300 completed with status: 'FAILURE'

@jinyangyuan-nvidia jinyangyuan-nvidia changed the title perf: Enable CUDA graphs when attention DP is used and the number of active real requests is not divisible by DP size perf: Enable CUDA graphs when attention DP is used and active requests on different GPUs are uneven Mar 25, 2025
@jinyangyuan-nvidia

jinyangyuan-nvidia commented Mar 25, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@niukuo

niukuo commented Mar 25, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #353 [ run ] triggered by Bot

yuxianq
yuxianq reviewed Mar 25, 2025
View reviewed changes
Comment thread tensorrt_llm/_torch/pyexecutor/py_executor.py Resolved
yuxianq
yuxianq reviewed Mar 25, 2025
View reviewed changes
Comment thread tensorrt_llm/_torch/pyexecutor/py_executor.py Resolved
yuxianq
yuxianq reviewed Mar 25, 2025
View reviewed changes
Comment thread tensorrt_llm/_torch/pyexecutor/py_executor.py Resolved
yuxianq
yuxianq reviewed Mar 25, 2025
View reviewed changes
Comment thread tensorrt_llm/_torch/pyexecutor/py_executor.py Resolved
yuxianq
yuxianq reviewed Mar 25, 2025
View reviewed changes
Comment thread tensorrt_llm/_torch/pyexecutor/py_executor.py Resolved
@niukuo

niukuo commented Mar 25, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #353 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #323 completed with status: 'FAILURE'

@jinyangyuan-nvidia

jinyangyuan-nvidia commented Mar 25, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented Mar 25, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #410 [ ] completed with state FAILURE

@tensorrt-cicd

tensorrt-cicd commented Mar 25, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #414 [ ] completed with state FAILURE

@niukuo

niukuo commented Mar 25, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #418 [ run ] triggered by Bot

yuxianq
yuxianq reviewed Mar 25, 2025
View reviewed changes
Comment thread tensorrt_llm/_torch/pyexecutor/py_executor.py Resolved
@niukuo

niukuo commented Mar 25, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #418 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #362 completed with status: 'SUCCESS'

@jinyangyuan-nvidia

jinyangyuan-nvidia commented Mar 25, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@niukuo

niukuo commented Mar 25, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #450 [ run ] triggered by Bot

@niukuo

niukuo commented Mar 25, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #450 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #385 completed with status: 'FAILURE'

@jinyangyuan-nvidia

jinyangyuan-nvidia commented Mar 26, 2025

Copy link
Copy Markdown
Collaborator Author

/bot --help

@jinyangyuan-nvidia

jinyangyuan-nvidia commented Mar 26, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@niukuo

niukuo commented Mar 26, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #504 [ run ] triggered by Bot

@niukuo

niukuo commented Mar 26, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #504 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #433 completed with status: 'FAILURE'

@jinyangyuan-nvidia

jinyangyuan-nvidia commented Mar 26, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@niukuo

niukuo commented Mar 26, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #513 [ run ] triggered by Bot

@niukuo

niukuo commented Mar 26, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #513 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #438 completed with status: 'FAILURE'

@jinyangyuan-nvidia @raccoonliukai
perf: Enable CUDA graphs when attention DP is used and active request…
5a7fa6a 
…s on different GPUs are uneven

Co-authored-by: raccoonliukai <raccoonliu@tencent.com>
Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>
@jinyangyuan-nvidia

jinyangyuan-nvidia commented Mar 26, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@niukuo

niukuo commented Mar 26, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #558 [ run ] triggered by Bot

@jinyangyuan-nvidia
jinyangyuan-nvidia enabled auto-merge (squash) March 26, 2025 11:48
@niukuo

niukuo commented Mar 26, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #558 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #474 completed with status: 'SUCCESS'

@yuxianq

yuxianq commented Mar 26, 2025

Copy link
Copy Markdown
Collaborator

/bot reuse-pipeline

@niukuo

niukuo commented Mar 26, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #590 [ reuse-pipeline ] triggered by Bot

@niukuo

niukuo commented Mar 26, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #590 [ reuse-pipeline ] completed with state SUCCESS
Reusing PR_Github #558 for commit a6532d6

@jinyangyuan-nvidia
jinyangyuan-nvidia merged commit 6b583f6 into NVIDIA:main Mar 26, 2025
@Shixiaowei02

Shixiaowei02 commented Mar 29, 2025

Copy link
Copy Markdown
Collaborator

Hi @jinyangyuan-nvidia, this PR broke the L0 multi-GPU pipeline. Could you please have a look? Thanks!

@jinyangyuan-nvidia

jinyangyuan-nvidia commented Mar 29, 2025

Copy link
Copy Markdown
Collaborator Author

Sorry for the inconvenience. A fix has been created as #3146.

@jinyangyuan-nvidia
jinyangyuan-nvidia deleted the dev/perf_dp_cuda_graph branch April 2, 2025 08:23
@juney-nvidia juney-nvidia added Community want to contribute PRs initiated from Community Community Engagement help/insights needed from community labels Apr 2, 2025
wu1du2 pushed a commit to wu1du2/TensorRT-LLM that referenced this pull request May 11, 2025
@jinyangyuan-nvidia @raccoonliukai
perf: Enable CUDA graphs when attention DP is used and active request…
e14d3b3 
…s on different GPUs are uneven (NVIDIA#3010)

Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>
Co-authored-by: raccoonliukai <raccoonliu@tencent.com>
@karljang karljang removed the Community want to contribute PRs initiated from Community label Oct 6, 2026
@chuangz0 chuangz0 mentioned this pull request Oct 9, 2026
Merged
1 task done
@yuxianq yuxianq mentioned this pull request Oct 19, 2026
Merged
1 task done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yuxianq yuxianq yuxianq approved these changes

Assignees

@jinyangyuan-nvidia jinyangyuan-nvidia

Labels

Community Engagement help/insights needed from community

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@jinyangyuan-nvidia @niukuo @tensorrt-cicd @yuxianq @Shixiaowei02 @karljang @juney-nvidia