perf(rollout): profile prefill and decode forwards - #8350
Conversation
Signed-off-by: nathon-lee <leejianwoo@gmail.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d3ee2a5429
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Signed-off-by: nathon-lee <leejianwoo@gmail.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c20d693af8
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Remove CPU-only mocked tests for accelerator event timing per review. Keep the production forward profiling implementation and profile schema unchanged. Signed-off-by: nathon-lee <leejianwoo@gmail.com>
2 participants
Summary
Extend the opt‑in HybridEngineRollout profiling with:
Accelerator events are recorded around top‑level model forwards without synchronizing every decode step. Existing profiling fields and rollout behavior remain unchanged.
For CUDA graph generation, forward timings are reported as unavailable because graph replay bypasses model forward hooks.
Follow‑up to #8197.
Validation
GPU Environment
The measurements show that shared prefill reduces prefill latency by about 70%, while decode latency remains nearly unchanged.
Thank you for reviewing!