Skip to content

refactor: Remove speculative decoding parameters from stateful decoders - #3024

Merged
Funatiq merged 1 commit into
NVIDIA:mainfrom
Funatiq:dev/decoder_refactor_stateful
Mar 26, 2025
Merged

refactor: Remove speculative decoding parameters from stateful decoders#3024
Funatiq merged 1 commit into
NVIDIA:mainfrom
Funatiq:dev/decoder_refactor_stateful

Conversation

@Funatiq

@Funatiq Funatiq commented Mar 24, 2025

Copy link
Copy Markdown
Collaborator

Simplify StatefulGptDecoderBatched constructor:

  • Remove speculative decoding mode parameter
  • Initialize with default mode=None
  • Update GptSession class accordingly

Simplify setup method signatures in StatefulGptDecoder and StatefulGptDecoderBatched:

  • Remove maxTokensPerStep parameter
  • Initialize decoders with default maxTokensPerStep=1
  • Update GptSession class accordingly

@Funatiq
Funatiq requested a review from dcampora March 24, 2025 10:35
@Funatiq

Funatiq commented Mar 24, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@niukuo

niukuo commented Mar 24, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #290 [ run ] triggered by Bot

@niukuo

niukuo commented Mar 24, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #290 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #279 completed with status: 'FAILURE'

@Funatiq

Funatiq commented Mar 24, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@niukuo

niukuo commented Mar 24, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #320 [ run ] triggered by Bot

@niukuo

niukuo commented Mar 24, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #320 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #299 completed with status: 'FAILURE'

@Funatiq
Funatiq force-pushed the dev/decoder_refactor_stateful branch from 222ea23 to 4752679 Compare March 25, 2025 07:43
@Funatiq

Funatiq commented Mar 25, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@niukuo

niukuo commented Mar 25, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #396 [ run ] triggered by Bot

@niukuo

niukuo commented Mar 25, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #396 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #352 completed with status: 'FAILURE'

@Funatiq

Funatiq commented Mar 25, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@niukuo

niukuo commented Mar 25, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #421 [ run ] triggered by Bot

@niukuo

niukuo commented Mar 25, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #421 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #363 completed with status: 'FAILURE'

@Funatiq
Funatiq force-pushed the dev/decoder_refactor_stateful branch from 4752679 to 42c8ab9 Compare March 25, 2025 18:24
@Funatiq

Funatiq commented Mar 25, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@niukuo

niukuo commented Mar 25, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #462 [ run ] triggered by Bot

@niukuo

niukuo commented Mar 25, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #462 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #394 completed with status: 'SUCCESS'

@DomBrown
DomBrown self-requested a review March 26, 2025 11:54
@DomBrown

DomBrown commented Mar 26, 2025

Copy link
Copy Markdown
Collaborator

Looks good to me

@Funatiq
refactor: Remove speculative decoding parameters from stateful decoders
094c3de 
Simplify StatefulGptDecoderBatched constructor:
  - Remove speculative decoding mode parameter
  - Initialize with default mode=None
  - Update GptSession class accordingly

Simplify setup method signatures in StatefulGptDecoder and StatefulGptDecoderBatched:
  - Remove maxTokensPerStep parameter
  - Initialize decoders with default maxTokensPerStep=1
  - Update GptSession class accordingly

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
@Funatiq
Funatiq force-pushed the dev/decoder_refactor_stateful branch from 42c8ab9 to 094c3de Compare March 26, 2025 12:02
@Funatiq
Funatiq enabled auto-merge (squash) March 26, 2025 12:02
@Funatiq

Funatiq commented Mar 26, 2025

Copy link
Copy Markdown
Collaborator Author

/bot reuse-pipeline

@niukuo

niukuo commented Mar 26, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #583 [ reuse-pipeline ] triggered by Bot

@niukuo

niukuo commented Mar 26, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #583 [ reuse-pipeline ] completed with state SUCCESS
Reusing PR_Github #462 for commit 094c3de

@Funatiq
Funatiq merged commit 94dd456 into NVIDIA:main Mar 26, 2025
@Funatiq
Funatiq deleted the dev/decoder_refactor_stateful branch March 26, 2025 12:26
wu1du2 pushed a commit to wu1du2/TensorRT-LLM that referenced this pull request May 11, 2025
@Funatiq
refactor: Remove speculative decoding parameters from stateful decode…
7add354 
…rs (NVIDIA#3024)

Simplify StatefulGptDecoderBatched constructor:
  - Remove speculative decoding mode parameter
  - Initialize with default mode=None
  - Update GptSession class accordingly

Simplify setup method signatures in StatefulGptDecoder and StatefulGptDecoderBatched:
  - Remove maxTokensPerStep parameter
  - Initialize decoders with default maxTokensPerStep=1
  - Update GptSession class accordingly

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
@chenfeiz0326 chenfeiz0326 mentioned this pull request Oct 17, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DomBrown DomBrown DomBrown approved these changes
@dcampora dcampora Awaiting requested review from dcampora

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Funatiq @niukuo @DomBrown