Skip to content

[AutoDeploy][Feature]: Draft-Target Speculative Decoding #9147

Closed
#9275
Closed
[AutoDeploy][Feature]: Draft-Target Speculative Decoding#9147
#9275
Assignees
govind-ramnarayan
Labels
AutoDeploy<NV> AutoDeploy Backendfeature requestNew feature or request. This includes new model, dtype, functionality support

Description

@govind-ramnarayan
govind-ramnarayan
opened on Nov 13, 2025

🚀 The feature, motivation and pitch

A basic implementation of two-model speculative decoding, with draft and target models that run in separate engines. The target model should run in AutoDeploy.

  • Perf not important
  • TP=1

Alternatives

No response

Additional context

No response

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Activity

  1. govind-ramnarayan
    assigned
    laikhtewari
    on Nov 13, 2025
  2. govind-ramnarayan
    added this to AutoDeploy Boardon Nov 13, 2025
  3. govind-ramnarayan
    moved this to In progress in AutoDeploy Boardon Nov 13, 2025
  4. govind-ramnarayan
    added
    feature requestNew feature or request. This includes new model, dtype, functionality support
    on Nov 13, 2025
  5. govind-ramnarayan
    assigned
    govind-ramnarayan
    and unassigned
    laikhtewari
    on Nov 13, 2025

  6. suyoggupta commented on Nov 13, 2025

    @suyoggupta
    suyoggupta
    on Nov 13, 2025
    Collaborator

    Step1:
    Any draft + any target (with compatible tokenizers) working with AD using 2 model paradigm. ETA 11/24

  7. pcastonguay
    added
    AutoDeploy<NV> AutoDeploy Backend
    on Nov 17, 2025
  8. govind-ramnarayan
    moved this from In progress to In review in AutoDeploy Boardon Nov 18, 2025
  9. govind-ramnarayan
    closed this as completedin #9275on Dec 3, 2025
  10. govind-ramnarayan
    added a commit that references this issue on Dec 3, 2025

    [#9147][feat] AutoDeploy: Draft Target Speculative Decoding (#9275)

    098b9ff
  11. github-project-automation
    moved this from In review to Done in AutoDeploy Boardon Dec 3, 2025
  12. MinaHuai
    added a commit that references this issue on Dec 10, 2025

    [TRTLLM-7963][feat] Cold L2 cache when doing autotune benchmarking. (N…

    3ff5534
  13. usberkeley
    added a commit that references this issue on Dec 11, 2025

    [NVIDIA#9147][feat] AutoDeploy: Draft Target Speculative Decoding (NV…

    409349d
  14. codego7250
    added a commit that references this issue on Dec 11, 2025

    [NVIDIA#9147][feat] AutoDeploy: Draft Target Speculative Decoding (NV…

    2008040
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

AutoDeploy<NV> AutoDeploy Backendfeature requestNew feature or request. This includes new model, dtype, functionality support

Type

No type

Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions