Repository navigation
[TRTLLM-16628][refactor] BREAKING: Remove unused LlmRequest fields - #21036
Conversation
|
/bot run |
|
PR_Github #80664 [ run ] triggered by Bot. Commit: |
|
PR_Github #80664 [ run ] completed with state
|
|
/bot run |
|
PR_Github #80693 [ run ] triggered by Bot. Commit: |
|
PR_Github #80693 [ run ] completed with state
|
|
/bot run |
|
PR_Github #80814 [ run ] triggered by Bot. Commit: |
|
PR_Github #80814 [ run ] completed with state
|
4a5d8fd to
ea50a8f
Compare
|
/bot run |
|
PR_Github #80854 [ run ] triggered by Bot. Commit: |
|
/bot run |
|
PR_Github #80866 [ run ] triggered by Bot. Commit: |
|
PR_Github #80854 [ run ] completed with state |
|
PR_Github #80866 [ run ] completed with state
|
|
PR_Github #81664 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #81707 [ run ] triggered by Bot. Commit: |
|
PR_Github #81707 [ run ] completed with state |
Audit prompt-adapter preparation through request translation, model inputs, and cache keys. BaseWorker prepended virtual tokens and forwarded the table, but the PyTorch model path only consumes normal embeddings or Python-owned multimodal embeddings. No production reader uses the native prompt table or vocabulary size, and LlmRequest::validate has no production caller. Reject high-level prompt adapters before loading or token mutation. Translate direct executor prompt configs into a rejection marker and fail them during active admission, preserving request-scoped errors and valid batch peers. Reject native executor-to-LlmRequest prompt-config conversion as well. Remove native prompt table/vocabulary storage, constructor arguments, accessors, bindings, dead translation, and obsolete storage/validation tests. Retain independently live extra IDs, unique cache tokens, original lengths, and child behavior. Document the unsupported public parameter behavior and correct the stale VILA model-input comment. Internal C++ request layouts/signatures and nanobind constructor/property surfaces change and require matching rebuilt libraries and bindings. Public PromptTuningConfig and executor Request serialization/pickle layouts remain unchanged; public API retirement is outside this change. Validation: - Rebuilt and installed editable package with run/run_build_wheel.sh, --job_count 68, RelWithDebInfo, SM80. - 394 CPU executor/request-utils/worker tests passed. - 57 native binding tests passed, including public Request/config round trips. - 120 of 129 native request/cache/scheduler/agent tests passed. Nine existing KV partial-copy cases require libtensorrt_llm_nixl_wrapper.so, absent from this build without NIXL_ROOT. - Admission regressions cover identical virtual IDs with different tables, optional extra IDs, request phases, cache-manager selections, children, native pickle transport, and unaffected valid peers. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Audit all native Result logprob assignments and the shared response path. LlmRequest::createResult no longer produces these values; remaining writes only restore serialization/pickle state or populate test fixtures. Live sampling writes PyResult through the ordinary and beam-search samplers. No production repository caller supplies native Result logprobs. Remove cumLogProbs/logProbs from executor::Result, binary serialization, size accounting, nanobind properties, and pickle state. Adapt the shared response consumer to read logprobs from Python results while preserving native token/completion handling, prompt probabilities, beam indexing, streaming deltas, and postprocessor behavior. Replace synthetic native logprob fixtures with Python-owned payloads or absent-output expectations. This intentionally changes the native Result source/ABI and binary format, including serialized Responses and wrappers containing native result bytes. All binary producers and consumers must use matching builds. Native Result pickle state shrinks from 11 to 9 entries; old tuples are explicitly rejected. Python sampler and high-level completion logprob contracts remain intact. Validation: - Rebuilt matching RelWithDebInfo SM80 libraries/bindings and installed the editable package using run/run_build_wheel.sh --job_count 68. - 450 selected CPU request/executor/response/worker tests passed. - 58 binding tests passed, including the new pickle-layout rejection check. - 62 native response/serialization/request tests passed, including size accounting, context metadata, and request performance metric round trips. - 10 direct GPU sampler cases passed for ordinary, processed, speculative, and beam sampling logprobs. - Extended response tests exercise native/Python formats, greedy/beam, streaming/non-streaming, prompt logprobs, pickle transport, postprocessing, and existing immutable Python logprob snapshots. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Explicitly disable py_prompt_tuning on the six MagicMock requests used by pipelined-transfer validation tests. Otherwise MagicMock synthesizes a truthy attribute and the early PyExecutor prompt-tuning rejection prevents these tests from reaching their intended checks. Validation: python -m pytest -q tests/unittest/disaggregated/test_chunked_transfer.py (58 passed). Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Remove the unsupported prompt-tuning field from the internal executor Request constructor, accessors, storage, native serialization, and Python pickle layout. Stop carrying the Python rejection marker and remove both executor admission guards now that requests cannot configure prompt tuning. Remove obsolete rejection tests and the disaggregated MagicMock defaults. Extend existing binding coverage to reject the retired keyword and verify fields following it still round-trip through pickle. Validation: rebuilt the development wheel; 47 executor binding tests, 337 affected CPU Python tests, and 66 focused C++ request/conversion/block-key tests passed. Repository formatting and lint hooks passed. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Executor requests no longer accept prompt tuning, leaving PromptTuningConfig without execution consumers. Delete the class, implementation, build entry, native serialization helpers, Python binding and pickle support, and the class-only binding test. Validation: development wheel rebuilt and binding stubs regenerated; the rebuilt module no longer exports PromptTuningConfig. All 46 executor binding tests and 61 focused native request/serialization tests passed. No tracked references to the deleted class remain. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Prompt adapters no longer have a supported execution path. Document prompt_adapter_request as deprecated across LLM, executor, and SMG entry points, and identify PromptAdapterRequest as a deprecated compatibility container. Emit DeprecationWarning from shared GenerationRequest construction for non-None adapters, while leaving omitted or None arguments warning-free and preserving the worker rejection. Keep signatures unchanged and document omitting the argument as the migration path. Validation: 75 CPU tests passed across base worker, generation request, and LLM API stability coverage (20 GPU tests deselected). Repository formatting and lint hooks passed. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Record status: deprecated for prompt_adapter_request on both committed LLM.generate and LLM.generate_async reference entries, matching their documented deprecation. Validation: all 38 LLM API stability tests passed. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
BaseWorker.submit must register its GenerationResult before enqueue so fast responses can find it. Remove that registration whenever enqueue fails, propagating the original exception unchanged and preserving other requests. Keep unsupported prompt-adapter rejection centralized in _enqueue_request, which also covers direct callers such as Ray workers. Extend CPU regressions to verify cleanup for rejected prompt adapters, RequestError, ValueError, and RuntimeError. Check successful registration, visibility during enqueue, original exception identity, and preservation of unrelated results and request-ID mappings. The test module is already listed in l0_cpu.yml. Validation: 41 CPU worker/request tests passed; 20 GPU tests deselected. Repository formatting and lint hooks passed. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
The upstream child-request refactor introduced a delegating constructor that still forwarded mPromptEmbeddingTable and mPromptVocabSize after this branch removed those members and constructor parameters. Remove the stale arguments so child construction matches the remaining request inputs. Validation: rebuilt matching RelWithDebInfo SM80 libraries and bindings; 632 CPU request/executor/response/API tests, 46 executor binding tests, and 150 focused native request/child/block-key/result/serialization/scheduler/agent tests passed. All modified native test targets, including kvCacheManagerTest, compiled. GPU integration and sampler execution were unavailable in this container. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
RpcExecutorMixin.submit sent unsupported prompt adapters with need_response=False. The worker rejection could not reach the proxy result, leaving generation pending indefinitely. Reject non-None adapters locally with RequestError before allocating a client ID, dispatching RPC, or registering a result. Add CPU regression coverage for immediate rejection, no remote submit, unchanged request ID, and preservation of existing results. A valid-request control verifies the ordinary fire-and-forget path and result registration remain intact. The test module is already covered by l0_cpu.yml. Validation: reproduced the missing rejection before the fix; 100 CPU worker/request/RPC/routing/API tests passed, 20 GPU tests deselected. Repository formatting and lint hooks passed. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
RayExecutor overrides RpcExecutorMixin.submit and bypassed its prompt-adapter preflight validation. Raise RequestError in the Ray override before allocating a client ID, issuing fire-and-forget RPC, or creating a pending result. Extend the CPU regression to invoke the actual RayExecutor.submit override as well as the RPC mixin. Load the Ray executor with scoped mocks for optional cluster import dependencies so the test runs without Ray installed. Verify rejection does not dispatch or alter IDs/results, and valid requests still submit normally. The existing test module is covered by l0_cpu.yml. Validation: reproduced the missing Ray rejection before the fix; 108 CPU request/worker/RPC/routing/Ray-stub/API tests passed, 20 GPU tests deselected. Repository formatting and lint hooks passed. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
2232901 to
cbabd8a
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #81745 [ run ] triggered by Bot. Commit: |
|
PR_Github #81745 [ run ] completed with state
|
|
/bot run |
|
PR_Github #81787 [ run ] triggered by Bot. Commit: |
|
PR_Github #81787 [ run ] completed with state |
….3: Reject conflicting LoRA adapter identities Forward-port of release/1.3 PR NVIDIA#20344 (release/1.3 commit d747876) to main. Adapted to main: NVIDIA#20043 made BaseWorker._load_lora_adapter return None and load with _lora_model_config only, so the identity checks wrap main's call and the new tests check load calls and bindings instead of a return value. Main's test_lora_ownership.py (NVIDIA#20605) builds BaseWorker without __init__, so its _make_runtime also sets the adapter identity map and locks. Main's _enqueue_request (NVIDIA#21036) reads prompt_adapter_request before the LoRA load, so the conflicting-request test's fake request sets it to None. Signed-off-by: Yibin Li <109242046+yibinl-nvidia@users.noreply.github.com> Signed-off-by: Simeng Liu <simengl@nvidia.com>
….3: Reject conflicting LoRA adapter identities Forward-port of release/1.3 PR NVIDIA#20344 (release/1.3 commit d747876) to main. Adapted to main: NVIDIA#20043 made BaseWorker._load_lora_adapter return None and load with _lora_model_config only, so the identity checks wrap main's call and the new tests check load calls and bindings instead of a return value. Main's test_lora_ownership.py (NVIDIA#20605) builds BaseWorker without __init__, so its _make_runtime also sets the adapter identity map and locks. Main's _enqueue_request (NVIDIA#21036) reads prompt_adapter_request before the LoRA load, so the conflicting-request test's fake request sets it to None. Signed-off-by: Yibin Li <109242046+yibinl-nvidia@users.noreply.github.com> Signed-off-by: Simeng Liu <simengl@nvidia.com>
….3: Reject conflicting LoRA adapter identities Forward-port of release/1.3 PR NVIDIA#20344 (release/1.3 commit d747876) to main. Adapted to main: NVIDIA#20043 made BaseWorker._load_lora_adapter return None and load with _lora_model_config only, so the identity checks wrap main's call and the new tests check load calls and bindings instead of a return value. Main's test_lora_ownership.py (NVIDIA#20605) builds BaseWorker without __init__, so its _make_runtime also sets the adapter identity map and locks. Main's _enqueue_request (NVIDIA#21036) reads prompt_adapter_request before the LoRA load, so the conflicting-request test's fake request sets it to None. Signed-off-by: Yibin Li <109242046+yibinl-nvidia@users.noreply.github.com> Signed-off-by: Simeng Liu <simengl@nvidia.com>
….3: Reject conflicting LoRA adapter identities Forward-port of release/1.3 PR NVIDIA#20344 (release/1.3 commit d747876) to main. Adapted to main: NVIDIA#20043 made BaseWorker._load_lora_adapter return None and load with _lora_model_config only, so the identity checks wrap main's call and the new tests check load calls and bindings instead of a return value. Main's test_lora_ownership.py (NVIDIA#20605) builds BaseWorker without __init__, so its _make_runtime also sets the adapter identity map and locks. Main's _enqueue_request (NVIDIA#21036) reads prompt_adapter_request before the LoRA load, so the conflicting-request test's fake request sets it to None. Signed-off-by: Yibin Li <109242046+yibinl-nvidia@users.noreply.github.com> Signed-off-by: Simeng Liu <simengl@nvidia.com>
….3: Reject conflicting LoRA adapter identities Forward-port of release/1.3 PR NVIDIA#20344 (release/1.3 commit d747876) to main. Adapted to main: NVIDIA#20043 made BaseWorker._load_lora_adapter return None and load with _lora_model_config only, so the identity checks wrap main's call and the new tests check load calls and bindings instead of a return value. Main's test_lora_ownership.py (NVIDIA#20605) builds BaseWorker without __init__, so its _make_runtime also sets the adapter identity map and locks. Main's _enqueue_request (NVIDIA#21036) reads prompt_adapter_request before the LoRA load, so the conflicting-request test's fake request sets it to None. Signed-off-by: Yibin Li <109242046+yibinl-nvidia@users.noreply.github.com> Signed-off-by: Simeng Liu <simengl@nvidia.com>
Dev Engineer Review
Resultlogprob fields and their serialized representation. These API and layout changes require compatible library and binding rebuilds.tllm.Resulthandling skips the removed fields.prompt_adapter_requestremains as a deprecated compatibility parameter. A non-Nonevalue warns, and submission paths reject the request before dispatch or token changes. Enqueue failures also clean up pending result registration while preserving the original exception.QA Engineer Review
Per-File QA Perspective
cpp/include/tensorrt_llm/batch_manager/llmRequest.h— Removes prompt-table constructor arguments, accessors, and stored fields. Verify request construction uses the updated API.cpp/include/tensorrt_llm/executor/executor.h— RemovesPromptTuningConfig, itsRequestaccessors, and native result logprob fields. Verify consumers use supported result data.cpp/include/tensorrt_llm/executor/serialization.h— Removes prompt-tuning serialization declarations. Verify supported serialization types remain available.cpp/tensorrt_llm/batch_manager/llmRequest.cpp— Removes prompt-table validation. Verify remaining extra-token-ID and request validation.cpp/tensorrt_llm/executor/CMakeLists.txt— Removes the prompt-tuning source from the executor target. Verify the target builds without it.cpp/tensorrt_llm/executor/promptTuningConfig.cpp— Deletes the prompt-tuning implementation. Verify no remaining build or runtime path depends on it.cpp/tensorrt_llm/executor/request.cpp— Removes prompt-tuning constructor and accessor behavior. Verify request construction uses the updated argument list.cpp/tensorrt_llm/executor/requestImpl.h— Removes prompt-tuning storage and serialization. Verify request serialization matches the updated layout.cpp/tensorrt_llm/executor/serialization.cpp— Omits prompt-tuning and result logprob fields. Verify round trips for remaining fields.cpp/tensorrt_llm/executor/serializeUtils.h— Removes prompt-tuning serialization dispatch. Verify unsupported types fail through the expected compile-time path.cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp— Removes prompt-table properties and constructor arguments. Verify Python construction matches the native signature.cpp/tensorrt_llm/nanobind/executor/request.cpp— Removes prompt-tuning and native logprob properties and changes pickle state sizes. Verify new round trips and handling of incompatible state sizes.cpp/tests/unit_tests/batch_manager/agentTreeTest.cpp— Updates request construction for removed prompt-table arguments. This unit test is not an integration test; no test-list entry applies.cpp/tests/unit_tests/batch_manager/blockKeyTest.cpp— Updates request fixtures for the removed prompt-tuning argument. This unit test is not an integration test; no test-list entry applies.cpp/tests/unit_tests/batch_manager/kvCacheManagerTest.cpp— Updates cache-manager request construction. Verify multimodal, LoRA, extra-token, and cache-salt cases remain covered. This unit test is not an integration test; no test-list entry applies.cpp/tests/unit_tests/batch_manager/llmRequestTest.cpp— Adds extra-token-ID and child-request coverage, and removes prompt-table validation cases. Verify valid and invalid IDs and child-request token behavior. This unit test is not an integration test; no test-list entry applies.cpp/tests/unit_tests/batch_manager/microBatchSchedulerTest.cpp— Updates scheduler request construction. Verify the fixture uses the current constructor. This unit test is not an integration test; no test-list entry applies.cpp/tests/unit_tests/executor/requestTest.cpp— Removes prompt-tuning validation and serialization cases. Verify remaining request and serialization behavior. This unit test is not an integration test; no test-list entry applies.cpp/tests/unit_tests/executor/responseTest.cpp— Removes native result logprob assertions. Verify response fields and error status remain covered. This unit test is not an integration test; no test-list entry applies.cpp/tests/unit_tests/executor/serializeUtilsTest.cpp— Updates result round-trip checks for context and performance metrics. Verify these fields serialize and deserialize correctly. This unit test is not an integration test; no test-list entry applies.tensorrt_llm/_torch/models/modeling_vila.py— Updates documentation for token and multimodal embedding flow. Verify it matches model input preparation.tensorrt_llm/_torch/pyexecutor/llm_request.py— Stops forwarding prompt-tuning fields. Verify PyTorch request conversion works without them.tensorrt_llm/executor/base_worker.py— Rejects prompt-adapter requests and cleans up pending result registration when enqueue fails. Verify rejection leaves tokens and other requests unchanged, and enqueue errors preserve their original exception.tensorrt_llm/executor/executor.py— Documentsprompt_adapter_requestas deprecated and unsupported. Verify guidance matches warning and worker behavior.tensorrt_llm/executor/request.py— Warns whenprompt_adapter_requestis non-Noneand retains the value. Verify the warning category and thatNonedoes not warn.tensorrt_llm/executor/result.py— Skips native result logprob reads while retaining sampler logprob handling. Verify both response paths.tensorrt_llm/grpc/smg/request_manager.py— Updates adapter parameter documentation. Verify it matches runtime handling.tensorrt_llm/llmapi/llm.py— Updates adapter parameter documentation for generation APIs. Verify the documented PyTorch executor limitation.tensorrt_llm/executor/rpc_proxy_mixin.py— Rejects non-Noneprompt-adapter requests before submission dispatch. Verify rejection occurs before client-ID allocation, RPC dispatch, or result registration.tensorrt_llm/executor/ray/executor.py— Rejects prompt-adapter requests before assigning an ID or submitting to the RPC client. Verify valid requests still submit through the normal path.tests/unittest/_torch/executor/test_request_utils.py— Checks removed native logprob attributes. This unit test is not an integration test; no test-list entry applies.tests/unittest/api_stability/references_committed/llm.yaml— Marks adapter parameters deprecated in API references. Verify both generation methods reflect the deprecation.tests/unittest/bindings/test_bindings_ut.py— Removes prompt-table constructor and property checks. This unit test is not an integration test; no test-list entry applies.tests/unittest/bindings/test_executor_bindings.py— Covers removed request and result properties, updated pickle state, and rejection of retired result state. This unit test is not an integration test; no test-list entry applies.tests/unittest/executor/test_base_worker.py— Covers warnings, adapter rejection, token preservation, and enqueue-failure cleanup. This unit test is not an integration test; no test-list entry applies.tests/unittest/llmapi/test_executor.py— Expands streaming and postprocessing coverage for token and logprob alignment. It checks prompt logprobs and PyTorch-only cumulative logprobs. This unit test is not an integration test; no test-list entry applies.Description
executor.Request.prompt_tuning_config, and the unusedPromptTuningConfigclass, including bindings and serialization.prompt_adapter_requestin documentation and API stability references, with a warning for non-Nonevalues.Resultlogprob fields. Python sampler results continue supplying generation logprobs.Native layouts and serialization formats changed, so libraries and bindings require matching rebuilds. Rebuilds and targeted checks passed; the earlier broader cache run had nine cases blocked by the missing NIXL wrapper.
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.