[100milliongold]

Fixes #18609.

Problem

Every generation request against openai/whisper-large-v3 on the PyTorch backend dies in the C++ capacity scheduler, before the forward pass:

File ".../scheduler/scheduler.py", line 462, in schedule_request
    return self.impl(
RuntimeError: bad optional access

capacityScheduler.cpp dereferences the encoder unique tokens at three places, guarded only by the presence of a cross KV cache manager with block reuse on:

auto uniqueTokens = *(req->getEncoderUniqueTokens().value());

None of the three checks whether the optional holds anything.

Why it is empty

mEncoderUniqueTokens is derived from encoder tokens alone (llmRequest.h:2272):

if (mEncoderTokens.has_value() && mEncoderTokens.value())
{
    ...
    mEncoderUniqueTokens = encoderUniqueTokens;
}

Whisper's encoder input is features, not tokens: modeling_whisper.py:788 passes "encoder_input_features", and llm_request.py:928-934 leaves encoder_input_tokens as None. So the optional is never populated.

enable_block_reuse defaults to True (llm_args.py:4106), and an encoder-decoder run must set cross_kv_cache_fraction, which is what creates the cross manager. Both guard conditions therefore hold on a default run, which is why the reporter saw it on every request. It is not architecture-specific; the reporter's SM120 box is not what selects this path.

Change

Add the missing condition at each of the three sites.

Prefix reuse matches on token prefixes, so a features-based encoder has nothing for analyzePrefixReuse to walk. Leaving crossSummary unset is the state its two consumers already handle:

// oneManagerBeneficialToSkip
return summary.has_value() && summary->firstNewBlock.has_value() && ...

// registerContributedBlocks
if (crossSummary.has_value() && crossSummary->firstNewBlock.has_value()) { ... }

and it is the state they already see whenever there is no cross manager, block reuse is off, or the manager is variable-window.

The same header already distinguishes the two shapes of encoder input, which is what made the token-only assumption visible (llmRequest.h:663):

/// @return Encoder input length, or nullopt when this request has no encoder side.
[[nodiscard]] std::optional<SizeType32> tryGetEncoderInputLen() const
{
    if (mEncoderInputFeatures.has_value()) { ... }
    if (getEncoderTokens().has_value())    { ... }
    return std::nullopt;
}

Verification

• clang-format 16.0.0, the version pinned in .pre-commit-config.yaml, reports no diff on the file.
• grep finds three dereferences of getEncoderUniqueTokens().value() in this file; all three now sit behind has_value().

What I have not done

I did not reproduce the failure or run this patch myself. I have no SM120 GPU and no Whisper setup; the analysis is from reading main, and the reporter's trace is what located it.

Since writing that, @ravo42-maker has run it. They applied this hunk onto the v1.3.0rc25 tag (785c948, where all three sites are identical), built a wheel for 120-real, and ran openai/whisper-large-v3 on an RTX 5070 Ti. With max_input_len=1500 and block reuse left on, stock rc25 raises bad optional access and rc25 + this change transcribes the clip in 3.95 s. Their five-run table isolates block reuse as the only variable between the crash and a working run. See the comment below; the verification is theirs, not mine.

I also have not added a regression test. The natural one is a capacity-scheduler unit test with a request carrying encoder input features and no encoder tokens, with a cross manager and block reuse enabled. I would rather add it in the shape this repository prefers than guess, so please say if you want it here.

Note beyond this fix

The reporter lists four further gaps in the same issue: no /v1/audio/transcriptions route, no Whisper chat template shipped, no Whisper example or test in the container, and no release that currently provides a working Whisper path. This change addresses only the scheduler crash.

Dev Engineer Review

• The supplied diff changes only tests/integration/defs/accuracy/test_llm_api_pytorch.py.
• Changes are formatting-only: import grouping, multiline formatting, and two docstring conversions.
• The supplied diff does not contain the claimed capacityScheduler.cpp fix. The scheduler crash fix is therefore not verified by this diff.

QA Engineer Review

• tests/integration/defs/accuracy/test_llm_api_pytorch.py was modified.
• No test IDs, selectors, waivers, or functional test behavior changed.
• No scheduler or Whisper regression test was added. @ravo42-maker's run is a manual confirmation, not a test in CI.
• Test-list coverage was not changed or established.
• Coverage verdict: insufficient.

Per-File QA Perspective

• tests/integration/defs/accuracy/test_llm_api_pytorch.py: The changes affect formatting only and do not add behavioral coverage. Verify that the file remains included in its applicable CI test list.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: NVIDIA/TensorRT-LLM/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3ffa7e97-957a-485a-aa7a-93add60c89d6

📥 Commits

Reviewing files that changed from the base of the PR and between 9855bc8 and 3b82b78.

📒 Files selected for processing (2)

• cpp/tensorrt_llm/batch_manager/capacityScheduler.cpp
• tests/integration/defs/accuracy/test_llm_api_pytorch.py

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.

Walkthrough

The capacity scheduler now avoids unset encoder-token optional access during cross-KV analysis. Two NVFP4 CuTe DSL tests now state that GEMM+SwiGLU fusion supports SM100 and SM103.

Changes

Capacity scheduler guard

Layer / File(s)	Summary
Guard optional encoder-token analysis cpp/tensorrt_llm/batch_manager/capacityScheduler.cpp	Cross-context and guaranteed-no-evict prefix-reuse analysis now checks for encoder unique tokens before cross-KV processing.

NVFP4 skip messages

Layer / File(s)	Summary
Clarify architecture skip messages tests/integration/defs/accuracy/test_llm_api_pytorch.py	The single-GPU and four-GPU NVFP4 tests now identify GEMM+SwiGLU fusion and the SM100/SM103 restriction.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: bowenfu

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Out of Scope Changes check	⚠️ Warning	The changes to tests/integration/defs/accuracy/test_llm_api_pytorch.py only change NVFP4 CuTe DSL skip-message text for DeepSeekV3Lite tests. They do not implement or test the Whisper scheduler fix …	Remove the unrelated skip-message changes, or move them to a separate pull request.
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Issue #18609 reports a RuntimeError: bad optional access in the capacity scheduler for feature-based encoder inputs. The PR adds getEncoderUniqueTokens().has_value() guards at all three cross-KV p…
Title check	✅ Passed	The title clearly and concisely identifies the fix: it prevents encoder tokens from being required for cross-prefix reuse. It matches the primary code change and uses the required issue and type forma…
Description check	✅ Passed	The description clearly explains the problem, root cause, code change, scope, verification, and known limitations. It does not use the template's explicit Test Coverage or PR Checklist headings, and i…

Full details: Out of Scope Changes check

Explanation

The changes to tests/integration/defs/accuracy/test_llm_api_pytorch.py only change NVFP4 CuTe DSL skip-message text for DeepSeekV3Lite tests. They do not implement or test the Whisper scheduler fix in issue #18609. This is an unrelated test-message change in the pull request.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Create a new PR

_{Comment @coderabbitai help to get the list of available commands.}

[ravo42-maker]

Tested on the original hardware today (RTX 5070 Ti / sm_120, nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc25, openai/whisper-large-v3, 30 s 16 kHz mono clip, LLM API directly). Five runs, one variable at a time:

#	enable_block_reuse	max_input_len	max_seq_len	result
1	True (default)	1024 (default)	448	RuntimeError: bad optional access in the capacity scheduler (the original report)
2	False	1024 (default)	448	gets past the scheduler, then RequestError: The max KV cache length of input sequences (1500) exceeds the KV cache manager's maximum supported length (1024) from attention_backend/trtllm.py:754 via prepare_cross_metadata, followed by AssertionError: Sampling failed
3	False	1024	2048	No free block found. This shouldn't happen! (evictionPolicy.cpp:160) at executor init
4	False	1500	unset (infer_max_seq_len() = 1500)	works — correct transcript, 30 s clip in 3.5 s (cold, 8.7x realtime)
5	True (default)	1500	unset	bad optional access again

So the diagnosis holds: with everything else sized correctly (run 5 vs run 4), block reuse on vs off is exactly the difference between the scheduler crash and a working transcription. #19450 is aimed at the right thing.

Run 2 exposes a second, independent problem that anyone will hit right after #19450 lands: _get_cross_kv_cache_layout() in _torch/pyexecutor/_util.py sizes the cross-attention KV pool from llm_args.max_input_len, whose default is 1024 (llm_args.py:4675), and Whisper's encoder emits max_source_positions = 1500 positions. The config-attribute fallback list there (max_encoder_input_len, encoder_max_input_length, max_encoder_position_embeddings, encoder_max_position_embeddings, max_position_embeddings, n_positions) does not include max_source_positions, so nothing corrects it. The cross pool should be sized from the encoder length, not a decoder input limit. Workaround until then: pass max_input_len=1500 (and don't pass max_seq_len=448 — the model's infer_max_seq_len() already returns 1500 for the encoder pass; forcing 448 is what my original repro did wrong). Happy to open that as a separate issue if you'd prefer it tracked apart from this one.

Update — built and tested the PR itself. I applied the capacityScheduler.cpp hunk of #19450 (head 3b82b78) onto the v1.3.0rc25 tag (785c948; it applies cleanly, all three sites are identical there), built a wheel in nvcr.io/nvidia/tensorrt-llm/devel:1.3.0rc25 with scripts/build_wheel.py --cuda_architectures "120-real" --fast_build, and installed it over the release container with pip install --no-deps --force-reinstall. Same GPU, same clip, same scripts:

#	enable_block_reuse	max_input_len	rc25 stock	rc25 + #19450
5	True (default)	1500	bad optional access	works — same transcript as run 4, 30 s clip in 3.95 s cold (7.6x realtime)
1	True (default)	1024 (default)	bad optional access	past the scheduler; then the cross-KV sizing error (1500 exceeds 1024) + Sampling failed

So the guard does what it says on sm_120: block reuse stays on, the request reaches the model, and Whisper transcribes. Run 1 is the second, independent bug described above, now the only thing between a stock config and a working transcription.

[crazydemo]

Review summary - Approve

Reviewed the full diff; no blocking or major issues found.

Automated review by NVCortex Lite, run by @crazydemo.

[100milliongold]

Thank you — that is the hardware verification this PR did not have. The body says plainly that I had not run it on a GPU and that it carries no regression test. Your runs 4 and 5 isolate block reuse as the only variable, and the rc25 + #19450 build closes the gap on the guard itself. I have updated the description so that caveat no longer stands unqualified.

On the second problem: I read _get_cross_kv_cache_layout() on main and your diagnosis holds, but the missing attribute name is not the whole of it.

max_seq_len = fallback_max_seq_len or self._max_seq_len
max_input_len = getattr(self._llm_args, "max_input_len", None)
if isinstance(max_input_len, int) and max_input_len > 0:
    max_seq_len = max_input_len            # decoder limit overwrites encoder length
encoder_limit = self._get_config_int_attr(config, (...))
if encoder_limit is not None:
    max_seq_len = min(max_seq_len, encoder_limit)   # can only shrink it

max_input_len defaults to 1024 (llm_args.py:5052) and overwrites unconditionally; the encoder limit is then applied with min(), so it can only lower the result. Adding max_source_positions to that tuple on its own would therefore not fix Whisper — min(1024, 1500) is still 1024. The override is the defect; the missing name only hides it.

For openai/whisper-large-v3 the config carries max_source_positions: 1500 and neither max_position_embeddings nor n_positions, so encoder_limit is None today and the cross pool is sized at exactly 1024. That matches your run 2 error to the number.

Please do open it separately. It is a different layer — Python pool sizing, not the C++ capacity scheduler — and it should not wait on this one. If you would rather not, say so and I will file it using your run 2 as the reproduction, credited to you.

[ravo42-maker]

Agreed on the override being the defect — my patch removes it rather than just adding the attribute; the encoder's declared limit becomes authoritative and max_input_len is only a fallback when a model declares none. Filed as #19514 with the diff and the before/after runs. The repeated-request failure with reuse ON (multimodal_embed_mask_cumsum) is filed separately as #19515. Thanks for updating the description.