[Feature]: Support Dense Multi-Modal Qwen3.5 (e.g Qwen3.5-9b) unified HF checkpoints (ModelOpt NVFP4/FP8) in trtllm-serve PyTorch/AutoDeploy backends #11947

🚀 The feature, motivation and pitch

Summary

Please add first-class support for serving Qwen3.5 checkpoints (especially unified HF exports from NVIDIA ModelOpt, including NVFP4/FP8 mixed precision) with trtllm-serve on PyTorch and AutoDeploy backends,
without requiring local patching.

This currently fails on TensorRT-LLM 1.3.0rc6 with multiple compatibility/runtime issues.

Environment

• TensorRT-LLM: 1.3.0rc6
• GPU: Blackwell RTX 50 series
• CUDA: 13.1
• Python: 3.12
• Model: local Qwen3.5 checkpoint exported via ModelOpt unified HF format
• Checkpoint path example:
  • /home/yk/AI/nvidia-modelopt/qwen3.5-9b-vllm-nvfp4-mlp-only-kvnone-ptq
• Quant metadata present:
  • hf_quant_config.json with NVFP4 settings

What I expect
trtllm-serve should load and serve Qwen3.5 checkpoints directly (like Qwen3 path), including:

• model_type: qwen3_5 config recognition
• architecture mapping for Qwen3_5ForConditionalGeneration (and/or text-only Qwen3.5 variants)
• ModelOpt unified HF quant config ingestion for NVFP4/FP8
• stable operation for both:
  • --backend pytorch
  • --backend _autodeploy

What happens today

A) --backend pytorch

After resolving CLI/YAML schema issues, load fails with:

• unknown architecture in TRT-LLM mapping:
  • Unknown architecture for AutoModelForCausalLM: Qwen3_5ForConditionalGeneration
• or earlier (with pinned transformers):
  • model type qwen3_5 not recognized

B) --backend _autodeploy

Progresses further but fails in export patch pipeline:

• Patch bamba failed to apply
• expects BambaModel._update_causal_mask, but current Transformers path exposes _update_mamba_mask

Also saw other API-drift issues when using newer Transformers (5.x), indicating tight coupling to older internal HF APIs.

Minimal reproduction

1. Create YAML for serve options:

max_seq_len: 16384
max_num_tokens: 2048
max_batch_size: 512
tensor_parallel_size: 1
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.90
tokens_per_block: 64

2. Run (PyTorch backend):

trtllm-serve /path/to/qwen3.5-checkpoint
--backend pytorch
--trust_remote_code
--port 8000
--extra_llm_api_options /tmp/qwen35_trtllm_serve.yaml

3. Run (AutoDeploy backend):

trtllm-serve /path/to/qwen3.5-checkpoint
--backend _autodeploy
--trust_remote_code
--port 8000
--extra_llm_api_options /tmp/qwen35_autodeploy.yaml

Observed error signatures

• Unknown architecture for AutoModelForCausalLM: Qwen3_5ForConditionalGeneration
• Patch bamba failed to apply
• AttributeError: type object 'BambaModel' has no attribute '_update_causal_mask'. Did you mean: '_update_mamba_mask'?

Why this matters
Qwen3.5 is a major target for inference, and ModelOpt unified HF export explicitly aims to be framework-portable. Right now, serving this class of checkpoints in TRT-LLM requires fragile local patching and
breaks quickly across Transformers versions.

Requested feature scope

1. Add Qwen3.5 architecture/config support in TRT-LLM model registry for serve path.
2. Ensure Qwen3_5ForConditionalGeneration (and relevant text-only variants) are routable in AutoModelForCausalLM / multimodal factory logic.
3. Make AutoDeploy patch stack robust to HF API changes (e.g., Bamba method name changes).
4. Officially document supported Transformers range for Qwen3.5 in TRT-LLM release notes.
5. Add CI coverage:

• Qwen3.5 load test with trtllm-serve --backend pytorch
• same for --backend _autodeploy
• with a representative ModelOpt quantized checkpoint config (NVFP4/FP8 metadata)

Nice to have

• a dedicated example config for local Qwen3.5 checkpoints (non-HF-hub path)
• clearer error when model architecture is close-but-unsupported, with actionable guidance

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.