Verify and add tests for quantization support

The GraniteSwitch model loading should be evaluated in quantized mode:

Add quantization tests for HF backend

vllm quantization issues using BnB:

BnB dtype error in SwitchedLoRALinear - When BitsAndBytes quantizes base layers, the weight dtype becomes uint8. SwitchedLoRALinear.init used base_layer.weight.dtype to allocate LoRA params, causing them to be uint8 instead of float which breaking LoRA computation.
OOM on A100 80GB with BnB loading - BnB holds full-precision weights during quantization pass. Default gpu_memory_utilization=0.8 caused OOM.

Fix BnB quantization dtype error in vLLM SwitchedLoRALinear

Navigation Menu