support GLM-4.5V (108B VLM)

[ddh0]

Implementation of GLM-4.5V in llama.cpp

The architecture is Glm4vMoeForConditionalGeneration ("model_type": "glm4v_moe"). Internally, this consists of an LLM (text model) and a ViT (vision adapter / multimodal projector):

LLM (text model glm4v_moe_text)

• Based on GLM-4.5-Air
• Tensor names start with model.language_model.
• Uses a "multimodal 3D RoPE" - in apply_multimodal_rotary_pos_emb, it applies rotary embeddings across temporal, height, and width dimensions for visual tokens

ViT (vision adapter glm4v_moe)

• Adapted from apple/aimv2-huge-patch14-336:
  • Architecture Aimv2VisionModel
  • ~681M params
  • 24 layers
  • hidden_size (n_embd): 1536
  • intermediate_size (n_ff): 4096
  • image_size: 336
  • patch_size: 14
  • num_channels: 3
  • depth: 24
• Tensor names start with model.visual.
• Its 2D positional embeddings are dynamically adapted via bicubic interpolation within the Glm4vMoeVisionEmbeddings module to handle varied image resolutions
• It also applies its own rotary position embeddings within the self-attention blocks (via apply_rotary_pos_emb_vision)

Other notes:

• Native context length is 65,536 (as opposed to 131,072 for GLM-4.5-Air)
• RoPE theta (θ): 10,000.0 (as opposed to 100,000.0 for GLM-4.5-Air)
• The model supports video input, but I currently do not plan to support video input in this PR (images only)
• Tokenizer has video-related special tokens - need to handle these during conversion

References:

• The 🤗 reference implementations:
  • modeling_glm4v_moe.py
  • modular_glm4v_moe.py
• The 🤗 model card
• The 🤗 config.json

See also:

• arXiv:2507.01006
• arXiv:2411.14402

[ddh0]

Moved to ggml-org#16600