[Maphist0]

Overview

This PR adds support for the SenseNova U1 text-generation branch in llama.cpp.

Supported use cases:

• Text-to-text generation
• Image-to-text generation
• Image-plus-text-to-text generation

The image path supports SenseNova U1’s dynamic-resolution preprocessing, 2D RoPE, patch embedding, spatial merge, and image-token injection.

The implementation is intentionally limited to U1’s understanding/text-generation branch. Image generation branch is not included.

Additional information

Related to discussion #23529

Validation

Compile:

cmake -B build-u1-cuda \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=90-real \
  -DLLAMA_BUILD_SERVER=ON \
  -DCMAKE_BUILD_TYPE=Release

cmake --build build-u1-cuda -j8 \
  --target llama-cli llama-completion llama-mtmd-cli

Convert ckpt:

python convert_hf_to_gguf.py \
  <official_u1_ckpt_path> \
  --outtype bf16 \
  --outfile SenseNova-U1.5-8B-bf16.gguf

python convert_hf_to_gguf.py \
  <official_u1_ckpt_path> \
  --mmproj \
  --outtype bf16 \
  --outfile SenseNova-U1.5-8B-mmproj-bf16.gguf

Case 1: Text generation:

build-u1-cuda/bin/llama-cli \
  -m SenseNova-U1.5-8B-bf16.gguf \
  --single-turn \
  -p <YOUR_PROMPT> \
  -n 128 \
  --temp 0 \
  --seed 0 \
  -ngl all \
  -c 4096

Case 2: Image-to-text generation:

build-u1-cuda/bin/llama-cli \
  -m SenseNova-U1.5-8B-bf16.gguf \
  -mm SenseNova-U1.5-8B-mmproj-bf16.gguf \
  --image test-image.jpg \
  --single-turn \
  -p <YOUR_PROMPT> \
  -n 128 \
  --temp 0 \
  --seed 0 \
  -ngl all \
  --mmproj-offload \
  -c 4096 \
  --no-display-prompt \
  --no-show-timings

Case 3: Image-text-to-text generation:

build-u1-cuda/bin/llama-cli \
  -m SenseNova-U1.5-8B-bf16.gguf \
  -mm SenseNova-U1.5-8B-mmproj-bf16.gguf \
  --image test-image.jpg \
  --single-turn \
  -p <YOUR_PROMPT> \
  -n 128 \
  --temp 0 \
  --seed 0 \
  -ngl all \
  --mmproj-offload \
  -c 4096 \
  --no-display-prompt \
  --no-show-timings

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. I've used GPT 5.5 for majority of code generation under my review. Code generated by AI was reviewed before adding into the codebase.

[ngxson]

for vision support: mirror what gemma4uv.cpp is doing. keep it simple.

[Maphist0]

@ngxson Thank you for the constructive feedback. I’ve updated the patches to address the review comments and look forward to further discussion.

[ngxson]

mtmd part is ok

[Maphist0]

mtmd part is ok

Many thanks!

Would other reviewers (@CISC @ggerganov @JohannesGaessler) please also take a look and make some review comments when available? That would be very helpful!

[Maphist0]

@CISC Thank you very much for the constructive comments. I’ve updated the patches to address them and look forward to further discussion.