[boshjerns]

Overview

The Gemma 4 vision encoder (PROJECTOR_TYPE_GEMMA4V) runs its MLP with gelu_quick, but the model's vision_config.hidden_activation is gelu_pytorch_tanh (same in the E2B, E4B, 26B-A4B and 31B configs).

clip.cpp falls back to FFN_GELU_QUICK when an mmproj has neither use_gelu nor use_silu. Gemma4VisionAudioModel in conversion/gemma.py doesn't write either key, so Gemma 4 vision mmproj files without the key get gelu_quick. Gemma 3 writes use_gelu in its converter, so it isn't affected.

On photos the image embeddings drift from transformers: on E2B the worst image token has a cosine of 0.33. Running transformers with quick_gelu instead reproduces llama.cpp's current output (mean cosine 0.99998 or higher on the 4 images I checked).

Changes:

• The Gemma 4 converter now writes use_gelu and raises NotImplementedError for any other vision_config.hidden_activation.
• The FFN activation choice moves into clip_get_ffn_op(). For GEMMA4V files that have neither use_gelu nor use_silu it returns FFN_GELU, so existing files work without reconversion. A key that is set is always honored, and other projectors behave as before.
• test-mtmd-impl gets test_clip_ffn_op, which checks the choice on synthetic GGUF metadata (no model needed). The GEMMA4V case with no key fails without the fallback and passes with it.

The converter also writes use_gelu into GEMMA4UV files, but GEMMA4UV and the Gemma 4 audio encoder compute the same as before, because their graphs don't use ffn_op.

To reproduce:

1. Convert an F32 mmproj:
   python convert_hf_to_gguf.py google/gemma-4-E2B-it --remote --mmproj --outtype f32 --outfile mmproj-gemma-4-E2B-it-F32.gguf
2. Resize a photo to the size transformers uses, so preprocessing is identical on both sides. tools/mtmd/test-1.jpeg (640x488) becomes 912x672 with the default max_soft_tokens=280, which is 266 image tokens:
   python -c "from PIL import Image; Image.open('tools/mtmd/test-1.jpeg').resize((912, 672), Image.BICUBIC).save('img.png')"
3. Dump the image embeddings:
   MTMD_DEBUG_EMBEDDINGS=out.bin llama-mtmd-cli -hf ggml-org/gemma-4-E2B-it-GGUF:Q8_0 --mmproj mmproj-gemma-4-E2B-it-F32.gguf --image img.png --image-min-tokens 266 --image-max-tokens 266 --jinja -p "Describe this image." -n 1
4. Compare with transformers:

import numpy as np, torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText

mid = "google/gemma-4-E2B-it"
proc = AutoProcessor.from_pretrained(mid)
model = AutoModelForImageTextToText.from_pretrained(mid, dtype=torch.float32).eval()
inp = proc.image_processor(images=[Image.open("img.png").convert("RGB")], return_tensors="pt")
with torch.no_grad():
    f = model.model.get_image_features(pixel_values=inp["pixel_values"], image_position_ids=inp["image_position_ids"])
ref = f.pooler_output[0].float().numpy()
ref = ref.reshape(-1, ref.shape[-1])

raw = open("out.bin", "rb").read()
n, d = np.frombuffer(raw[:8], np.int32)
emb = np.frombuffer(raw[8:], np.float32).reshape(n, d)
cos = (emb * ref).sum(1) / (np.linalg.norm(emb, axis=1) * np.linalg.norm(ref, axis=1))
print(f"{n} tokens, mean cosine {cos.mean():.6f}, worst token {cos.min():.4f}")

On that image (266 tokens):

	mean cosine	worst token
master, mmproj converted on master (no activation key)	0.990677	0.3262
this PR, same old mmproj	0.999974	0.9952
this PR, mmproj converted with this PR	0.999974	0.9952

The old file and the reconverted file give byte-identical embeddings with this PR.

Additional information

Per-token cosine against transformers fp32 on E2B (max_soft_tokens=280), 7 images resized to the transformers size. llama.cpp and transformers produce the same token count on all 7, so only the encoder differs:

mmproj	master mean / worst token	this PR mean / worst token
E2B F32, converted from google/gemma-4-E2B-it	0.99723 / 0.326	0.9999974 / 0.9990
E2B F16, converted	0.99720 / 0.326	0.9999942 / 0.9957

I only measured E2B. The other sizes have the same vision activation and take the same path.

Checks: CPU build with the ci/run.sh flags in Debug and Release (LLAMA_FATAL_WARNINGS=ON, no warnings), ctest -L main in both with the ci/run.sh exclusions (test-opt and test-llama-archs in Debug, test-opt in Release; test-tokenizers-ggml-vocabs fails locally only because git-lfs isn't installed), flake8 and ty on conversion/gemma.py (no new diagnostics), and editorconfig-checker.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. Claude and Codex assisted with the investigation, the change, verification and this description.

[CISC]

Deferring to @ngxson if test is wanted.

[ngxson]

the whole PR can be just 2 lines fix. see existing code before coding anything

hparams.ffn_op = FFN_GELU;
log_ffn_op = "gelu";

[ngxson]

test is useless

[ngxson]

see #30852

[nikwen]

Thanks for finding the inconsistency!

[boshjerns]

No problem! I will keep the code tighter for next fixes