mtmd: Nemotron 3 Nano Omni (nemotron_v2_vl) encodes every image at a fixed 256 tokens — OCR effectively unusable; --image-min/max-tokens silently ignored #25317

Summary

The nemotron_v2_vl projector path encodes every image at exactly 256 tokens regardless of input resolution. The model's HF preprocessor_config.json specifies dynamic native-resolution processing with 1,024–13,312 patches (≈256–3,328 image tokens per image). Being pinned to the bottom of that range makes the model effectively text-blind on document images (A4 invoices/scans): it produces fluent, plausible hallucinated content instead of reading the page.

--image-min-tokens / --image-max-tokens have no effect for this projector (docs say they only apply to dynamic-resolution models) — but they are accepted silently, which makes the problem hard to diagnose.

Environment

• llama.cpp b9848, macOS (Apple Silicon M3 Ultra, Metal backend)
• Model: unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF (UD-Q8_K_XL) + mmproj-BF16
• Server:

  llama-server -m NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-UD-Q8_K_XL.gguf \
    --mmproj mmproj-BF16.gguf --host 0.0.0.0 --port 8089 -c 32768 --parallel 1 \
    -ngl 999 --jinja --image-min-tokens 1024 --image-max-tokens 6144
  

Reproduction / evidence

1. Send any high-resolution text image (we used A4 invoice pages rendered at 200 dpi, ~1700×2340 px) via /v1/chat/completions with an image_url data URL.
2. Check timings.prompt_n: each image contributes exactly 256 tokens, whether the input is 512 px or 2340 px, and independent of --image-min/max-tokens.
3. Transcription output: headline numbers/IBANs/invoice numbers are largely invented (e.g. a "canonical" German invoice: netto 100.00 / USt 19.00 / brutto 119.00 on documents containing none of these values).

Cross-check: on a 10-invoice field-extraction benchmark against Qwen3-VL-32B (Q8, Ollama), Nemotron under llama.cpp scored 85/100 correct fields even with a client-side tiling workaround (thumbnail + 3×4 grid of 200 dpi crops per page — this restores basic legibility but digit-level errors in long strings like IBANs remain, consistent with each tile still being squashed to a single 512×512 view). Without tiling the output is near-total hallucination. Qwen3-VL: 95/100.

Where the limitation appears to live

PROJECTOR_TYPE_NEMOTRON_V2_VL in tools/mtmd/clip.cpp reads only PROJ_SCALE_FACTOR and registers no dynamic-resolution candidates (contrast INTERNVL, which sets preproc_min_tiles=1 / max_tiles=12 via set_internvl_dhr_res_candidates()). Per the design discussion in #19547, position embeddings are pre-downsampled at GGUF conversion time for a single fixed input size — so the runtime cannot honor the model's trained 256–3,328-token dynamic range. OCR degradation on small text vs Qwen3-VL was already reported by a user in the #19547 comments.

Why this matters particularly on Apple Silicon

NVIDIA's reference stack for this model is vLLM (and TRT-LLM), which is not usable on macOS/Metal. llama.cpp is the only practical way to run this model on Apple hardware — and the fixed-resolution port disables exactly the capability (document OCR / dense text) that the model is benchmarked on (OCRBench v2 etc.). GGUF users evaluating the model will conclude the model is bad, when it is the port.

Asks

1. Dynamic-resolution / tiled preprocessing support for nemotron_v2_vl (analogous to the InternVL anyres path), or documentation that the port is fixed-res.
2. Until then: warn (or error) when --image-min-tokens/--image-max-tokens are passed for a projector that ignores them. The silent no-op cost us a day of debugging.

Happy to provide test images, full logs, or run patches on M3 Ultra hardware.