[Alcpz]

Overview

Profiling of mtmd-cli showed that the CPU path used the non-repack codepath. This is caused by the CLIP context using default buffers (aka, not considering extra buffers), so this faster available codepath is never used.

This PR introduces a mechanism to create the encoder graph using extra buffers, which improves the performance of the encoders when they are run on CPU.

Caveats:

• DeepSeek-OCR/OCR2 are excluded from repack selection since they merge image tiles into the batch dimension at encode time. CPU_REPACK kernels don't support 4D activations at this moment. Not sure if the proper fix is the backend or with a similar solution to model : allow reshape of tensors during load #26531.
• I am limiting the implementation to the extra buffer type I could validate (CPU_REPACK). KleidiAI shows a numerical drift, so I am not confident for adding general support.

Additional information

More details into what was modified:

• A dummy graph is created to distinguish "Repackable" tensors. I've seen that CLIP uses external llama APIs to avoid the llama internal loader dependency.
• Weights are now allocated in one buffer per buffer type. Tensors placed in an extra buffer type are repacked by its tensor set API.
• If extra buffer types fail, it falls back to the default buffer type (i.e. the current behavior).
• Reuses the existing --no-repack flag for MTMD tools. use_extra_bufts added for this purpose. Defaults to ON (Repacks by default). Changes the default behavior for CPU users if they update.
• Audio generation is not scoped in this PR.

Performance comparison:

Model (mmproj)	Current	CPU_REPACK
LFM2.5-Audio-1.5B Q8_0	160 ms	110 ms
SmolVLM-500M-Instruct Q8_0	355 ms	172 ms

SmolVLM encodes each image as two batches, showing per-batch average.

# Apple M4 Max CPU backend (ARM i8mm codepath)
# Audio
./build/bin/llama-mtmd-cli -m <model> --mmproj <mmproj> --mmproj-device none --audio tools/mtmd/test-2.mp3 -p "a" -n 1 -t 8 -v [--no-repack]
# Vision
./build/bin/llama-mtmd-cli -m <model> --mmproj <mmproj> --mmproj-device none --image tools/mtmd/test-1.jpeg  -p "a" -n 1 -t 8 -v [--no-repack]

Output Parity:

Embeddings dumped with

MTMD_DEBUG_EMBEDDINGS=/tmp/[no-]repack.bin ./build/bin/llama-mtmd-debug -p encode \
    -m SmolVLM-500M-Instruct-Q8_0.gguf --mmproj mmproj-SmolVLM-500M-Instruct-Q8_0.gguf \
    --mmproj-device none --image cb -n 512 [--no-repack]
 
md5sum /tmp/repack.bin /tmp/no-repack.bin
a06dbead1bb8c087558c796ec5cc36f8  /tmp/repack.bin
a06dbead1bb8c087558c796ec5cc36f8  /tmp/no-repack.bin

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes. Draft of the code, and review.

[Alcpz]

@tdakhran