mtmd: support webp via ffmpeg

[ngxson]

Overview

Fix #27443 , fix #12410

Supersede #24217

This solution requires no cpp|h files vendor-ing, no extra wasting compile time, no bloating binary size for a feature that less than 1% users gonna use.

Based on same code path with video support.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: yes

[ngxson]

/bot review

[ggml-gh-bot]

Automated code review

Reviewing the diff for tools/mtmd/mtmd-helper.cpp and tools/mtmd/mtmd-helper.h (webp decoding via the existing ffmpeg video pipeline).

Scope gate: This is a small, single-purpose change that reuses the existing mtmd_helper_video ffmpeg/ffprobe path instead of vendoring a webp library. Linked issues exist (#27443, #12410) and it supersedes #24217. Approach is good - no new subsystem, no new dependency. No quick-reject concerns.

Security: is_webp_file bounds-checks len >= 12 before reading buf and buf+8 via memcmp, so no OOB read on short buffers. The buffer is forwarded to the ffmpeg subprocess exactly as the existing video path does; no new in-process parsing of attacker-controlled data. Safe.

Findings:

(will slow the review) is_webp_file (mtmd-helper.cpp:361) is defined unconditionally but is only referenced inside the #ifdef MTMD_VIDEO block in mtmd_helper_bitmap_init_from_buf. When MTMD_VIDEO is OFF (which happens whenever LLAMA_SUBPROCESS is OFF, see tools/mtmd/CMakeLists.txt:8-10), this becomes an unused static function and trips -Wunused-function under -Wall. The forward declaration block at lines 366-368 is correctly guarded; the definition is not. Move the is_webp_file definition inside the #ifdef MTMD_VIDEO region (or guard it with the same macro) so definition and use stay consistent.

(behavior worth confirming) Animated webp files share the same RIFF....WEBP header, so is_webp_file catches them and decode_webp_with_ffmpeg decodes only a single frame (read_next_frame is called once at mtmd-helper.cpp:864). They are then short-circuited and never reach the generic "last try: load as video" path that could decode all frames. If single-frame is the intended scope, say so in the header comment / PR description; if animated webp should be multi-frame, the detection needs to distinguish animated from still (e.g. via ffprobe nb_frames or codec_name). Either way the current silent truncation to frame 1 should be made explicit.

(robustness) decode_webp_with_ffmpeg calls probe(0.0f) (mtmd-helper.cpp:853), and probe returns false when orig_fps <= 0.0f (mtmd-helper.cpp:642). For some still-image webp inputs ffprobe can report r_frame_rate=0/0, which would make the whole webp decode fail with the generic "failed to decode webp buffer" error even though ffmpeg itself can decode the frame. Worth confirming this against the ffmpeg versions users are expected to have, or relaxing the orig_fps requirement for single-frame inputs (the fps value isn't used on the still path since fps_target is then forced to 0.0f).

Nits:

• In decode_webp_with_ffmpeg you set vctx.fps_target = 0.0f after probe() already set it to orig_fps. The overwrite is correct and the comment explains it, but the intermediate assignment in probe is dead for this caller - just confirming you're aware it relies on start_ffmpeg re-reading fps_target.

• The header comment (mtmd-helper.h:48) accurately documents the MTMD_VIDEO + ffmpeg-in-PATH requirement; consider also noting the single-frame behavior there once decided.

No correctness bugs, no lifetime/UAF issues (the local mtmd_helper_video vctx cleans up its subprocess via the subprocess_handle RAII destructor on return), and no new public API surface.

This review was generated automatically by pi coding agent using zai-org/GLM-5.2. It may contain mistakes. Maintainers make the final call.

[nikwen]

@ngxson Using ffmpeg for this is a cool idea. Much better than the other options that were proposed.

[nikwen]

@ngxson We might also need to remove this code to make the UI actually use it: https://github.com/ggml-org/llama.cpp/blob/d775b8967a46d8beb110d444aa3b8938179e0dd8/tools/ui/src/lib/utils/webp-to-png.ts

[ngxson]

it's fine to keep UI code as-is, it's too minor, zero-deps anyway