feat: pass an image as part of the evaluation

When llama.cpp's support for this will be stable.
Hopefully, there will be an official API for this after ggml-org/llama.cpp#11292 is implemented.

Interested in this kind of multimodal support. Any update on progress?

Does this encompass adding support for running llava models or should that be a separate feature request? I noticed that llama-cpp-python already includes llava support from llama.cpp so this shouldn't be too hard with setting up the bindings.

I haven't started working on this yet, but it is planned as part of the roadmap.
The plan is to add support for llama.cpp's ability to pass an image to a model, which right now only supports LLaVA.

I'll work on this once llama.cpp's API for this is final, to prevent frequent breaking API changes (unlike what happens on some other libraries)

Make sense. Hopefully llama.cpp finalizes that API.

Any update on this? Would love to leverage multimodal models! Love the library so far :)

An official API for this is in active development on llama.cpp; one it's ready we can start working on adding support for it in node-llama-cpp.
It appears that the API being worked on will also include support for other modalities, such as audio and video, so this is going to be a major feature once it lands (node-llama-cpp will support everything).

An official API for this is in active development on llama.cpp; one it's ready we can start working on adding support for it in node-llama-cpp.
It appears that the API being worked on will also include support for other modalities, such as audio and video, so this is going to be a major feature once it lands (node-llama-cpp will support everything).

Beautiful. Looking forward to it, this library is insane.

Any updates on this feature, it seems there were some experimental support for gemma 3 vision last week ggml-org/llama.cpp#12344

@wisng

@wisng I'm waiting for an official stable API for this, which is still in the works.

🔥 Multimodal support arrived in llama-server: ggml-org/llama.cpp#12898 | documentation

Perhaps we can't call it stable, but it is there now.

I've started poking around with the mtmd API to integrate multimodality into node-llama-cpp.
There were too many breaking changes around it recently, so I'll wait a bit more before I spend more time integrating it, but it's coming up!
Can't commit to a timeline yet, but I'll release a few beta versions with it before a stable release to gather feedback and iron out any bugs.

Any progress on this one? Huge thanks for the amazing work you're doing on this project! Greatly appreciate it.

I've got it working but it still needs more polish to be usable. I hope to finish the remaining work and release a beta version soon.

Is there anything the community could do to help you move this forward? This library is awesome, but the lack of images is limiting deployment. Given this has been open for over 2 years, I was wondering if there was anything we could do to help you polish / release the work to date?

@rogerdcarvalho

@rogerdcarvalho I'm working on node-llama-cpp v4 which revolves mostly around multimodality support (also including modalities other than just images) and it entails a big refactor, which is mostly synchronous work.
Also, the mtmd (multimodality) library in llama.cpp requires adjusting many parameters to work properly with different models, and I find it important to do all those adjustments automatically as the default so you wouldn't have to spend effort manually investigating a model just to get it running properly (just to know what to look for is a lot of effort in itself).
While this issue is open for a long time, it only became actionable quite recently with the introduction (and recent refactor and stabilization) of mtmd.

It's important to get the fundamentals right first before I can publish it as a beta that we can iterate on based on feedback.
Currently, that's the main thing I'm working on in the time I can allocate for it, but recently I don't have as much time to work on it as I'd like to.
Maintaining the current v3 with upstream changes happening in llama.cpp is the first priority, and it takes quite a lot of time already. Any help with it would greatly allow me to allocate more time to work on v4 for sure.
If you're interested in ways you can help with this, don't hesitate to open a discussion about it and I can share there some things there that would surely help out.

I'm definitely interested in ways I can help with maintaining the current v3 with upstream changes! I'll open a discussion about it.

Navigation Menu

Repository navigation

feat: pass an image as part of the evaluation #88

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

feat: pass an image as part of the evaluation #88

Description

Activity

samlhuillier commented on Nov 22, 2023

fozziethebeat commented on Dec 2, 2023

giladgd commented on Dec 3, 2023

fozziethebeat commented on Dec 5, 2023

AlexTech314 commented on Mar 27, 2025

giladgd commented on Mar 28, 2025

AlexTech314 commented on Mar 28, 2025

wisng commented on Apr 18, 2025

giladgd commented on May 9, 2025

Mihailoff commented on May 15, 2025

giladgd commented on May 16, 2025

AwesomeAnders commented on Nov 21, 2025

giladgd commented on Nov 25, 2025

rogerdcarvalho commented on Jan 8, 2026

giladgd commented on Jan 10, 2026

rogerdcarvalho commented on Jan 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions