Repository navigation
Update the llama.cpp to utilize OV frontend op translations - #342
preetha-intel wants to merge 2 commits into
Conversation
|
Thanks for contributing! Development happens on |
Add GGML decoder to support OV GGUF Frontend.
|
Could we stage this migration behind an opt-in CMake flag, keeping the existing translations as the default for now? This change replaces a substantial translation and fusion layer. Retaining the legacy path temporarily would let us compare correctness, model/device coverage, and performance, while providing an easy rollback. The GGUF frontend dependency should only be required when the new path is enabled. Once validation gives us confidence, we can switch the default and remove the legacy implementation in a follow-up PR. @ravi9, what do you think about taking this incremental approach? |
|
Sure, an incremental approach with the opt-in CMake flag sounds good. |
Add GGML decoder to support OV GGUF Frontend.
Overview
This PR removes the legacy in-tree OpenVINO translation layer from ggml-openvino and moves the backend onto the OpenVINO GGUF frontend (openvino::frontend::gguf).
Until now the backend carried its own copy of the GGML→OpenVINO conversion: a decoder interface, node context, per-op translators, graph-fusion passes and a translate session, all under ggml/src/ggml-openvino/openvino/.
After this change, the frontend owns the generic op translations and passes. The backend keeps a GgmlOvDecoder bridge that exposes the ggml graph to the frontend, plus the runtime pieces (device handling, KV cache, compile and cache logic).
Changes
CMake changes
Openvino Frontend changes PR : openvinotoolkit/openvino#38769