Skip to content

Update the llama.cpp to utilize OV frontend op translations - #342

Open
preetha-intel wants to merge 2 commits into
ravi9:dev_backend_openvinofrom
preetha-intel:ov-gguf-frontend
Open

preetha-intel wants to merge 2 commits into
ravi9:dev_backend_openvinofrom
preetha-intel:ov-gguf-frontend

Conversation

@preetha-intel

@preetha-intel preetha-intel commented Oct 12, 2026 •

Copy link
Copy Markdown

Add GGML decoder to support OV GGUF Frontend.

Overview

This PR removes the legacy in-tree OpenVINO translation layer from ggml-openvino and moves the backend onto the OpenVINO GGUF frontend (openvino::frontend::gguf).

Until now the backend carried its own copy of the GGML→OpenVINO conversion: a decoder interface, node context, per-op translators, graph-fusion passes and a translate session, all under ggml/src/ggml-openvino/openvino/.
After this change, the frontend owns the generic op translations and passes. The backend keeps a GgmlOvDecoder bridge that exposes the ggml graph to the frontend, plus the runtime pieces (device handling, KV cache, compile and cache logic).

Changes

  • Removed the legacy bridge. The whole ggml-openvino/openvino/ tree is deleted: decoder.h, frontend., input_model., node_context.h, translate_session., utils., all of op/* (about 55 translators) and the fusion passes in pass/*. These include fuse_to_sdpa, fuse_to_conv, fuse_moe_router, squeeze_matmul and the dequantization marking passes.
  • Reworked GgmlOvDecoder (ggml-decoder.cpp/.h). It is now the bridge that exposes the ggml cgraph, tensor metadata and model-level info to the GGUF frontend.
  • Simplified ggml-openvino.cpp and utils.cpp. Graph conversion now goes through the frontend, and the per-op support logic that duplicated the translators is gone.
  • Moved the KV-state sequence-axis helper. It went from openvino/pass/kv_state_seq_axis.{h,cpp} to kv-state-seq-axis.{h,cpp} in the backend directory, because it is runtime logic and not a frontend pass.

CMake changes

  • ggml-openvino now links openvino::frontend::gguf.
  • Configuration fails with a clear error if that target isn't found.
  • The source globs are no longer recursive, since the subdirectories are gone.

Openvino Frontend changes PR : openvinotoolkit/openvino#38769

@preetha-intel
preetha-intel marked this pull request as draft October 12, 2026 10:17
@mostafafaheem
mostafafaheem changed the base branch from master to dev_backend_openvino October 12, 2026 12:58
@mostafafaheem
mostafafaheem changed the base branch from dev_backend_openvino to master October 12, 2026 12:59
@mostafafaheem

Copy link
Copy Markdown
Collaborator

Thanks for contributing! Development happens on dev_backend_openvino branch. Please rebase accordingly

@ravi9
ravi9 changed the base branch from master to dev_backend_openvino October 16, 2026 16:16
@ravi9
ravi9 marked this pull request as ready for review October 16, 2026 16:46
@cavusmustafa

Copy link
Copy Markdown
Collaborator

Could we stage this migration behind an opt-in CMake flag, keeping the existing translations as the default for now?

This change replaces a substantial translation and fusion layer. Retaining the legacy path temporarily would let us compare correctness, model/device coverage, and performance, while providing an easy rollback. The GGUF frontend dependency should only be required when the new path is enabled.

Once validation gives us confidence, we can switch the default and remove the legacy implementation in a follow-up PR.

@ravi9, what do you think about taking this incremental approach?

@ravi9

ravi9 commented Oct 16, 2026

Copy link
Copy Markdown
Owner

Sure, an incremental approach with the opt-in CMake flag sounds good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants