Update the llama.cpp to utilize OV frontend op translations

docs/backend/OPENVINO.md

@@ -113,13 +113,13 @@ Although, the validated models below were tested with `llama-cli` using the `Q4_
 | [bartowski/Qwen_Qwen3-1.7B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3-1.7B-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
 | [Qwen/Qwen3-4B-Q4_K_M](https://huggingface.co/Qwen/Qwen3-4B-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
 | [lm-kit/Qwen3-8B-Q4_K_M](https://huggingface.co/lm-kit/qwen-3-8b-instruct-gguf) | ✓ / ✓ | ✓ / ✓ | ✓ |
-| [bartowski/Qwen_Qwen3.5-0.8B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3.5-0.8B-GGUF) | ✓ / ✓ | ✓ / ~ | ✗ |
-| [bartowski/Qwen_Qwen3.5-2B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3.5-2B-GGUF) | ✓ / ✓ | ✓ / ~ | ✗ |
-| [bartowski/Qwen_Qwen3.5-4B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3.5-4B-GGUF) | ✓ / ✓ | ✓ / ~ | ✗ |
-| [lmstudio-community/Qwen3.5-9B-Q4_K_M](https://huggingface.co/lmstudio-community/Qwen3.5-9B-GGUF) | ✓ / ✓ | ✓ / ~ | ✗ |
+| [bartowski/Qwen_Qwen3.5-0.8B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3.5-0.8B-GGUF) | ✓ / ✓ | ✓ / ✓ | ✗ |
+| [bartowski/Qwen_Qwen3.5-2B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3.5-2B-GGUF) | ✓ / ✓ | ✓ / ✓ | ✗ |
+| [bartowski/Qwen_Qwen3.5-4B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3.5-4B-GGUF) | ✓ / ✓ | ✓ / ✓ | ✗ |
+| [lmstudio-community/Qwen3.5-9B-Q4_K_M](https://huggingface.co/lmstudio-community/Qwen3.5-9B-GGUF) | ✓ / ✓ | ✓ / ✓ | ✗ |
 |  |  |  |  |
 | [unsloth/gemma-3-4b-it-Q4_K_M](https://huggingface.co/unsloth/gemma-3-4b-it-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
-| [bartowski/google_gemma-4-E2B-it-Q4_K_M](https://huggingface.co/bartowski/google_gemma-4-E2B-it-GGUF) | ✓ / ✓ | ✓ / ~ | ~ |
+| [bartowski/google_gemma-4-E2B-it-Q4_K_M](https://huggingface.co/bartowski/google_gemma-4-E2B-it-GGUF) | ✓ / ✓ | ✓ / ✓ | ~ |
 | [bartowski/google_gemma-4-E4B-it-Q4_K_M](https://huggingface.co/bartowski/google_gemma-4-E4B-it-GGUF) | ✓ / ✓ | ✗ / ✗ | ✓ |
 | [bartowski/gemma-4-12B-it-Q4_K_M](https://huggingface.co/bartowski/gemma-4-12B-it-GGUF) | ✓ / ✓ | ✓ / ✓ | ✗ |
 |  |  |  |  |
@@ -722,7 +722,10 @@ Boolean flags follow a uniform convention: set to a **positive integer** (e.g. `
 | `GGML_OPENVINO_STATEFUL_EXECUTION`| Boolean   | `0`        | Keep KV and supported recurrent caches inside the model. Single-slot CPU/GPU execution only.               |
 | `GGML_OPENVINO_DISABLE_CACHE`     | Boolean   | `0`        | Disable the in-process compiled-model / decoder cache (cache is on by default). Set to `1` to disable.      |
 | `GGML_OPENVINO_DISABLE_KV_SLICE`  | Boolean   | `0`        | Disable the KV-cache input-tensor slicing optimization (slicing is on by default on CPU/GPU). Set to `1` to disable. |
+| `GGML_OPENVINO_DISABLE_KV_4D`     | Boolean   | `0`        | GPU, stateless: keep the KV cache in the `[1, 1, ctx, n_heads_kv * head_size]` layout instead of `[1, ctx, n_heads_kv, head_size]`, which lets the GPU plugin update and read the cache in place. Set to `1` to disable. |
 | `GGML_OPENVINO_DISABLE_KV_STATE_RELAYOUT` | Boolean | `0`     | Disable the stateful KV-state sequence-axis relayout (relayout is on by default). It moves the KV state sequence axis from dim 1 to dim 2, so the GPU plugin can append new tokens in place instead of copying the whole state every token, and the reader side no longer transposes the whole accumulated state. Set to `1` to disable. |
+| `GGML_OPENVINO_DISABLE_SHAPE_FROM_MASK` | Boolean | `0` | Dynamic CPU/GPU graphs read the active sequence count and the attended KV length from the shape of the KQ mask inside the model, so the GPU plugin can infer the attention shapes without waiting for input values and selects its fused SDPA kernel. Set to `1` to pass them as separate inputs instead. |
+| `GGML_OPENVINO_DISABLE_SCALE_REORDER` | Boolean | `0`      | GPU/NPU: a constant scale (Granite's residual and attention multipliers) is applied before the MatMul it follows instead of after it, and a small attention scale is folded into Q. The result is identical in exact arithmetic, but the f16 plugins no longer overflow on blocks whose unscaled result is out of range (e.g. Granite 4.0 350M/1B printed `?` on GPU and NPU). It is not applied on CPU, which runs in f32. Set to `1` to disable. |
 | `GGML_OPENVINO_MANUAL_GQA_ATTN`   | Boolean   | device-based | Tri-state. When **unset**, manual GQA attention is enabled by default on `GPU` and disabled on other devices. Set to a positive integer to force-enable, or `0` to force-disable. |
 | `GGML_OPENVINO_MEMORY_OPTIMIZE`   | Boolean   | `0`        | Umbrella switch for compile-time memory reductions. Enables `GGML_OPENVINO_REDUCE_COMPILE_MEM` and, on GPU, `GGML_OPENVINO_RELEASE_WEIGHTS` unless those fine-grained variables are explicitly set. |
 | `GGML_OPENVINO_REDUCE_COMPILE_MEM`| Boolean   | inherits from `GGML_OPENVINO_MEMORY_OPTIMIZE` | Reduce compile-time host memory use by streaming weight requantization and avoiding extra weight-node materialization where possible. Set explicitly to override the umbrella switch. |

ggml/src/ggml-openvino/CMakeLists.txt

@@ -1,17 +1,21 @@
 find_package(OpenVINO REQUIRED COMPONENTS Runtime Threading)
 find_package(OpenCL REQUIRED)
 
+if(NOT TARGET openvino::frontend::gguf)
+    message(FATAL_ERROR "OpenVINO GGUF frontend target openvino::frontend::gguf is required")
+endif()
+
 message(STATUS "Found OpenVINO: ${OpenVINO_DIR} (found version \"${OpenVINO_VERSION}\")")
 
-file(GLOB_RECURSE GGML_HEADERS_OPENVINO "*.h" "*.hpp")
-file(GLOB_RECURSE GGML_SOURCES_OPENVINO "*.cpp")
+file(GLOB GGML_HEADERS_OPENVINO "*.h" "*.hpp")
+file(GLOB GGML_SOURCES_OPENVINO "*.cpp")
 
 ggml_add_backend_library(ggml-openvino
     ${GGML_SOURCES_OPENVINO}
     ${GGML_HEADERS_OPENVINO}
 )
 
-target_link_libraries(ggml-openvino PRIVATE openvino::runtime openvino::threading OpenCL::OpenCL)
+target_link_libraries(ggml-openvino PRIVATE openvino::runtime openvino::threading openvino::frontend::gguf OpenCL::OpenCL)
 
 # the OpenVINO RTTI macros take one argument and leave __VA_ARGS__ empty, which -Wpedantic reports
 if (CMAKE_CXX_COMPILER_ID MATCHES "Clang" OR CMAKE_CXX_COMPILER_ID STREQUAL "IntelLLVM")

ggml/src/ggml-openvino/README.md

@@ -29,7 +29,7 @@ GGML_OPENVINO_COMPILED_MODEL_CACHE_ONLY=1 \
 
 Cache-only mode allocates backend address space without filling weight pages. On Windows, this also uses system commit capacity. The model-buffer size in the loader log is this virtual size. Weight uploads only record source identity; they do not read or requantize the weights. Graph conversion and compilation are skipped. Runtime buffers are still allocated and populated normally.
 
-Cache-only mode uses the settings provided by the current process. Keep these values exactly the same, including set versus unset: `GGML_OPENVINO_REQUANT_KQUANT`, `GGML_OPENVINO_NATIVE_SOFTPLUS`, `GGML_OPENVINO_DISABLE_KV_SLICE`, `GGML_OPENVINO_MANUAL_GQA_ATTN`, `GGML_OPENVINO_STATEFUL_EXECUTION`, `GGML_OPENVINO_DISABLE_KV_STATE_RELAYOUT`, `GGML_OPENVINO_DISABLE_REMOTE_OUTPUTS`, `GGML_OPENVINO_REDUCE_COMPILE_MEM`, `GGML_OPENVINO_MEMORY_OPTIMIZE`, `GGML_OPENVINO_PROFILING`, and `GGML_OPENVINO_DEBUG_NODE`. On GPU, also repeat `GGML_OPENVINO_MOE_OP=0` if used. `GGML_OPENVINO_SPILL_DIR` is optional on the first run and ignored in cache-only mode; host-weight release is disabled in cache-only mode.
+Cache-only mode uses the settings provided by the current process. Keep these values exactly the same, including set versus unset: `GGML_OPENVINO_REQUANT_KQUANT`, `GGML_OPENVINO_NATIVE_SOFTPLUS`, `GGML_OPENVINO_DISABLE_KV_SLICE`, `GGML_OPENVINO_MANUAL_GQA_ATTN`, `GGML_OPENVINO_STATEFUL_EXECUTION`, `GGML_OPENVINO_DISABLE_KV_STATE_RELAYOUT`, `GGML_OPENVINO_DISABLE_REMOTE_OUTPUTS`, `GGML_OPENVINO_REDUCE_COMPILE_MEM`, `GGML_OPENVINO_MEMORY_OPTIMIZE`, `GGML_OPENVINO_PROFILING`, `GGML_OPENVINO_DISABLE_SHAPE_FROM_MASK`, `GGML_OPENVINO_DISABLE_SCALE_REORDER`, and `GGML_OPENVINO_DEBUG_NODE`. On GPU, also repeat `GGML_OPENVINO_MOE_OP=0` if used. `GGML_OPENVINO_SPILL_DIR` is optional on the first run and ignored in cache-only mode; host-weight release is disabled in cache-only mode.
 
 A missing or incompatible graph fails with an error instead of compiling with absent weights. The fingerprint uses the dynamic graph's topology, ports, model parameters, weights, settings, and OpenVINO version; changing only dynamic token or KV sizes does not require a new entry. A different workload can still require another graph; populate it first without cache-only mode.
 

ggml/src/ggml-openvino/align-eltwise-ranks.cpp

@@ -0,0 +1,91 @@
+#include "align-eltwise-ranks.h"
+
+#include <numeric>
+#include <openvino/op/add.hpp>
+#include <openvino/op/constant.hpp>
+#include <openvino/op/divide.hpp>
+#include <openvino/op/multiply.hpp>
+#include <openvino/op/power.hpp>
+#include <openvino/op/sqrt.hpp>
+#include <openvino/op/subtract.hpp>
+#include <openvino/op/unsqueeze.hpp>
+#include <openvino/pass/pattern/op/wrap_type.hpp>
+
+namespace ggml_openvino {
+
+namespace {
+
+// True for an RMS-norm output, x * (1 / sqrt(mean(x^2) + eps)) or x * pow(mean(x^2) + eps, -0.5), optionally scaled
+// by the norm weight.
+bool is_rms_norm_output(const ov::Output<ov::Node> & value, int depth = 1) {
+    const auto * node = value.get_node();
+    if (!ov::is_type<ov::op::v1::Multiply>(node)) {
+        return false;
+    }
+    for (const auto & input : node->input_values()) {
+        const auto * src = input.get_node();
+        if (ov::is_type<ov::op::v1::Divide>(src) && ov::is_type<ov::op::v0::Sqrt>(src->get_input_node_ptr(1))) {
+            return true;
+        }
+        if (ov::is_type<ov::op::v1::Power>(src) && ov::is_type<ov::op::v0::Constant>(src->get_input_node_ptr(1))) {
+            return true;
+        }
+        if (depth > 0 && is_rms_norm_output(input, depth - 1)) {
+            return true;
+        }
+    }
+    return false;
+}
+
+}  // namespace
+
+AlignEltwiseOperandRanks::AlignEltwiseOperandRanks() {
+    auto eltwise_m = ov::pass::pattern::wrap_type<ov::op::v1::Add, ov::op::v1::Multiply, ov::op::v1::Subtract>();
+
+    const auto callback = [this](ov::pass::pattern::Matcher & m) {
+        auto node = m.get_match_root();
+        if (node->get_input_size() != 2) {
+            return false;
+        }
+
+        auto lhs = node->input_value(0);
+        auto rhs = node->input_value(1);
+
+        // A tensor-vs-Constant mismatch is the norm's own eps / 1-over-sqrt arithmetic, which folds into the `rms`
+        // primitive itself instead of becoming a fused eltwise post-op, so it is not affected.
+        if (ov::is_type<ov::op::v0::Constant>(lhs.get_node()) || ov::is_type<ov::op::v0::Constant>(rhs.get_node())) {
+            return false;
+        }
+
+        const auto lhs_rank = lhs.get_partial_shape().rank();
+        const auto rhs_rank = rhs.get_partial_shape().rank();
+        if (lhs_rank.is_dynamic() || rhs_rank.is_dynamic() || lhs_rank == rhs_rank) {
+            return false;
+        }
+
+        const size_t shorter_idx = lhs_rank.get_length() < rhs_rank.get_length() ? 0 : 1;
+        const auto & shorter = shorter_idx == 0 ? lhs : rhs;
+
+        // The defect is with the norm output as the higher-rank operand. When the norm output is the lower-rank
+        // one, unsqueezing it puts the Unsqueeze between `rms` and its post-op, and the plugin then computes the
+        // layer wrongly.
+        if (is_rms_norm_output(shorter)) {
+            return false;
+        }
+        const int64_t diff = std::abs(lhs_rank.get_length() - rhs_rank.get_length());
+
+        std::vector<int64_t> axes(static_cast<size_t>(diff));
+        std::iota(axes.begin(), axes.end(), 0);
+        auto unsqueeze = std::make_shared<ov::op::v0::Unsqueeze>(
+            shorter, ov::op::v0::Constant::create(ov::element::i64, ov::Shape{ axes.size() }, axes));
+
+        node->input(shorter_idx).replace_source_output(unsqueeze->output(0));
+        register_new_node(unsqueeze);
+        return true;
+    };
+
+    register_matcher(std::make_shared<ov::pass::pattern::Matcher>(eltwise_m, "ggml_openvino::AlignEltwiseOperandRanks"),
+                     callback);
+}
+
+}  // namespace ggml_openvino

ggml/src/ggml-openvino/align-eltwise-ranks.h

@@ -0,0 +1,25 @@
+#pragma once
+
+#include "openvino/pass/matcher_pass.hpp"
+
+namespace ggml_openvino {
+
+// Gives a binary eltwise op's two operands the same rank, by unsqueezing leading axes onto the shorter one.
+//
+// This is semantically a no-op (NUMPY broadcasting already left-pads the lower-rank operand with 1s, and the result
+// rank is max(rank_a, rank_b) either way). It exists purely to work around an OpenVINO GPU-plugin defect: an eltwise
+// op whose operands differ in rank is computed incorrectly once the plugin fuses it as a post-op into an `rms`
+// primitive (e.g. a rank-3 residual added to a rank-4 RMS-norm output; the model degenerates into repeated tokens on
+// GPU while CPU is correct). Equalising the ranks keeps the fusion and makes it compute the right answer.
+//
+// Only applies when both operands are real tensors: a tensor-vs-Constant mismatch is the RMS norm's own eps/rsqrt
+// arithmetic, which folds into the `rms` primitive and is not affected. The norm output itself is never unsqueezed:
+// when it is the lower-rank operand, that breaks the fused path instead. Registered for GPU only, disabled by
+// GGML_OPENVINO_DISABLE_ELTWISE_RANK_ALIGN. Remove once the plugin guards that fusion.
+class AlignEltwiseOperandRanks : public ov::pass::MatcherPass {
+public:
+    OPENVINO_MATCHER_PASS_RTTI("ggml_openvino::AlignEltwiseOperandRanks")
+    AlignEltwiseOperandRanks();
+};
+
+}  // namespace ggml_openvino