Move llama.cpp to v0.9.0 and audit its CPU ISA source pair

.github/workflows/validate_wrapper.yml

@@ -18,7 +18,8 @@ env:
   # commit is the submodule pin builds that checkout as is (tools/ci_scope.py).
   LLAMADART_UPSTREAMS: >-
     {"v0.7.0": "620d2225cc436ab0b001e3200e590e3b286ef9ab",
-    "v0.8.0": "8642541b255bfb5dc39cd351a178be09013a55d6"}
+    "v0.8.0": "8642541b255bfb5dc39cd351a178be09013a55d6",
+    "v0.9.0": "cd959bae99ff4bed22bef2604f45241c09bbb8c8"}
 
 jobs:
   changes:

AGENTS.md

@@ -60,6 +60,9 @@ See `docs/v080_qualification.md` for the v0.8.0 candidate, its wrapper gate,
 carried-patch status and consumer ABI delta.
 See `docs/v080_android_isa_qualification.md` for the v0.8.0/KleidiAI v1.32.0
 source pair, its SVE I8MM-gated Q8 kernels and remaining gates.
+See `docs/v090_qualification.md` for the v0.9.0 candidate, retired patch 0004
+and consumer ABI delta, and `docs/v090_android_isa_qualification.md` for its
+source pair and ggml SVE256 repack paths.
 See `docs/v080_1_wrapper_rebuild.md` for the `v0.8.0-1` Metal kernel patch and
 the OS floor check `metal_os_floor.yml`, which runs on `macos-15`, `macos-26`
 and the oldest iOS simulator because ggml compiles Metal source on the user's

docs/native174_f16_dot_accumulation.md

@@ -92,8 +92,8 @@ conversion tile (+270% time) because its accumulators exceed the register file.
 ## Before publication
 
 These gates hold before any release carries the patches, including the
-`v0.9.0` release that the daily stable detector publishes once its ISA audit
-passes on `main`:
+`v0.9.0` release, which a maintainer dispatches manually once the v0.9.0 pin
+is on `main`:
 
 - Run the test against each published CPU module: Android variants on a
   device, Linux variants, Apple.

docs/platform_backend_strategy.md

@@ -132,8 +132,8 @@ Version-specific source and function-range evidence for v0.7.0 lives in
     which KleidiAI compiles with `-march=armv8.2-a+sve+sve2`. The baseline ISA
     audit reads `.dynsym` and cannot attribute a local routine, so it fails.
     GCC 13 emits no SVE in that file. ggml reaches the routine only through
-    `q4_k_rhs_pack`, whose one table entry requires `SME2 | FP16` (v0.7.0 and
-    v0.8.0; KleidiAI v1.31.0 and v1.32.0 have the same file). That gate does
+    `q4_k_rhs_pack`, whose one table entry requires `SME2 | FP16` (v0.7.0 to
+    v0.9.0; KleidiAI v1.31.0 and v1.32.0 have the same file). That gate does
     not make the SVE safe: the routine runs outside streaming mode, and SME2
     does not imply SVE there. On an Apple M4 Max (SME2, no FEAT_SVE) this
     instruction raises SIGILL outside streaming mode and runs inside it.

docs/v090_android_isa_qualification.md

@@ -0,0 +1,119 @@
+# v0.9.0 Android CPU ISA qualification
+
+Candidate upstream is stable v0.9.0
+`cd959bae99ff4bed22bef2604f45241c09bbb8c8`, compared with audited v0.8.0
+`8642541b255bfb5dc39cd351a178be09013a55d6`. KleidiAI stays at v1.32.0 and no
+dispatch input changes, so v0.9.0 grants exactly the v0.8.0 function set. Its
+complete source subtree pair is:
+
+- ggml/src: `fead7601e3b35651ee62764c76f28b50ee564160b543ff79be6fcc81837216c4`
+- kai: `fd5b9ca98d07b81671eca377c258da74e72173f92f3cab0acdf6c17ab870bd9d` (v0.8.0's)
+
+The 2026-11-05 automatic stable dispatch of v0.9.0
+([run 42016701448](https://github.com/leehack/llamadart-native/actions/runs/42016701448))
+failed only where this pair was unaudited: the Linux arm64 Vulkan core lane and
+both Android arm64 lanes. It published nothing. This document does not claim a
+published or device-qualified v0.9.0 runtime.
+
+## Source and dispatch audit
+
+v0.9.0 is v0.8.0 plus 150 commits. These are unchanged: the KleidiAI release
+and archive MD5 in CPU CMake, `ggml-cpu/kleidiai/` (tables, probes, SME
+override, thread cap, fallback mask), `arch/arm/cpu-feats.cpp`,
+`arch-fallback.h`, `ggml-cpu.cpp`, the generic `repack.cpp`/`repack.h`
+selection, `arch/arm/quants.c` and `ggml-backend-reg.cpp` scoring and loading.
+No feature probe, selector, KleidiAI call or CMake flag changes.
+
+The only ARM change is `arch/arm/repack.cpp`, which adds three ggml-owned
+scalable paths, each behind `__ARM_FEATURE_SVE` and a runtime 256-bit vector
+check:
+
+| Function | Compile guard | Runtime guard |
+| --- | --- | --- |
+| `ggml_gemv_mxfp4_4x4_q8_0` | SVE | `svcntb() * 8 == 256` |
+| `ggml_gemv_q8_0_4x8_q8_0` | SVE | `ggml_cpu_get_sve_cnt() == QK8_0` |
+| `ggml_gemm_q6_K_8x8_q8_K` | SVE and `__ARM_FEATURE_MATMUL_INT8` | `svcntb() * 8 == 256` |
+
+`armv8.2-a+dotprod+fp16` and every other non-SVE variant (Android
+`armv8.0_1` to `armv8.6_1`, Linux arm64 baseline, `linux_armv8.2_2` and
+`linux_armv8.6_1`) do not define `__ARM_FEATURE_SVE`, so these paths are not
+compiled there and their NEON paths are unchanged. They compile only into the
+Android SVE variants (`armv9.0_1`, `armv9.2_1`, `armv9.2_2`), which ggml scores
+only on CPUs reporting SVE/SVE2, and they run only with 256-bit vectors. Android
+SVE devices with 128-bit vectors take the existing NEON path. The `q6_K` GEMM's
+`svmmla` is SVE I8MM, which the variant scores infer from NEON I8MM plus SVE;
+v0.8.0's existing SVE I8MM paths in `arch/arm/quants.c` and `repack.cpp`
+already rely on that.
+
+The other CPU changes add no scalable code to non-SVE builds: OCP FP8 E4M3
+(`GGML_TYPE_F8_E4M3`) conversion, dup and `ggml_vec_dot_f8_e4m3_f32`, whose
+SIMD loop is compiled only without `__ARM_FEATURE_SVE`; `kv_rows` support in
+flash attention; empty batched-copy buffer callbacks; and RISC-V guards changed
+from `__riscv_v_intrinsic` to `__riscv_v`. The rest of `ggml/src` is
+accelerator backends (CUDA, Metal, OpenCL, SYCL, Vulkan and others), the meta
+backend, batched tensor copies in `ggml-backend.cpp`, and FP8 in `ggml.c` and
+`ggml-quants.c`, none of which is in `libggml-cpu.so`. This review covers ISA
+containment only, not numerical changes or accelerator backends.
+
+## Compiled evidence
+
+The production `build_android_arm64_cpu_variant` helper built
+`android_armv8.2_2` (`armv8.2-a+dotprod+fp16`, KleidiAI on, `GGML_NATIVE=OFF`)
+with NDK `28.2.13676358` on macOS arm64. Before the policy change the audit
+failed with the same unaudited-pair error as the release. With the v0.9.0 pair
+it passes: 265,940 instructions (v0.8.0: 261,765), with all 3,130 scalable
+instructions in the same 22 exact functions. As on v0.8.0, the 20 v0.7.0
+functions hold 2,542 of them and the two SVE Q8_0 kernels 201 and 387.
+
+```bash
+LLVM="$ANDROID_NDK_HOME/toolchains/llvm/prebuilt/darwin-x86_64/bin"
+python3 tools/validate_android_cpu_isa.py \
+  --objdump "$LLVM/llvm-objdump" --readelf "$LLVM/llvm-readelf" \
+  --llama-source third_party/llama.cpp \
+  --kleidiai-source build/android-isa/_deps/kleidiai-src \
+  build/android-isa/bin/libggml-cpu.so
+```
+
+`llamadart_kleidiai_dispatch_test` against v0.9.0 checked all 128 feature masks
+and the 12 quantized matmuls against the scalar reference in each run:
+
+| Host | CPU | Selected (Q4_0 / Q8_0) | Result |
+| --- | --- | --- | --- |
+| Apple M4 Max, native | SME2, no SVE | SME2, with and without `GGML_KLEIDIAI_SME=1` | pass |
+| `ubuntu:24.04` arm64 container, GCC 13.3, `armv8-a`, QEMU 8.2.2 | `cortex-a53` | NEON baseline | pass |
+| same | `max,sve=off,sme=off` | I8MM | pass |
+| same | `max,sve256=on,sme=off` | SVE | pass |
+
+Each QEMU row also passed with `GGML_KLEIDIAI_SME=1`. The `sve256` row executes
+the two SVE Q8_0 kernels that v0.8.0 added. No lane had run them before.
+
+### With patches 0005 and 0006
+
+Re-checked 2026-11-07 on v0.9.0 with `main`'s patch series. The source pair is
+unchanged because the audit fingerprints the unpatched submodule, not the
+patched copies. Patch 0005 adds FMLAL/FMLAL2 (FHM) to `ggml_vec_dot_f16` only
+where the compiler defines `__ARM_FEATURE_FP16_FML`: FP16 on ARMv8.4 or later.
+
+| Module | Toolchain | Audit | FMLAL/FMLAL2 |
+| --- | --- | --- | --- |
+| Android `armv8.2_2` | NDK 28.2, macOS arm64 | PASS: 266,016 instructions; 3,130 scalable in the 22 functions | 0 |
+| Android `armv8.6_1` | same | PASS: 269,285 instructions; 3,130 scalable in the 22 functions | 10, all in `ggml_vec_dot_f16` |
+| Linux baseline, `linux_armv8.2_2`, `linux_armv8.6_1` | `ubuntu:24.04` arm64 container, GCC 13.3, `tools/check_linux_arm64_cpu_variants.sh` | PASS for all three; QEMU selects the expected module on `cortex-a53`, `cortex-a72`, `neoverse-n1`, `neoverse-v1` and `neoverse-n2`, and the finite F16 dot check passes on each | 0, 0 and 10 (`fhm=10/0`) |
+
+`validate_linux_arm64_cpu_isa.py` allows the `fhm` family only for a variant
+with FP16 on ARMv8.4 or later, so `linux_armv8.6_1` passes and an FMLAL in
+`linux_armv8.2_2` would fail. `validate_android_cpu_isa.py` checks only SVE/SME
+containment; it decodes with `+fp16fml` so FMLAL is not an `<unknown>` word, and
+CI and the release run it on `android_armv8.2_2` only. Applying the Linux family
+policy to the two Android modules above also passes, with `fhm` allowed only for
+`armv8.6_1`.
+
+## Remaining gates
+
+- Hosted `android-arm64-isa`, `kleidiai-dispatch-emulated`,
+  `windows-arm64-kleidiai` and `linux-artifact-contract (arm64, *)` v0.9.0
+  rows. Windows needs no `cmake/kleidiai_windows.cmake` change because KleidiAI
+  is unchanged.
+- No lane runs the new ggml SVE256 repack paths. That needs an SVE variant
+  under QEMU with 256-bit vectors, or Neoverse V1 hardware. Physical Android
+  and SME-enabled device runs remain separate layers.

docs/v090_qualification.md

@@ -0,0 +1,86 @@
+# v0.9.0 qualification
+
+Stable upstream v0.9.0, `cd959bae99ff4bed22bef2604f45241c09bbb8c8`
+(2026-11-04), is the next runtime candidate. It is v0.8.0
+`8642541b255bfb5dc39cd351a178be09013a55d6` plus 150 commits. Its native release
+identity is the stable tag `v0.9.0`, which `latest` selects.
+
+This change moves the production submodule to v0.9.0 and adds it as the
+pinned `LLAMADART_UPSTREAMS` row of `validate_wrapper.yml`, next to the v0.7.0
+and v0.8.0 rows. Native `v0.9.0` is not published: the 2026-11-05 automatic
+dispatch failed at the CPU ISA audits
+([v090_android_isa_qualification.md](v090_android_isa_qualification.md)).
+Merging this change publishes nothing; `auto_native_release.yml` runs only on
+manual dispatch.
+
+## Wrapper
+
+The wrapper builds against v0.9.0 without source changes, and every
+`CMakeLists.txt` header probe succeeds. The exported API is unchanged.
+
+- `mtmd_gen_audio_info` gains `n_voices`. The wrapper reads only `type` and
+  `sample_rate`.
+- `mtmd_helper_gen_audio_inp` loses `voice`, which the wrapper never set.
+- `MTMD_GEN_AUDIO_TYPE_CHATTERBOX` maps to `LLAMA_DART_TTS_MODEL_TYPE_UNKNOWN`,
+  so the TTS entry points report it unsupported. Chatterbox and the new
+  predefined-voice API are not wrapped.
+
+Evidence, 2026-11-06, Apple M4 Max, macOS arm64, Metal on, Release,
+`LLAMADART_BUILD_TESTS=ON`, `LLAMADART_BUILD_TTS_SMOKE=ON`: native CTest
+84/84; `tools/validate_exports.py` with `--forbid-import __cxa_atexit` passes
+(92 required exports); `tools/llama_cpp_patches.py check-apple` passes on
+`libllamadart.dylib`. With patches 0005 and 0006 (2026-11-07, preset
+`macos-arm64-full`, `LLAMADART_BUILD_TESTS=ON`): CTest 85/85, including
+`llamadart_f16_dot_accumulation_test`; the same export and `check-apple`
+checks pass.
+
+## Carried patches
+
+Reviewed 2026-11-07 against v0.9.0 and upstream `master`
+`f7591a6a01f9273de03331732bdc0b39721168be` (b12190).
+`tools/llama_cpp_patches.py apply --strict` succeeds on both and applies 0001,
+0002, 0005 and 0006, the four that `manifest --upstream` lists. Every patch also
+passes `git apply --check` on both. `series.json` is unchanged.
+
+| Patch | v0.9.0 | Upstream | Status |
+| --- | --- | --- | --- |
+| 0001 Adreno 750 matvec (#79) | Applied | [#29165](https://github.com/ggml-org/llama.cpp/pull/29165) open draft; `use_subgroups` selection unchanged on `master` | Keep |
+| 0002 buffer device address (leehack/llamadart#782) | Applied | No fix: `master` still takes `bufferDeviceAddress` from `VkPhysicalDeviceVulkan12Features` without checking the entry point; upstream supports Vulkan 1.2 and later only | Keep |
+| 0003 embedding-mode reserve (#114) | Not applied: `upstream_fix` present ([#31086](https://github.com/ggml-org/llama.cpp/pull/31086)) | Fixed since v0.8.0 | Needed only by the v0.7.0 row |
+| 0004 Metal unary cast (#154) | Not applied: `upstream_fix` present, `required_text` absent ([#31360](https://github.com/ggml-org/llama.cpp/pull/31360)) | Fixed after v0.8.0 | Needed only by the v0.8.0 row |
+| 0005 ARM F16 dot (#174) | Applied | [#31734](https://github.com/ggml-org/llama.cpp/issues/31734) open, no fix; non-SVE NEON `ggml_vec_dot_f16` still accumulates in `float16x8_t` | Keep |
+| 0006 ARM F16 GEMM (#174) | Applied | Same issue; llamafile still uses `tinyBLAS<8, float16x8_t, …>` | Keep |
+
+v0.7.0 needs 0001–0003, 0005 and 0006 (0004 does not fit it, since it predates
+the regression). v0.8.0 needs 0001, 0002 and 0004–0006. Deleting 0003 means
+dropping the v0.7.0 `LLAMADART_UPSTREAMS` row (cores 0.15–0.17); deleting 0004
+means dropping the v0.8.0 row (core 0.18). Both are support-policy decisions
+for the maintainer.
+
+`check-apple` does not require the 0004 marker on v0.9.0, and
+`metal_os_floor.yml` checks the unpatched v0.9.0 Metal kernels on the oldest
+runners.
+
+## Consumer ABI delta from v0.8.0
+
+No public C symbol is removed. Regenerated bindings must account for:
+
+| Header | Change |
+| --- | --- |
+| `llama.h` | Adds `LLAMA_FTYPE_MOSTLY_F8_E4M3 = 42`. |
+| `ggml.h` | Adds `ggml_fp8_e4m3_t`, `GGML_TYPE_F8_E4M3 = 43` (`GGML_TYPE_COUNT` becomes 44), `GGML_FTYPE_MOSTLY_F8_E4M3 = 29` and `ggml_flash_attn_ext_rows`. |
+| `ggml-backend.h` | Adds `ggml_backend_tensor_copy_n`, `ggml_backend_tensor_set_n`, `ggml_backend_tensor_get_n`, and the `ggml_backend_comm_bootstrap_callback` and `ggml_backend_comm_init_rank_t` typedefs. |
+| `mtmd.h` | `mtmd_gen_audio_info`, returned by value, appends `int32_t n_voices` and grows from 16 to 24 bytes on 64-bit targets. Appends `MTMD_VIDEO_TEMPLATE_QWEN3VL` and `MTMD_GEN_AUDIO_TYPE_CHATTERBOX`. Adds `mtmd_bitmap_is_voice`, `mtmd_gen_audio_read_tensor`, `mtmd_gen_audio_voice_name`, `mtmd_gen_audio_get_voice` and `mtmd_gen_state_set_ref`. |
+| `mtmd-helper.h` | `mtmd_helper_gen_audio_inp` drops `voice` after `lang`, so every later field moves back 8 bytes and the struct shrinks from 72 to 64 bytes on 64-bit targets. The C++ `gen_audio::inp` drops `set_voice`. Adds the C++-only test types `mtmd_helper_video_src` and `mtmd_helper_video_seq`. |
+
+`common.h`, which the bindings do not include, drops `fs_path_to_utf8` and
+gives `common_control_vector_load` an `n_layer` argument; the wrapper uses
+neither.
+
+## Remaining gates
+
+- Hosted `v0.9.0` rows of every lane in `validate_wrapper.yml`, and
+  `validate_linux_release_rows.yml` and `metal_os_floor.yml` at the pin.
+- Android ISA: [v090_android_isa_qualification.md](v090_android_isa_qualification.md).
+- Model, family and device qualification stays with the consumer release; this
+  document claims none of it.

tests/test_android_cpu_isa.py

@@ -113,6 +113,9 @@ def test_exact_reviewed_source_pairs_are_required(self):
                              ), (
                                  "0b14cd6eac9832e7e50cee546495581c0067e7bca5c0db1dee6c3f7c0f7cf1f3",
                                  "fd5b9ca98d07b81671eca377c258da74e72173f92f3cab0acdf6c17ab870bd9d",
+                             ), (
+                                 "fead7601e3b35651ee62764c76f28b50ee564160b543ff79be6fcc81837216c4",
+                                 "fd5b9ca98d07b81671eca377c258da74e72173f92f3cab0acdf6c17ab870bd9d",
                              )})
         self.assertEqual(audit.SOURCE_PAIRS, expected)
         for ggml, kai in expected:
@@ -138,7 +141,8 @@ def test_cli_binds_exact_function_ranges_to_the_reviewed_source_pair(self):
         args = ["audit", "--objdump", "objdump", "--readelf", "readelf",
                 "--llama-source", "llama", "--kleidiai-source", "kleidiai", "test.so"]
         allowed_by_pair = {audit.V070_SOURCE_PAIR: audit.DISPATCHED_FUNCTIONS_V070,
-                           audit.V080_SOURCE_PAIR: audit.DISPATCHED_FUNCTIONS_V080}
+                           audit.V080_SOURCE_PAIR: audit.DISPATCHED_FUNCTIONS_V080,
+                           audit.V090_SOURCE_PAIR: audit.DISPATCHED_FUNCTIONS_V080}
         for pair in audit.SOURCE_PAIRS:
             allowed = allowed_by_pair.get(pair, audit.DISPATCHED_FUNCTIONS)
             for artifact_ranges in (audit.DISPATCHED_FUNCTIONS, audit.DISPATCHED_FUNCTIONS_V070,

tools/validate_android_cpu_isa.py

@@ -21,8 +21,8 @@
 # or a different KleidiAI combination invalidates the dispatch audit.
 # Candidate audit evidence: docs/post_v040_qualification.md,
 # docs/v041_android_isa_qualification.md, docs/v050_android_isa_qualification.md
-# docs/v060_android_isa_qualification.md, docs/v070_android_isa_qualification.md
-# and docs/v080_android_isa_qualification.md.
+# docs/v060_android_isa_qualification.md, docs/v070_android_isa_qualification.md,
+# docs/v080_android_isa_qualification.md and docs/v090_android_isa_qualification.md.
 V070_SOURCE_PAIR = (
     "b571df853210d243a83b1ec34a6936f147916f0369d792f8bb0d17f37fc7894b",
     "deb324dbcf8c952ec149433ab58cbed0b2ab2374a557f0a4eea575b2da761377",
@@ -31,6 +31,10 @@
     "0b14cd6eac9832e7e50cee546495581c0067e7bca5c0db1dee6c3f7c0f7cf1f3",
     "fd5b9ca98d07b81671eca377c258da74e72173f92f3cab0acdf6c17ab870bd9d",
 )
+V090_SOURCE_PAIR = (
+    "fead7601e3b35651ee62764c76f28b50ee564160b543ff79be6fcc81837216c4",
+    "fd5b9ca98d07b81671eca377c258da74e72173f92f3cab0acdf6c17ab870bd9d",
+)
 SOURCE_PAIRS = frozenset({
     (  # llama.cpp v0.4.0 / KleidiAI v1.24.0
         "c4dc92a7d95ebfad7f5f55e75be2ae773b7d95faf72a9581c9479c42bc41bca0",
@@ -54,6 +58,7 @@
     ),
     V070_SOURCE_PAIR,  # llama.cpp v0.7.0 / KleidiAI v1.31.0
     V080_SOURCE_PAIR,  # llama.cpp v0.8.0 8642541b255bfb5dc39cd351a178be09013a55d6 / KleidiAI v1.32.0
+    V090_SOURCE_PAIR,  # llama.cpp v0.9.0 cd959bae99ff4bed22bef2604f45241c09bbb8c8 / KleidiAI v1.32.0
 })
 
 # Exact ELF STT_FUNC ranges; never allow by kai_* prefix or disassembly label.
@@ -92,7 +97,8 @@
 """.split())
 
 # KleidiAI v1.32.0 adds a Q8_0 table entry reachable only with runtime SVE256,
-# SVE I8MM and DOTPROD. Its two kernels are exclusive to the v0.8.0 source pair.
+# SVE I8MM and DOTPROD. v0.9.0 keeps KleidiAI and its ggml callers unchanged, so
+# only the v0.8.0 and v0.9.0 source pairs grant these two kernels.
 DISPATCHED_FUNCTIONS_V080 = DISPATCHED_FUNCTIONS_V070 | frozenset("""
 kai_kernel_matmul_clamp_f32_qai8dxp1x8_qsi8cxp8x8_1x8_sve_dot
 kai_kernel_matmul_clamp_f32_qai8dxp4x8sf32_qsi8cxp4vsx8sf32bf32_16x4vs_sve_i8mm
@@ -101,6 +107,7 @@
 PAIR_FUNCTIONS = {
     V070_SOURCE_PAIR: DISPATCHED_FUNCTIONS_V070,
     V080_SOURCE_PAIR: DISPATCHED_FUNCTIONS_V080,
+    V090_SOURCE_PAIR: DISPATCHED_FUNCTIONS_V080,
 }
 
 SCALABLE_REG = re.compile(r"\b(?:z\d+|p\d+|pn\d+|za\d*[hv]?|zt\d+)\b")