Repository navigation
Conversation
The project can be built for Android with NDK and CMake like this: cmake -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake -DANDROID_ABI='arm64-v8a' -DANDROID_PLATFORM=android-23 .. However, vdotq_* intrinsics are not available on Android. Fix this by checking for __ANDROID__ and use the code replaced by commit 84d9015 in this case.
Member
Author
|
Turns out this is not needed as long as we have |
Deadsg
pushed a commit
to Deadsg/llama.cpp
that referenced
this pull request
Dec 19, 2023
…ortable Make app server module importable
thedanhoffman
pushed a commit
to thedanhoffman/llama.cpp
that referenced
this pull request
Apr 14, 2026
ggerganov
pushed a commit
that referenced
this pull request
Apr 21, 2026
#21944) * Thread safety per request only * Fix ROPE yarn case * Fix sticky stateful config * Use i4/i8 directly for symmetric quant * Use weightless caching * Add WeightlessCacheAttribute to reduce NPU memory usage * Gelu tanh support (#125) * Imrope support (#126) * fix(openvino): explicit ov::Tensor frees in ggml_backend_openvino_free * add GPU,NPU support in OV Dockerfile * add build-openvino.yml ci * Fix sticky stateful config * add concurrency to ov-gpu ci runs. Move OV CI to build-openvino.yml * fix thread-safety of shared runtime context * rope type abstraction for frontend translations * fix editorconfig --------- Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com> Co-authored-by: Dan Hoffman <dhoff749@gmail.com> Co-authored-by: Ravi Panchumarthy <ravi.panchumarthy@intel.com>
phuongncn
pushed a commit
to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4
that referenced
this pull request
Apr 28, 2026
* q4_0_r4: 6% faster PP on NEON * qx_0_r4_q8_0 template Applied to q4_0_r4 and q5_0_r4. It makes q5_0_r4 PP ~7% faster. * Apply qx_0_r4_q8_0 template also to q6_0_r4 and iq4_nl_x4 * Simplify * Minor iq4_xs_r4 improvement on NEON --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
my-other-github-account
pushed a commit
to my-other-github-account/llama.cpp
that referenced
this pull request
May 15, 2026
ggml-org#21944) * Thread safety per request only * Fix ROPE yarn case * Fix sticky stateful config * Use i4/i8 directly for symmetric quant * Use weightless caching * Add WeightlessCacheAttribute to reduce NPU memory usage * Gelu tanh support (ggml-org#125) * Imrope support (ggml-org#126) * fix(openvino): explicit ov::Tensor frees in ggml_backend_openvino_free * add GPU,NPU support in OV Dockerfile * add build-openvino.yml ci * Fix sticky stateful config * add concurrency to ov-gpu ci runs. Move OV CI to build-openvino.yml * fix thread-safety of shared runtime context * rope type abstraction for frontend translations * fix editorconfig --------- Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com> Co-authored-by: Dan Hoffman <dhoff749@gmail.com> Co-authored-by: Ravi Panchumarthy <ravi.panchumarthy@intel.com>
This was referenced Jun 4, 2026
marksverdhei
referenced
this pull request
in heiervang-technologies/ht-llama.cpp
Jun 12, 2026
… (#71) Measured perplexity on Qwen3.5-0.8B-BF16 / wikitext-2 / ctx=512: | cache-type | PPL | vs f16 | |------------|--------|--------| | f16 | 19.08 | baseline | | q8_0 | 19.08 | lossless | | tbq3_0 | 1252.30 | 65x worse | | tbq4_0 | 1393.00 | 73x worse | TBQ KV-cache produces near-random output. Likely root cause is statistical: TBQ's rotated-domain codebook was calibrated for weight distributions, not the K/V tensor distributions seen during inference. The encoding scheme itself cannot faithfully represent KV values. Snoop-kube's cluster audit confirms zero deployments use tbq* KV-cache (every host uses q8_0 or q4_0). DFlash also defaults to q8_0 (PR #65). No production consumer exists. This PR adds a one-line experimental note to the --cache-type-k/v and --cache-type-k-draft/v-draft help text, referencing issue #70 for the full data + recommendation. Code path stays in place — Markus may have roadmap intent I'm not aware of; this just stops anyone reading --help from assuming tbq* is a usable choice without checking. Follow-ups if Markus prefers full removal: * drop tbq3_0/tbq4_0 from common/arg.cpp's kv_cache_types list * keep the ftypes (TBQ weight quantization is separate from KV use) * close issues #124 + #125 as wont-fix
MrLordCat
referenced
this pull request
in MrLordCat/llama.cpp-rdna-lab
Jul 16, 2026
…s (#21944) * Thread safety per request only * Fix ROPE yarn case * Fix sticky stateful config * Use i4/i8 directly for symmetric quant * Use weightless caching * Add WeightlessCacheAttribute to reduce NPU memory usage * Gelu tanh support (#125) * Imrope support (#126) * fix(openvino): explicit ov::Tensor frees in ggml_backend_openvino_free * add GPU,NPU support in OV Dockerfile * add build-openvino.yml ci * Fix sticky stateful config * add concurrency to ov-gpu ci runs. Move OV CI to build-openvino.yml * fix thread-safety of shared runtime context * rope type abstraction for frontend translations * fix editorconfig --------- Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com> Co-authored-by: Dan Hoffman <dhoff749@gmail.com> Co-authored-by: Ravi Panchumarthy <ravi.panchumarthy@intel.com>
zommiommy
pushed a commit
to zommiommy/llama.cpp
that referenced
this pull request
Aug 18, 2026
ggml-org#21944) * Thread safety per request only * Fix ROPE yarn case * Fix sticky stateful config * Use i4/i8 directly for symmetric quant * Use weightless caching * Add WeightlessCacheAttribute to reduce NPU memory usage * Gelu tanh support (ggml-org#125) * Imrope support (ggml-org#126) * fix(openvino): explicit ov::Tensor frees in ggml_backend_openvino_free * add GPU,NPU support in OV Dockerfile * add build-openvino.yml ci * Fix sticky stateful config * add concurrency to ov-gpu ci runs. Move OV CI to build-openvino.yml * fix thread-safety of shared runtime context * rope type abstraction for frontend translations * fix editorconfig --------- Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com> Co-authored-by: Dan Hoffman <dhoff749@gmail.com> Co-authored-by: Ravi Panchumarthy <ravi.panchumarthy@intel.com>
ckti
pushed a commit
to ckti/llama.cpp-consolidated
that referenced
this pull request
Sep 13, 2026
Use the hardware byte permute (v_perm_b32) for the Q1_0 sign unpack and the Q2_0/PQ2_0 symbol expansion on HIP, where __byte_perm is a software routine, and keep the default split-K MMVQ geometry for these types on CDNA, where the small_k geometry runs several times slower per call. Bit-exact: test-backend-ops MUL_MAT and MUL_MAT_ID pass unchanged, greedy generations match the CPU backend, and KL divergence against the base build is 0 (mean) with 99.99+% top-token agreement on 8B and 27B Q1_0 and PQ2_0 models. MI300X decode: 4B Q1_0 112 -> 322 tok/s, 8B Q1_0 81 -> 269, 27B Q1_0 31.5 -> 97, 8B PQ2_0 113 -> 259, 27B PQ2_0 48.6 -> 91.7. CUDA and MUSA are unchanged (HIP device-compile guards); other quant types are unchanged (type-scoped rule).
bong-water-water-bong
referenced
this pull request
in 1bit-MONSTER/llama.cpp
Oct 6, 2026
…andalone ggml-hrx: reuse Q5_K/IQ4_XS block metadata in SwiGLU decode
imaami
pushed a commit
to imaami/llama.cpp
that referenced
this pull request
Oct 10, 2026
Use the hardware byte permute (v_perm_b32) for the Q1_0 sign unpack and the Q2_0/PQ2_0 symbol expansion on HIP, where __byte_perm is a software routine, and keep the default split-K MMVQ geometry for these types on CDNA, where the small_k geometry runs several times slower per call. Bit-exact: test-backend-ops MUL_MAT and MUL_MAT_ID pass unchanged, greedy generations match the CPU backend, and KL divergence against the base build is 0 (mean) with 99.99+% top-token agreement on 8B and 27B Q1_0 and PQ2_0 models. MI300X decode: 4B Q1_0 112 -> 322 tok/s, 8B Q1_0 81 -> 269, 27B Q1_0 31.5 -> 97, 8B PQ2_0 113 -> 259, 27B PQ2_0 48.6 -> 91.7. CUDA and MUSA are unchanged (HIP device-compile guards); other quant types are unchanged (type-scoped rule). (cherry picked from commit a7cc296) Assisted-by: GitHub Copilot
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The project can be built for Android with NDK and CMake like this:
cmake -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake -DANDROID_ABI='arm64-v8a' -DANDROID_PLATFORM=android-23 ..
However, vdotq_* intrinsics are not available on Android. Fix this by checking for ANDROID and use the code replaced by commit 84d9015 in this case.