Repository navigation
lora : support prism.hadamard folded weights - #30910
Merged
Merged
Conversation
CISC
approved these changes
Oct 21, 2026
Member
There was a problem hiding this comment.
Test fails to build, but not sure we should keep it anyway, @ggerganov?
Member
|
The test seems to much for relatively niche application. Better to remove it and if needed we can add tests in the future. |
Contributor
Author
|
@ggerganov @CISC I just removed the test. Thanks for your reviews! |
CISC
approved these changes
Oct 21, 2026
CISC
added
merge ready
ngxson
approved these changes
Oct 21, 2026
2 tasks done
voidstackloop
added a commit
to voidstackloop/llama.cpp
that referenced
this pull request
Oct 22, 2026
Conflict in build_lora_mm_id: master's Hadamard change (ggml-org#30910: the folded weight reads the transformed input, the LoRA delta reads cur) applied to the PR's host-ids and cache-slot branches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3 of 6 tasks
auRose94
pushed a commit
to auRose94/llama.cpp
that referenced
this pull request
Oct 28, 2026
* lora : support prism.hadamard folded weights * tests : remove LoRA test from test-llama-archs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR fixes an issue where models like Ternary-Bonsai-2 failed with a refusal error (targets a prism.hadamard folded weight, which is not supported) when initializing LoRA adapters, allowing LoRA to be used on prism.hadamard folded weights.
The refusal was the safe choice because the LoRA would read the rotated input and compute B*(A*(H*(Dx))) instead of BAx . Solution: the LoRA branch gets the original input (cur) and only the base weight gets the rotated input -> the adapter computes BA*x as trained (same approach as PrismML's prism branch (build_lora_mm)).
Changes:
Fixes #30909
Additional information
Model: prism-ml/Ternary-Bonsai-2-2B-gguf (Ternary-Bonsai-2-2B-Q2_0.gguf)
KLD, 16x512 tokens, all 150 rotated weights:
Requirements