Segfault in CPU repack loading Ternary-Bonsai-2-27B-PQ2_0.gguf (output.weight)

Note: all interaction beyond this header is my agent posting on my behalf. What is says is plausible but I haven't personally verified every single thing.

Segfault in CPU repack loading Ternary-Bonsai-2-27B-PQ2_0.gguf (output.weight)

Loading Ternary-Bonsai-2-27B-PQ2_0.gguf segfaults (exit 139) in ggml_backend_cpu_repack_buffer_set_tensor during load_tensors, on both prebuilt releases I tried (prism-b10683-d8f26ee, prism-b10685-7dffb15; ubuntu-x64 CPU and ubuntu-vulkan-x64).

Repro

./llama-cli -m Ternary-Bonsai-2-27B-PQ2_0.gguf -ngl 0 -c 4096 -p "hi" -n 4
# => Segmentation fault during "Loading model..."

File: prism-ml/Ternary-Bonsai-2-27B-gguf, Ternary-Bonsai-2-27B-PQ2_0.gguf (7,206,168,928 bytes, ftype: PQ2_0 - 2.13 bpw (group 128)).
CPU: AMD Ryzen 9 7900X (Zen 4). Crashes with the auto-picked libggml-cpu-zen4.so and with zen4 removed (falls back to libggml-cpu-cooperlake.so, same crash address offset) — looks variant-independent.
Also crashes with -ngl 99 (Vulkan, RX 9070 XT) and with LLAMA_ATTN_ROT_DISABLE=1.
Verbose log shows all 64 blocks repack fine (repack tensor blk.63.ffn_up.weight with pq2_0_4x8 is the last line), so the crash is on a later tensor — almost certainly output.weight / token_embd.weight.
gdb backtrace top frames:
- ggml_backend_cpu_repack_buffer_set_tensor (libggml-cpu-*.so)
- llama_model_base::load_tensors
- llama_model_load / llama_model_load_from_file_impl

Workarounds (verified)

LLAMA_ARG_REPACK=0 (aka --no-repack): loads and runs correctly (sane thinking + answers), so the weight file itself is intact. CPU-only is slow (~3 tok/s), as expected.
Ternary-Bonsai-2-27B-PTQ1_0.gguf from the same repo loads fine with repack on, and runs on Vulkan at ~57 t/s prompt / ~9 t/s decode on an RX 9070 XT. So the bug looks PQ2_0-repack-specific (possibly the repack of the large vocab-shaped output.weight).

Happy to provide a core dump / run test builds.

I noticed the same segfault when my Vulkan + AMD system, can confirm the PT models semi-works.

Update: root cause found (with a fix)

The crashing tensor is not output.weight -- my title was wrong. Disassembly of the fault (mov (%rdi),%rax with rdi == 0) plus a struct dump shows tensor->extra == NULL on a tensor named prism.hadamard.1024.

Mechanism, as far as I can tell: the Hadamard setup inherits the weight's buffer type for its F32 helper tensors (src/llama-model.cpp, the buft taken from weight->buffer). With -ngl 0 the weights live in the CPU repack buffer, so the rotation tensor is allocated from it too. The repack buffer's init_tensor then stores nullptr as extra (F32 has no repack type), and set_tensor dereferences it unconditionally (ggml/src/ggml-cpu/repack.cpp). I suspect the same applies to prism.hadamard.signs.*, which shares the buft, but I did not verify that one crashing independently.

Fixing only set_tensor moved the crash, which suggests a second spot with the same assumption: first forward pass died in ggml_backend_tensor_copy (null callee) during graph_compute, presumably copying the rotation tensor out of a buffer whose get/cpy_tensor are null. Keeping the F32 helpers out of CPU_REPACK buffers resolved both crashes for me on CPU, Vulkan (-ngl 99), and a HIP build. I kept the set_tensor null guard as well, since a null extra previously meant a guaranteed segfault and I could not find another path that depends on it crashing.

The two hunks (against prism @ f0a2b5d):

// repack.cpp, in ggml_backend_cpu_repack_buffer_set_tensor, after the asserts:
// no repack type (e.g. F32 helper tensors sharing the buffer): store as-is
if (tensor->extra == nullptr) {
    GGML_ASSERT(tensor->data != nullptr);
    memcpy(tensor->data, data, size);
    GGML_UNUSED(buffer);
    return;
}

// llama-model.cpp, after the preferred_buft resolution in the hadamard block:
// repack buffers only host repacked weights: keep the F32 helpers out
if (strcmp(ggml_backend_buft_name(buft), "CPU_REPACK") == 0) {
    buft = ggml_backend_cpu_buffer_type();
}

Caveats: I matched the repack buffer by name, which is crude -- happy to do it differently if there is a cleaner predicate. Also, I may be missing why the helpers inherit the weight's buffer in the first place (locality for GPU-offloaded weights, presumably); the fallback preserves that for non-repack buffer types and only redirects the CPU-repack case to the plain CPU buffer. If this direction looks right I can submit it as a PR.

@theoparis

Muse Spark 1.3 got ROCm Q2 running after some effort, FWIW @theoparis, that greatly improved my throughput to 800t/s prefill, 45t/s decode.

I did a vulkan build (as the release page has broken link for vulkan )

 LD_LIBRARY_PATH="~/llamacpps/prism/build/bin" \
"~/llamacpps/prism/build/bin/llama-server" -m ~/models/Ternary-Bonsai-2-27B-gguf/Ternary-Bonsai-2-27B-PQ2_0.gguf -a Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf --spec-type draft-mtp --spec-draft-n-max 3 --mmproj ~/models/Ternary-Bonsai-2-27B-gguf/Ternary-Bonsai-2-27B-mmproj-BF16.gguf --load-mode none -fa 1 -ctk q8_0 -ctv q8_0 -c 162144 -np 1 --jinja --host 0.0.0.0 --port 8888 --image-min-tokens 1024
0.00.025.400 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.025.631 W srv  llama_server: -----------------
0.00.025.632 W srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.025.632 W srv  llama_server: this can be a security risk (cross-origin attacks)
0.00.025.632 W srv  llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.025.632 W srv  llama_server: -----------------
0.00.026.854 I srv    load_model: loading model '/home/dinesh/models/Ternary-Bonsai-2-27B-gguf/Ternary-Bonsai-2-27B-PQ2_0.gguf'
0.00.659.742 W operator(): failed to measure the memory of the extra model, fitting without it: failed to create llama_context from model
Segmentation fault         (core dumped) LD_LIBRARY_PATH="/home/dinesh/llamacpps/prism/build/bin" "/home/dinesh/llamacpps/prism/build/bin/llama-server" -m /home/dinesh/models/Ternary-Bonsai-2-27B-gguf/Ternary-Bonsai-2-27B-PQ2_0.gguf -a Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf --spec-type draft-mtp --spec-draft-n-max 3 --mmproj /home/dinesh/models/Ternary-Bonsai-2-27B-gguf/Ternary-Bonsai-2-27B-mmproj-BF16.gguf --load-mode none -fa 1 -ctk q8_0 -ctv q8_0 -c 162144 -np 1 --jinja --host 0.0.0.0 --port 8888 --image-min-tokens 1024

Independent reproduction on an AMD Ryzen AI 7 PRO 350, Ubuntu 24.04.5 LTS, CPU-only inference, using Ternary-Bonsai-2-27B-PQ2_0.gguf and the prism-b10683-d8f26ee Ubuntu x64 CPU binaries.

The backtrace also ends in ggml_backend_cpu_repack_buffer_set_tensor -> llama_model_base::load_tensors. --no-repack allows loading and inference.

The diagnosis in the earlier comment matches our findings: the F32 Hadamard helper tensors inherit a weight's CPU repack buffer. A local variant of the loader fix uses the buffer's device type rather than comparing its name. Immediately after buft is taken from weight->buffer, before the preferred_buft selection, we added:

// Transform tables are F32, so they cannot use quantized CPU repack buffers.
ggml_backend_dev_t dev = ggml_backend_buft_get_device(buft);
if (dev && ggml_backend_dev_type(dev) == GGML_BACKEND_DEVICE_TYPE_CPU) {
    buft = ggml_backend_dev_buffer_type(dev);
}

Rebuilt commit d8f26eec76da6d09bb708bcba51ef64b8cd868a3 with GGML_NATIVE=ON. This loads and generates with repacking enabled, without changing repack.cpp. Functional checks passed for a Python code correction, image reading with the Q8_0 projector, and browser chat. The code-correction answer matched the prebuilt --no-repack baseline.

This was tested only on the CPU above; no GPU validation or broad numerical equivalence testing was performed. Diagnosis, patching, and testing were AI-assisted.

Confirmed on a third, distinct GPU/CPU combo — AMD Ryzen AI 7 350 w/ Radeon 860M (gfx1152, RDNA 3.5 iGPU), GGML_VULKAN=ON build (RADV driver). Segfaults on both offload modes, matching the pattern already reported on Zen4 CPU and RX 9070 XT:

./llama-cli -m Ternary-Bonsai-2-27B-PQ2_0.gguf -ngl 0 -c 4096 -p "hi" -n 4
# => Segmentation fault
./llama-cli -m Ternary-Bonsai-2-27B-PQ2_0.gguf -ngl 99 -c 4096 -p "hi" -n 4
# => Segmentation fault

Same file as reported (Ternary-Bonsai-2-27B-PQ2_0.gguf, 7,206,168,928 bytes — matches the LFS size on prism-ml/Ternary-Bonsai-2-27B-gguf).

gdb backtrace from the -ngl 0 core, top frames match yours exactly:

#0  ggml_backend_cpu_repack_buffer_set_tensor(ggml_backend_buffer*, ggml_tensor*, void const*, unsigned long, unsigned long) ()
#1  llama_model_base::load_tensors(llama_model_loader&) ()
#2  llama_model_load(...)
#3  llama_model_load_from_file_impl(...)
#4  llama_model_load_from_file ()
#5  common_init_result::common_init_result(common_params&, bool) ()
#6  common_init_from_params(common_params&, bool) ()

Three architecturally unrelated backends (x86 CPU repack, RDNA4 discrete, RDNA3.5 iGPU) hitting the identical crash frame is a strong signal this is generic to the PQ2_0 repack path itself (likely the output.weight/vocab-shaped tensor as you suspected), not anything backend- or driver-specific. Happy to test --no-repack or provide the full core dump if useful.

Disclosure: I had help from Claude, if you would like; I can delete this draft.

Tested baogorek's device-type-based variant (the refined version, matching by ggml_backend_dev_type(dev) == GGML_BACKEND_DEVICE_TYPE_CPU rather than the "CPU_REPACK" name string) on GPU, since that specific variant hadn't been GPU-validated yet -- Mattwmaster58's original name-matched version was already confirmed on Vulkan/HIP, but the cleaner device-type predicate was CPU-only until now.

Hardware: AMD Ryzen AI 7 350 / Radeon 860M (gfx1152, RDNA 3.5 iGPU).

Applied only the llama-model.cpp hunk (device-type check after buft is taken from weight->buffer, before preferred_buft resolution), no changes to repack.cpp. Rebuilt both a GGML_VULKAN=ON and a GGML_HIP=ON tree from the same source, same commit (1a07bfa5f) plus this patch.

Same repro as before, Ternary-Bonsai-2-27B-PQ2_0.gguf:

Build	Flags	Result
Vulkan	`-ngl 0`	loads, generates, exits clean
Vulkan	`-ngl 99`	loads, generates (5.9 pp / 3.8 tg t/s)
HIP	`-ngl 99`	loads, generates (21.5 pp / 7.2 tg t/s)

No segfaults on either backend at either offload level. Didn't apply the repack.cpp null-guard hunk -- the llama-model.cpp fix alone was enough here too, consistent with what's already been reported.

Patch application, rebuilds, and testing were AI-assisted.

Thanks for the report and the detailed repro, and thanks to everyone who confirmed it on other machines. This was fixed by #245, which keeps the Hadamard rotation tensors out of the CPU repack buffers, and the fix is in the new release prism-b10735-842b188.

Could you give that build a try and confirm the model loads? I'll close this once someone confirms, and I'm closing #204, #219 and #224 as duplicates so the discussion stays in one place.

Confirmed fixed on prism-b10735-842b188 (commit 842b18804), clean builds with no local patches (the device-type patch I tested earlier was reverted first).

Hardware: AMD Ryzen AI 7 350 / Radeon 860M (gfx1152, RDNA 3.5 iGPU), Mesa 26.0.8 RADV, ROCm 10.0.

Ternary-Bonsai-2-27B-PQ2_0.gguf, -c 4096 -p "hi":

Build	Flags	Result
CPU only (`-dev none -ngl 0`, weights in `CPU_REPACK`)	repack enabled	loads, generates correctly (12.6 pp / 5.7 tg t/s)
Vulkan	`-ngl 0`	loads, generates (15.0 pp / 4.6 tg t/s)
Vulkan	`-ngl 99`	loads, generates (18.4 pp / 8.8 tg t/s)
HIP	`-ngl 99`	loads, generates (38.3 pp / 7.7 tg t/s)

The CPU-only run is the one that exercises the original crash path: the log shows CPU_REPACK model buffer size = 6484.57 MiB, and it answers "The capital of France is" with "Paris." No segfaults anywhere. Good to close from my side.

Builds and testing were AI-assisted.

Thanks for the report and the detailed repro, and thanks to everyone who confirmed it on other machines. This was fixed by #245, which keeps the Hadamard rotation tensors out of the CPU repack buffers, and the fix is in the new release prism-b10735-842b188.

Tested the Linux (vulkan) backend build on AMD Ryzen AI Max+ 395(128GB) chip with OS Ubuntu 26.04 LTS AMD Radeon 8060S (gfx1151)
it is working fine. now THANKS!!
Token Generation speed: 24.35496 tok/s
Prompt Processing speed: 260.999674 tok/s

llm: https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/tree/main

image recognition accuracy is awesome too !!

Confirmed on the same AMD Ryzen AI 7 PRO 350 / Ubuntu 24.04.5 LTS system from my earlier report.

Tested the official Ubuntu x64 CPU binaries from prism-b10735-842b188 (build 10735, commit 842b18804), with no local patches. The downloaded archive matches the release's published SHA-256.

Ternary-Bonsai-2-27B-PQ2_0.gguf loads and generates with -dev none -ngl 0 --repack -c 4096. A separate llama-completion run confirms CPU_REPACK model buffer size = 6484.57 MiB, so the original crash path is exercised. Both runs exit with status 0.

The single-turn llama-cli test answered “What is the capital of France?” with “The capital of France is Paris.”

CPU-only validation; I haven't tested GPU backends with this release. Testing was AI-assisted. Thanks for shipping the fix. Good to close from my side.

Navigation Menu

Repository navigation

Segfault in CPU repack loading Ternary-Bonsai-2-27B-PQ2_0.gguf (output.weight) #180