Please provide how to use the release downloads as they're not complete llama.cpp archives.

[Zethalion]

It's not obvious how to use that download as is doesn't include the complete llama.cpp but just a single or several files. I can't get this running on Windows with Vulkan despite I tried getting the same build number from llama.cpp github and replacing ggml-vulkan.dll with the one provided here. That didn't work. Tell me what went wrong because I'm blind how to continue:

C:\Apps\bonsai\llama-server.exe --metrics
--cont-batching -ctk q8_0
-ctv q8_0 --no-mmap --no-warmup --jinja
--flash-attn on --model C:\Apps\KoboldCCP\prism-ml\Ternary-Bonsai-27B-gguf\Ternary-Bonsai-27B-Q2_0.gguf
--mmproj C:\Apps\KoboldCCP\prism-ml\Ternary-Bonsai-27B-gguf\Ternary-Bonsai-27B-mmproj-Q8_0.gguf -b 1024
-ub 512 -c 131072
-lv 4

0.00.501.794 E gguf_init_from_reader: tensor 'output_norm.weight' has offset 337715200, expected 357580800
0.00.501.799 E gguf_init_from_reader: failed to read tensor data
0.00.507.120 E llama_model_load: error loading model: llama_model_loader: failed to load model from C:\Apps\KoboldCCP\prism-ml\Ternary-Bonsai-27B-gguf\Ternary-Bonsai-27B-Q2_0.gguf
0.00.507.128 E llama_model_load_from_file_impl: failed to load model
0.00.507.160 E common_fit_params: encountered an error while trying to fit params to free device memory: failed to load model
0.00.507.168 I common_fit_params: fitting params to free memory took 0.06 seconds
0.00.558.886 E gguf_init_from_reader: tensor 'output_norm.weight' has offset 337715200, expected 357580800
0.00.558.892 E gguf_init_from_reader: failed to read tensor data
0.00.564.023 E llama_model_load: error loading model: llama_model_loader: failed to load model from C:\Apps\KoboldCCP\prism-ml\Ternary-Bonsai-27B-gguf\Ternary-Bonsai-27B-Q2_0.gguf
0.00.564.029 E llama_model_load_from_file_impl: failed to load model
0.00.564.036 E cmn common_init_: failed to load model 'C:\Apps\KoboldCCP\prism-ml\Ternary-Bonsai-27B-gguf\Ternary-Bonsai-27B-Q2_0.gguf'
0.00.564.041 E srv load_model: failed to load model, 'C:\Apps\KoboldCCP\prism-ml\Ternary-Bonsai-27B-gguf\Ternary-Bonsai-27B-Q2_0.gguf'

[khosravipasha]

Please follow the guide in our demo repo:
https://github.com/PrismML-Eng/Bonsai-demo

it does all the setup and download the correct models, it also has an AGENT.md if you want your agent to set it up.

[khosravipasha]

For Q2_0 might need to user our preebuild binaries or build from source specially for vulkan. Hopefully better out of the box support coming soon. The Bonsai-demo repo tries to do that automaticaly for different hardware/backends.

[Zethalion]

For Q2_0 might need to user our preebuild binaries or build from source specially for vulkan. Hopefully better out of the box support coming soon. The Bonsai-demo repo tries to do that automaticaly for different hardware/backends.

The provided binaries there in the ./bin folder are exactly what I needed to run the Q2_0 model using Vulkan on Windows. Thanks for pointing to that. I suggest to provide these complete binaries instead of just the ggml-vulkan.dll zipped which is confusing how to use without proper instructions.

But I'm now happily running the 27B model on my 12GB RX6700XT with 131072 ctx using KV Q4_0 to prevent overflowing.
Thanks!

[khosravipasha]

@Zethalion nice, glad it works now, could you send a PR or an issue with more details how you fixed it to unblock others.
https://github.com/PrismML-Eng/Bonsai-demo

I can review and merge it. In the meantime I will also take a look to do that in our release github action.

[sbzoutianxia]

I compiled this version of llama.cpp from source with Vulkan support enabled. However, when loading the model, I get an error saying:  shared memory size too small for matrix multiplication . Could someone help me figure out what went wrong?

[elthariel]

It looks related so I'm adding my report here. I've followed the instructions of the HF repo for the Ternary Bonsai 27B (i.e. using your fork master branch on linux and Ternary-Bonsai-27B-Q2_0.gguf, no extra parameters) and I'm getting a similar error:

[37765] 0.00.109.767 I srv    load_model: loading model 'Ternary-Bonsai-27B-Q2_0.gguf'
[37765] 0.00.145.135 E gguf_init_from_reader: tensor 'output_norm.weight' has offset 337715200, expected 357580800
[37765] 0.00.145.137 E gguf_init_from_reader: failed to read tensor data
[37765] 0.00.147.514 E llama_model_load: error loading model: llama_model_loader: failed to load model from Ternary-Bonsai-27B-Q2_0.gguf
[37765] 0.00.147.520 E llama_model_load_from_file_impl: failed to load model
[37765] 0.00.147.535 E common_fit_params: encountered an error while trying to fit params to free device memory: failed to load model
[37765] 0.00.174.047 E gguf_init_from_reader: tensor 'output_norm.weight' has offset 337715200, expected 357580800
[37765] 0.00.174.049 E gguf_init_from_reader: failed to read tensor data
[37765] 0.00.176.010 E llama_model_load: error loading model: llama_model_loader: failed to load model from Ternary-Bonsai-27B-Q2_0.gguf

[sbzoutianxia]

It looks related so I'm adding my report here. I've followed the instructions of the HF repo for the Ternary Bonsai 27B (i.e. using your fork master branch on linux and Ternary-Bonsai-27B-Q2_0.gguf, no extra parameters) and I'm getting a similar error:

[37765] 0.00.109.767 I srv    load_model: loading model 'Ternary-Bonsai-27B-Q2_0.gguf'
[37765] 0.00.145.135 E gguf_init_from_reader: tensor 'output_norm.weight' has offset 337715200, expected 357580800
[37765] 0.00.145.137 E gguf_init_from_reader: failed to read tensor data
[37765] 0.00.147.514 E llama_model_load: error loading model: llama_model_loader: failed to load model from Ternary-Bonsai-27B-Q2_0.gguf
[37765] 0.00.147.520 E llama_model_load_from_file_impl: failed to load model
[37765] 0.00.147.535 E common_fit_params: encountered an error while trying to fit params to free device memory: failed to load model
[37765] 0.00.174.047 E gguf_init_from_reader: tensor 'output_norm.weight' has offset 337715200, expected 357580800
[37765] 0.00.174.049 E gguf_init_from_reader: failed to read tensor data
[37765] 0.00.176.010 E llama_model_load: error loading model: llama_model_loader: failed to load model from Ternary-Bonsai-27B-Q2_0.gguf

I disabled Vulkan support and recompiled it, but I'm still getting the exact same error as you at runtime.

[khosravipasha]

@Zethalion I think got the release script fixed so it includes full binaries with vulkan, testing it now.

@sbzoutianxia is yours also windows? I was able to run some of the vulkan on my Mac through MoltenVK in that past but windows is probably different.

[sbzoutianxia]

@khosravipasha I'm on Linux.

Currently, if I compile without Vulkan, I can run  bonsai-1.7B-Q1_0.gguf  successfully, but running  Ternary-Bonsai-27B-Q2_g64.gguf  fails with the following error:
 [37765] 0.00.145.135 E gguf_init_from_reader: tensor 'output_norm.weight' has offset 337715200, expected 357580800 
 [37765] 0.00.145.137 E gguf_init_from_reader: failed to read tensor data 

On the other hand, if I compile with Vulkan enabled, even running  bonsai-1.7B-Q1_0.gguf  triggers the  shared memory too small  error.

[khosravipasha]

Yeah Ternary-Bonsai-27B-Q2_g64.gguf won't work in this fork, that is designed for the main llama.cpp, we already have CPU/Metal/Vulkan backend merged there, only one left is cuda.

For this fork should use the ones ending in Q2_0.gguf.

I think I fixed the pre-release build to include full builds for vulkan. Will do some testing and do another release soon.