Repository navigation
When loading the model I get the following error: #17
Description
llm_load_tensors: ggml ctx size = 0.16 MB
llm_load_tensors: using CUDA for GPU acceleration
llm_load_tensors: mem required = 9363.40 MB
llm_load_tensors: offloading 6 repeating layers to GPU
llm_load_tensors: offloaded 6/43 layers to GPU
llm_load_tensors: VRAM used: 1637.37 MB
.................................................................................GGML_ASSERT: D:\a\llama-cpp-python-cuBLAS-wheels\llama-cpp-python-cuBLAS-wheels\vendor\llama.cpp\ggml-cuda.cu:5925: false
Activity
jllllll commentedon Oct 11, 2023
What model are you trying to load? This error is indicative of an incompatible model.
JeisonJimenezA commentedon Oct 11, 2023
I'm loading this model ----> TheBloke/sqlcoder-GGUF
jllllll commentedon Oct 11, 2023
What version of llama-cpp-python are you using?
JeisonJimenezA commentedon Oct 11, 2023
llama_cpp_python 0.2.11+cu117
jllllll commentedon Oct 12, 2023
Yeah, just finished downloading and got the same error. There may be something wrong with the model.
Not much I can do on my end as I'm only building wheels for llama-cpp-python. As far as I can tell, the issue is with llama.cpp itself.
jllllll commentedon Oct 12, 2023
Could simply be that StarCoder models aren't supported with CUDA? Not sure.
I do know that only some model types are supported by the cuBLAS implementation.
jllllll commentedon Oct 12, 2023
This does seem to be the case: ggml-org/llama.cpp#3187 (comment)
Guess CUDA support for it just hasn't been done yet.
JeisonJimenezA commentedon Oct 12, 2023
Thank you for your help. Where can I see supported models for CUDA?
jllllll commentedon Oct 12, 2023
Only thing I can find so far is this in the source code:
https://github.com/ggerganov/llama.cpp/blob/b8fe4b5cc9cb237ca98e5bc51b5d189e3c446d13/llama.cpp#L5840-L5844
The
REFACTandMPTentries are new model arch support that isn't present yet in the current version of llama-cpp-python.That leaves current llama-cpp-python cuBLAS support at these models: