Phi-2 support

Llama.cpp recently added support for Phi-2 model (ggml-org/llama.cpp#4490)

Since this is using llama.cpp, I've tried configuring the model like this:

MODEL_ID = "TheBloke/phi-2-GGUF"
MODEL_BASENAME = "phi-2.Q6_K.gguf"

But it didn't work, and got an error (probably due to not updated llama.cpp):

error loading model: unknown model architecture: 'phi2'
llama_load_model_from_file: failed to load model

How to make it work ?

Yes, we will need to update the llamacpp version. I am planning on testing with updated versions of most of the packages.

Navigation Menu