llama.cpp ./embedding #17

Activity

1. mdrokz commented on Oct 25, 2023

   

   mdrokz

   on Oct 25, 2023

   Owner

   More actions

   Is there no rust binding to get the embeddings?

   Using llama.cpp one would use:

   ./embedding -m ./path/to/model --log-disable -p "Hello World!" 2>/dev/null

   have you tried this

   rust-llama.cpp/src/lib.rs

   Line 287 in baa1bcf

   pub fn embeddings(

   method ?

   this will get you the embeddings for the prompt
2. mdrokz

   added

   questionFurther information is requestedFurther information is requested

   on Oct 25, 2023
3. Philipp-Sc commented on Oct 26, 2023

   

   Philipp-Sc

   on Oct 26, 2023

   Author

   More actions

   @mdrokz thank you for your response.

   I tried it before trying the direct llama.cpp ./embedding executable.

   The function would always return an empty vector:

   []

   I tried multiple configurations but could not fix the issue.
4. mdrokz commented on Oct 26, 2023

   

   mdrokz

   on Oct 26, 2023

   Owner

   More actions

   @mdrokz thank you for your response.

   I tried it before trying the direct llama.cpp ./embedding executable.

   The function would always return an empty vector:

   []

   I tried multiple configurations but could not fix the issue.

   Alright i will test on my end see whats happening. Thanks

   👍React with 👍1Reacted by Philipp
5. Philipp-Sc commented on Oct 27, 2023

   

   Philipp-Sc

   on Oct 27, 2023

   Last edited by Philipp-Sc

   Author

   More actions

   I was using zephyr-7B-alpha-GGUF with:

   context_size: 8192 
   n_batch: 512 
   embeddings: true
   

   without any GPU assistance.

   ---

   Note:
   There also was some strange behavior regarding n_batch and n_token where a longer prompt (still way below the context length) lead to an unexpected error:

   GGML_ASSERT: n_token <= n_batch
   

   ---

   Right now my workaround is to use a rust wrapper (Command::new) around the ./embedding binary and parse the float values from stdout back into a string, then vector. The only parameter I set is --ctx-size 8192 and --mlock.
   I imagine this is less efficient as the model needs to be reloaded for each call.