[ivanstepanovftw]

Added in #242 without a strong rationale.
Users can already insert a space manually at the beginning of the prompt if desired.

For example, I cannot get rid of token 15629 -> ' Manager'. I am wanted it to be 3260 -> 'Manager':

main: prompt: ' Manager's Persona: Manager I work with in my company.
Manager: I am waiting.'
main: number of tokens in prompt = 22
     1 -> ''
 15629 -> ' Manager'
 29915 -> '''
 29879 -> 's'
  5196 -> ' Person'
 29874 -> 'a'
 29901 -> ':'
 15629 -> ' Manager'
   306 -> ' I'
   664 -> ' work'
   411 -> ' with'
   297 -> ' in'
   590 -> ' my'
  5001 -> ' company'
 29889 -> '.'
    13 -> '
'
  3260 -> 'Manager'
 29901 -> ':'
   306 -> ' I'
   626 -> ' am'
 10534 -> ' waiting'
 29889 -> '.'

[DannyDaemonic]

I could be wrong about what's happening here, but I think with OpenLLaMA the BOS token is a lot more important: #1291

[ivanstepanovftw]

I am trying pygmalion-7b model, which prompt looks like this example:

Assistant's Persona: Assistant is a highly intelligent language model trained to comply with user requests.
<START>
Assistant: Hello! How may I help you today?
You: What is Zork?
Assistant:

[slaren]

The rationale is just duplicating what the SentencePiece tokenizer does.

[Green-Sky]

The rational was that the llama models where trained with a prefixed space. I agree that not every model has that requirement, but this was done to make it easier for users without this knowledge.

[ivanstepanovftw]

Oh I see

% echo "I saw a girl with a telescope." | spm_encode --model=m.model
▁I ▁saw ▁a ▁girl ▁with ▁a ▁ te le s c o pe .

[ivanstepanovftw]

This is called dummy prefix google/sentencepiece#282
Closing as LLaMA and derivatives have default tokenizer settings.

[ggerganov]

You can put this functionality behind a cmd arg