You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Maybe lower default temp and switch to top_k 40 #42
AFAIK, there is no top k filtering in the current version. The main code uses the llama_sample_top_p, and not gpt_sample_top_k_top_p which is the only piece of code that actually uses the top_k parameter.
The repetition penalty could maybe be ported to this sampler and used instead?
I've seen multiple people reporting that FB's default sampler is not adequate for comparing LLaMA's outputs with davinci's. Thus, enabling top k filtering could allow people to experiment with and compare different sampling strategies.
It does seem to work much better with these options, based on shawwn's patch:
--temp 0.7 --top_k 40 --top_p 0 --repeat_last_n 256 --repeat_penalty 1.1764705882352942
I'm not sure what value is ideal for repeat_last_n, but with a little testing, 256 seems to be enough, while 128 wasn't.
Activity
G2G2G2G commentedon Mar 12, 2023
Piezoid commentedon Mar 12, 2023
AFAIK, there is no top k filtering in the current version. The main code uses the
llama_sample_top_p, and notgpt_sample_top_k_top_pwhich is the only piece of code that actually uses thetop_kparameter.The repetition penalty could maybe be ported to this sampler and used instead?
I've seen multiple people reporting that FB's default sampler is not adequate for comparing LLaMA's outputs with davinci's. Thus, enabling top k filtering could allow people to experiment with and compare different sampling strategies.
sswam commentedon Mar 12, 2023
It does seem to work much better with these options, based on shawwn's patch:
--temp 0.7 --top_k 40 --top_p 0 --repeat_last_n 256 --repeat_penalty 1.1764705882352942
I'm not sure what value is ideal for repeat_last_n, but with a little testing, 256 seems to be enough, while 128 wasn't.
Piezoid commentedon Mar 12, 2023
llama.cpp with
--top_p 0is greedy inference, picking the highest probability token.beiller commentedon Mar 12, 2023
FYI Top K isn't used this PR tho should fix it:
#56
ggerganov commentedon Mar 13, 2023
Need a better strategy to determine default parameters.
Single examples do not show anything of value
Merge pull request ggml-org#42 from MariusCiocanel/master
Merge pull request ggml-org#42 from gianni-cor/fix_cl_mtmd
feat: perf opt part3 (ggml-org#42)
docs: benchmark results for turbo4 fix, sparse V, multi-model validation
Adding fused rms_norm (ggml-org#42)
webui: promote tool_artifacts SSE events into message.extra
27 remaining items
rdna-boosts: block 05: flash-attention band, native quantized K/V and…
rdna-boosts: block 05: flash-attention band, native quantized K/V and…
rdna-boosts: block 05: flash-attention band, native quantized K/V and…