You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
llama.cpp supports negative prompt strings (see main.cpp in Examples), but I cannot find any support here. Haven't I looked well or it does not exist yet?
LLamaSharp doesn't support negative prompts in an easy to use way.
All the necessary low level components are there, but they haven't been put together into an easy to use high level system that allows you to use it by just supplying a string as a negative prompt.
I see. This is an important feature since we can control model ethics with it easily. It is probably not a lot of work for someone who knows the code already very well.
Applying the guidance logits is fairly simple (I've actually just put together the code to do that).
The tricky part is that you need a second entire sequence running in parallel to the first to generate those guidance logits. This really needs batched execution to be fast, which is supported in LLamaSharp, but only very recently and it doesn't work with any kind of high level executor yet.
@zsogitbe I've implemented a demo for how to use classifier free guidance using the new batched executor. As I mentioned this is not as high level as the existing executors, but it should at least demo how things work at the lower level for now. If you could pull #536, test it out and leave any feedback on that PR, it would be very appreciated!
Activity
martindevans commentedon Feb 24, 2024
LLamaSharp doesn't support negative prompts in an easy to use way.
All the necessary low level components are there, but they haven't been put together into an easy to use high level system that allows you to use it by just supplying a string as a negative prompt.
zsogitbe commentedon Feb 24, 2024
I see. This is an important feature since we can control model ethics with it easily. It is probably not a lot of work for someone who knows the code already very well.
martindevans commentedon Feb 24, 2024
Applying the guidance logits is fairly simple (I've actually just put together the code to do that).
The tricky part is that you need a second entire sequence running in parallel to the first to generate those guidance logits. This really needs batched execution to be fast, which is supported in LLamaSharp, but only very recently and it doesn't work with any kind of high level executor yet.
martindevans commentedon Feb 24, 2024
@zsogitbe I've implemented a demo for how to use classifier free guidance using the new batched executor. As I mentioned this is not as high level as the existing executors, but it should at least demo how things work at the lower level for now. If you could pull #536, test it out and leave any feedback on that PR, it would be very appreciated!
zsogitbe commentedon Feb 25, 2024
Added a possible bug issue: ggml-org/llama.cpp#5709