Add support for tokenize and untokenize of UTF-8 encoding in prompt/output - #87

Add support for tokenize and untokenize of UTF-8 encoding in prompt/output#87

wizd wants to merge 8 commits into

ggml-org:masterggml-org/llama.cpp:masterfrom

wizd:masterwizd/llama.cpp:masterCopy head branch name to clipboard