llama : refactor src/llama.cpp

[ggerganov]

Attempting to split the src/llama.cpp into a few separate modules. Very work-in-progress, mainly opening this PR for people to keep track and suggest improvements as we move along. This part does not involve functional changes, just code reorganization and decoupling to make it easier to work with the codebase. The batch and KV cache abstractions and reimplementations will be done in follow-up PRs.

graph TD;
chat;
model_loader;
model   --> arch[<b>arch </b>];
model   --> hparams[<b>hparams </b>];
model   ----> mmap[<b>mmap </b> <br><br> llama_file <br> llama_mmap <br> llama_mlock];
model   -.-> model_loader;
model   --> vocab;
vocab   --> unicode;
adapter -.-> model;
kv_cache -.-> batch;
kv_cache -.-> cparams;
kv_cache -.-> model;
context --> adapter[<b>adapter</b> <br><br> llama_adapter_cvec <br> llama_adapter_lora];
context -.-> batch;
context --> cparams;
context --> kv_cache;
context --> model;

style adapter fill:green
style arch fill:green
style batch fill:green
style chat fill:green
style cparams fill:green
style hparams fill:green
style kv_cache fill:green
style mmap fill:green
style model fill:green
style model_loader fill:green
style unicode fill:green
style vocab fill:green

Loading

TODO

• move the llama_mmaps and llama_mlocks from llama_model to llama_context? (no)
• change _internal suffix to _impl (next PR)
• add llama_tensor_loader ?
• model loading
• quantization

Conflicts

• llama : add support for Cohere2ForCausalLM #10900

[ngxson]

I think control_vector and lora related stuff should be re-grouped into a module, maybe called adapters (if someone has a better naming, feel free to comment). That's because they work kinda the same way, by "adding things" on top of the original cgraph.

[MaggotHATE]

After this PR, common_tokenize fails to tokenize any text, exits without console errors on llama_tokenize. Windows 10, static cpu build. Model and ctx are loaded at that moment (main-based application).

What change caused this? I don't see changes to llama_tokenize_internal function, for example. Are models required to be reconverted after this PR?

[ggerganov]

There should be no functional changes. Try to clean your build folder.

[MaggotHATE]

There should be no functional changes.

There's definitely a change, I see llama_n_ctx returning random numbers instead of the actual context size. At the same time, the context size is reported correctly on loading:

llama_new_context_with_model: n_ctx_per_seq (16384) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_kv_cache_init: kv_size = 16384, offload = 0, type_k = 'f16', type_v = 'f16', n_layer = 56, can_shift = 1

but later:

load: llama_n_ctx = 458134640

[ngxson]

There is a slight change in common_init_result, both model and ctx are now unique pointer. I'm not sure if it's the root problem. (i.e. you need to modify your code to use model.get() instead of just model)

Please note that we don't provide stable interface for common library, so undocumented breaking changes are expected.

[MaggotHATE]

you need to modify your code to use model.get() instead of just model

That's already done according to the changes in main this PR introduces. The problem is that everything before llama_n_ctx seems to work fine, but it looks like ctx either doesn't load properly, or is unloaded (which didn't happen before and probably shouldn't happen).

undocumented breaking changes

That's exactly what I'm trying to find.

[ngxson]

Be careful that with unique ptr, the ctx maybe deallocated when it goes out of scope. You should make sure that common_init_result always valid in the scope, otherwise .release() the ptr from unique ptr and use it as raw ptr.

[MaggotHATE]

ctx maybe deallocated when it goes out of scope

Yes, looks like that's what happens here. I see that llama_free(ctx) and llama_free_model(model) are no longer used, so deallocation and unloading happen automatically. It there a way to lock model and ctx manually now?

Although, it's a bit weird that this wasn't needed previously.

[ngxson]

llama_free is no longer needed because common_init_result is in scope of main(), and deleter will be called automatically when we exit from main() (go out of scope)

---

Another way is: (but remember to free it manually afterwards)

common_init_result llama_init = common_init_from_params(params);
llama_model * model = llama_init.model.release();
llama_context * ctx = llama_init.context.release();

llama_free(ctx);
llama_free_model(model);

[MaggotHATE]

llama_model * model = llama_init.model.release();
llama_context * ctx = llama_init.context.release();

Yes, thank you, that's exactly what I needed!

So, release() allows to match previous behavior (for example, reloading works exactly as it used to), while .get() relies on common_init_result (I tested it, and keeping common_init_result as a part of the object helps, but regen doesn't work - looks like it should be cleared/reset manually too).

[ngxson]

In cpp, you have notion of "ownership". For example, std::string owns a char * buffer under the hood that you can access it using str.data(), and buffer is freed once std::string goes out of scope.

Same for unique_ptr, llama_model * model is owned by unique_ptr and get() let you access it. You can also release() it from being owned.

unique_ptr cannot be reassigned, you can only reset() it or std::move it. Edit: it can be reassigned, which will be equivalent to reset()

[MaggotHATE]

Same for unique_ptr, llama_model * model is owned by unique_ptr and get() let you access it. You can also release() it from being owned.

I understand that, but the problem is that release() isn't used anywhere for model and ctx, so there are no examples to learn that from. Plus, in practical sense, functionality has changed in the examples (they are examples to learn from!), because it used to rely on free functions, while now they rely on simply going out of scope.