You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
you probably don't want to use madvise+MADV_SEQUENTIAL, as in addition to increasing the amount of readahead it also causes pages to be evicted after they've been read - the entire model is going to be executed at least once per output token and read all the weights, MADV_SEQUENTIAL would potentially kick them all out and reread them repeatedly.
what may be more appropriate is to use MADV_WILLNEED on the whole model to kick off paging it all in without needing to wait for it to finish - but mmap can be tricky and you would probably want to make it an option rather than the default as it may not be a perf improvement on all setups and can wind up being slower than regular I/O due to causing lots of TLB shootdowns - you would want to benchmark it, as its not unlikely you may be trading improved time-to-first-token for worse overall throughput
That will definitely happen with posix_fadvise(sequential), which has a very gentle impact on file caches on Linux. What we might end up wanting here is madvise(random). In order to do that though, we first would need to find a way to avoid the loading and deserialization process where c/c++ data structures are constructed in memory, and instead have the runtime data structures just be mapped directly from the file. That would ensure 100% reduction in startup time, which means we can start generating tokens asap, and pages get loaded off disk on an as-needed basis. Once we're able to implement that design pattern, madvise(random) vs. madvise(sequential) would be a tool that lets the kernel know how to utilize an under-utilized disk, to make predictions on avoiding page faults.
I'm still getting up to speed on this codebase, so I'd like to hear everyone's ideas on how best we'd ensure object data structures (or their floating point content at the very least) could be made directly mappable, thus side-stepping the loading process. One dirty hack for example I've been considering, would be overriding the memory allocators to get all objects at a fixed address, and persisting that to disk. That way, when all the C/C++ objects are loaded back into memory using MAP_FIXED, no relocations would need to be performed. That's obviously a less portable and non-ideal solution, but it'd help us get instant loading happening as quickly as possible, and furthermore permit us an opportunity to explore precisely how sparse the model's memory usage patterns actually are.
Reacted by lin72h, Jeff Emanuel, Andrew Johnston, Felina Rivera Calzadillas (roguh), Zhennan LI, Yazan Agha-Schrader and Geunsik Lim
@jart
Thanks for stepping in. I will share briefly an idea that might be useful. Just mind I haven't looked into details of the discussion - will do in a few days once things cool off a bit here.
I think ggml_context is extremely well fit for mmap if I understand how it works. The ggml_context uses an externally provided buffer of memory with a pre-determined size:
All tensors and model parameters are "emplaced" into this buffer. There are no extra allocations ocuring.
Once you load the model, you can simply dump the memory buffer provided to ggml_context and next time, you can simply load this buffer instead of constructing it. Everything should work.
Edit: So above I incorrectly referenced the "eval" ggml_context which has it's own buffer. The "model" ggml_context is here:
All tensors and model parameters are "emplaced" into this buffer. There are no extra allocations ocuring.
In other words, there is a set memory cap?
Like many other people I have basically no experience in AIs or memory buffers
I'm just a guy pushing buttons until either something explodes or an ai becomes self aware lol
For loading from a physical/network drive resharding the larger models to a single file might help, imho. Whereas loading multiple files in parallel would be slower.
I'm getting ready to take another swing at it. My idea of what to do so far:
Create functions in utils.h called llama_load_buffer(), llama_save_buffer(), and llama_destroy_buffer(). These will mmap() (or just malloc() and read), save, and munmap() (or just free()) the buffers respectively. So, files saved on one machine can't necessarily be loaded by another. These files would be stored in some folder, and have the names of the original files. Either in models, or /tmp, or a new folder. This will also hopefully be useful for implementing saving the model state.
Add a new command line argument that tells llama_model_load() to look in this cache folder first. If it finds the file, llama_load_buffer() the file to get your ggml_init_params. Then do whatever else needs to be done (initialize the vocab, get hparams, etc) and exit the function. If the argument is present, call llama_save_buffer() first. Also, call llama_destroy_buffer() at the appropriate location.
I can do 1, I'll submit a PR for that shortly, but it isn't super clear to me how memory is laid out so that I can do 2. In particular, I'm wondering about the "whatever else needs to be done" part. I'm certain that I'm missing something, and it probably wouldn't be obvious to me what I'm breaking even after hours of monkeying.
My concern with doing that is, wouldn't it effectively double the disk usage? LLaMA is big enough that folks are already likely to be stretched thin on disk space after creating the second copy needed to quantize the model. I'm still working on studying the codebase to determine what exactly are the transformation that need to be made at runtime. For example, if it's just a bunch of float16's on disk, and we're using a bunch of float16's in memory, then I don't see why the buffer field of these tensors couldn't just be populated with pointers to the appropriate positions in the file. Unless of course it needed to be reshaped or shifted to meet things like AVX alignment requirements. In that case, we'd ideally want to modify the quantizer script so that it generates something suitable for our purposes, so that only a single conversion step needs to be performed.
After the model loads and the prompt is tokenized, create a hash of the context. If that hash exists in the cache directory (and flag is set), load state.
Saving the state has two modes, imho: post-prompt (only works when state hasn't been loaded) and at end of generation. The post-prompt mode allows jump-starting the model. End saving allows to start-up where one left off.
As to the memory organization - I'd leave that to existing code. The hash would act as pseudo-verification of that it's okay to load a buffer of bytes. The model and prompt would need to be the same, maybe even other options.
Just throwing out some head-space. Haven't starting coding anything.
Wait. What problem are we trying to solve here exactly? Are we trying to (1) eliminate the three second startup delay? Or are we trying to (2) store the changes made to memory back to disk? Because if your goal is to solve (2) then the only thing you need to save are the random seed and the prompt, since that would restore the state deterministically. Right now I'm focusing on (1) since having fast mmap() loading would not change llama.cpp's behavior, and would instead simply make it go faster. If you want (2) then this change could be the stepping stone you need. All you'd have to do is change MAP_PRIVATE to be MAP_SHARED instead, and whatever mutations are made to the tensors in memory will be transparently remembered on disk. However that's orthogonal to my intended goals at the moment.
@jart It would double the disk usage, yes. But so does converting the model, and so does quantizing it. I think people are prepared for this.
You're right though in that the scripts that convert the model are probably the best way to do this. I was only thinking about implementing a cache for ggml_init_params as originally suggested. Ideally though, everything should just be one call, for everything from vocab/hparams to model weights.
So I added some logging statements to track the read() operations that are happening. It's 200k+ lines that look like this:
moving 0x640 bytes from offset 0x4a607 to offset 0 (n_dims=2 n_parts=2)
moving 0x640 bytes from offset 0x4ac47 to offset 0xc80 (n_dims=2 n_parts=2)
moving 0x640 bytes from offset 0x4b287 to offset 0x1900 (n_dims=2 n_parts=2)
moving 0x640 bytes from offset 0x4b8c7 to offset 0x2580 (n_dims=2 n_parts=2)
moving 0x640 bytes from offset 0x4bf07 to offset 0x3200 (n_dims=2 n_parts=2)
moving 0x640 bytes from offset 0x4c547 to offset 0x3e80 (n_dims=2 n_parts=2)
moving 0x640 bytes from offset 0x4cb87 to offset 0x4b00 (n_dims=2 n_parts=2)
moving 0x640 bytes from offset 0x4d1c7 to offset 0x5780 (n_dims=2 n_parts=2)
moving 0x640 bytes from offset 0x4d807 to offset 0x6400 (n_dims=2 n_parts=2)
moving 0x640 bytes from offset 0x4de47 to offset 0x7080 (n_dims=2 n_parts=2)
moving 0x640 bytes from offset 0x4e487 to offset 0x7d00 (n_dims=2 n_parts=2)
moving 0x640 bytes from offset 0x4eac7 to offset 0x8980 (n_dims=2 n_parts=2)
moving 0x640 bytes from offset 0x4f107 to offset 0x9600 (n_dims=2 n_parts=2)
moving 0x640 bytes from offset 0x4f747 to offset 0xa280 (n_dims=2 n_parts=2)
moving 0x640 bytes from offset 0x4fd87 to offset 0xaf00 (n_dims=2 n_parts=2)
moving 0x640 bytes from offset 0x503c7 to offset 0xbb80 (n_dims=2 n_parts=2)
All it's doing is (1) reshaping and (2) aligning the data in the file. That's why llama.cpp takes several seconds to start. It wouldn't make sense to cache a bunch of memcpy() operations. The quickest thing we could do is introduce a third conversion step that creates a new file format, where the data is in the appropriate shape and alignment ahead of time. Then we could work our way backwards through the conversion tools, to reduce the number of pipeline chores from 3 to 1.
Reacted by Erik Scholz, TheCreeper, apaz, Nich, Wellington Nunes, joexner and Felina Rivera Calzadillas (roguh)
Activity
apaz-cli commentedon Mar 14, 2023
@ggerganov I'm working on putting together the PR. Almost done.
I don't know anything about the order that ggml accesses the weights in. Would you say that it's sequential? If so, there's also
madvise().apage43 commentedon Mar 14, 2023
you probably don't want to use madvise+
MADV_SEQUENTIAL, as in addition to increasing the amount of readahead it also causes pages to be evicted after they've been read - the entire model is going to be executed at least once per output token and read all the weights,MADV_SEQUENTIALwould potentially kick them all out and reread them repeatedly.what may be more appropriate is to use
MADV_WILLNEEDon the whole model to kick off paging it all in without needing to wait for it to finish - butmmapcan be tricky and you would probably want to make it an option rather than the default as it may not be a perf improvement on all setups and can wind up being slower than regular I/O due to causing lots of TLB shootdowns - you would want to benchmark it, as its not unlikely you may be trading improved time-to-first-token for worse overall throughputjart commentedon Mar 16, 2023
That will definitely happen with
posix_fadvise(sequential), which has a very gentle impact on file caches on Linux. What we might end up wanting here ismadvise(random). In order to do that though, we first would need to find a way to avoid the loading and deserialization process where c/c++ data structures are constructed in memory, and instead have the runtime data structures just be mapped directly from the file. That would ensure 100% reduction in startup time, which means we can start generating tokens asap, and pages get loaded off disk on an as-needed basis. Once we're able to implement that design pattern, madvise(random) vs. madvise(sequential) would be a tool that lets the kernel know how to utilize an under-utilized disk, to make predictions on avoiding page faults.I'm still getting up to speed on this codebase, so I'd like to hear everyone's ideas on how best we'd ensure object data structures (or their floating point content at the very least) could be made directly mappable, thus side-stepping the loading process. One dirty hack for example I've been considering, would be overriding the memory allocators to get all objects at a fixed address, and persisting that to disk. That way, when all the C/C++ objects are loaded back into memory using
MAP_FIXED, no relocations would need to be performed. That's obviously a less portable and non-ideal solution, but it'd help us get instant loading happening as quickly as possible, and furthermore permit us an opportunity to explore precisely how sparse the model's memory usage patterns actually are.ggerganov commentedon Mar 16, 2023
@jart
Thanks for stepping in. I will share briefly an idea that might be useful. Just mind I haven't looked into details of the discussion - will do in a few days once things cool off a bit here.
I think
ggml_contextis extremely well fit formmapif I understand how it works. Theggml_contextuses an externally provided buffer of memory with a pre-determined size:https://github.com/ggerganov/llama.cpp/blob/721311070e31464ac12bef9a4444093eb3eaebf7/main.cpp#L569-L575
All tensors
and model parametersare "emplaced" into this buffer. There are no extra allocations ocuring.Once you load the model, you can simply dump the memory buffer provided to
ggml_contextand next time, you can simply load this buffer instead of constructing it. Everything should work.Edit: So above I incorrectly referenced the "eval"
ggml_contextwhich has it's own buffer. The "model"ggml_contextis here:https://github.com/ggerganov/llama.cpp/blob/721311070e31464ac12bef9a4444093eb3eaebf7/main.cpp#L228-L240
Same stuff. If the pointer is
NULLit's allocated insideggmlfor convenience.nazthelizard122 commentedon Mar 16, 2023
In other words, there is a set memory cap?
Like many other people I have basically no experience in AIs or memory buffers
I'm just a guy pushing buttons until either something explodes or an ai becomes self aware lol
bitRAKE commentedon Mar 16, 2023
For loading from a physical/network drive resharding the larger models to a single file might help, imho. Whereas loading multiple files in parallel would be slower.
https://github.com/jankais3r/LLaMA_MPS/blob/main/reshard.py
apaz-cli commentedon Mar 16, 2023
I'm getting ready to take another swing at it. My idea of what to do so far:
Create functions in
utils.hcalledllama_load_buffer(),llama_save_buffer(), andllama_destroy_buffer(). These willmmap()(or justmalloc()and read), save, andmunmap()(or justfree()) the buffers respectively. So, files saved on one machine can't necessarily be loaded by another. These files would be stored in some folder, and have the names of the original files. Either inmodels, or/tmp, or a new folder. This will also hopefully be useful for implementing saving the model state.Add a new command line argument that tells
llama_model_load()to look in this cache folder first. If it finds the file,llama_load_buffer()the file to get yourggml_init_params. Then do whatever else needs to be done (initialize the vocab, get hparams, etc) and exit the function. If the argument is present, callllama_save_buffer()first. Also, callllama_destroy_buffer()at the appropriate location.I can do 1, I'll submit a PR for that shortly, but it isn't super clear to me how memory is laid out so that I can do 2. In particular, I'm wondering about the "whatever else needs to be done" part. I'm certain that I'm missing something, and it probably wouldn't be obvious to me what I'm breaking even after hours of monkeying.
jart commentedon Mar 16, 2023
My concern with doing that is, wouldn't it effectively double the disk usage? LLaMA is big enough that folks are already likely to be stretched thin on disk space after creating the second copy needed to quantize the model. I'm still working on studying the codebase to determine what exactly are the transformation that need to be made at runtime. For example, if it's just a bunch of float16's on disk, and we're using a bunch of float16's in memory, then I don't see why the buffer field of these tensors couldn't just be populated with pointers to the appropriate positions in the file. Unless of course it needed to be reshaped or shifted to meet things like AVX alignment requirements. In that case, we'd ideally want to modify the quantizer script so that it generates something suitable for our purposes, so that only a single conversion step needs to be performed.
bitRAKE commentedon Mar 16, 2023
(comment is more a #202 thing)
This is the way I was thinking about it:
After the model loads and the prompt is tokenized, create a hash of the context. If that hash exists in the cache directory (and flag is set), load state.
Saving the state has two modes, imho: post-prompt (only works when state hasn't been loaded) and at end of generation. The post-prompt mode allows jump-starting the model. End saving allows to start-up where one left off.
As to the memory organization - I'd leave that to existing code. The hash would act as pseudo-verification of that it's okay to load a buffer of bytes. The model and prompt would need to be the same, maybe even other options.
Just throwing out some head-space. Haven't starting coding anything.
jart commentedon Mar 16, 2023
Wait. What problem are we trying to solve here exactly? Are we trying to (1) eliminate the three second startup delay? Or are we trying to (2) store the changes made to memory back to disk? Because if your goal is to solve (2) then the only thing you need to save are the random seed and the prompt, since that would restore the state deterministically. Right now I'm focusing on (1) since having fast mmap() loading would not change llama.cpp's behavior, and would instead simply make it go faster. If you want (2) then this change could be the stepping stone you need. All you'd have to do is change
MAP_PRIVATEto beMAP_SHAREDinstead, and whatever mutations are made to the tensors in memory will be transparently remembered on disk. However that's orthogonal to my intended goals at the moment.bitRAKE commentedon Mar 16, 2023
I did not intent to expand the meaning of the thread. (2) should probably be addressed elsewhere. #202
apaz-cli commentedon Mar 16, 2023
@jart It would double the disk usage, yes. But so does converting the model, and so does quantizing it. I think people are prepared for this.
You're right though in that the scripts that convert the model are probably the best way to do this. I was only thinking about implementing a cache for
ggml_init_paramsas originally suggested. Ideally though, everything should just be one call, for everything from vocab/hparams to model weights.jart commentedon Mar 16, 2023
So I added some logging statements to track the read() operations that are happening. It's 200k+ lines that look like this:
All it's doing is (1) reshaping and (2) aligning the data in the file. That's why llama.cpp takes several seconds to start. It wouldn't make sense to cache a bunch of memcpy() operations. The quickest thing we could do is introduce a third conversion step that creates a new file format, where the data is in the appropriate shape and alignment ahead of time. Then we could work our way backwards through the conversion tools, to reduce the number of pipeline chores from 3 to 1.
96 remaining items
vulkan: chunked gated delta net prefill on RDNA3
hip : let routed experts take the measured mmb weight types, dense GE…