You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Q] Memory Requirements for Different Model Sizes #13
Since the original models are using FP16 and llama.cpp quantizes to 4-bit, the memory requirements are around 4 times smaller than the original:
7B => ~4 GB
13B => ~8 GB
30B => ~16 GB
65B => ~32 GB
Reacted by Georgi Gerganov, parallelo3301, Satyajit Ghana, Hanusz Leszek, Zhicheng Sun, shraiwi, Satvik Pendem, leoCottret, Vigilence, Marius Nedelcu and 38 more
65B: 38.5GB, 850 ms per token
30B: 19.5GB, 450 ms per token
13B: 7.8GB, 150 ms per token
7B: 4.0GB, 75 ms per token
Reacted by Satyajit Ghana, Hanusz Leszek, Zhicheng Sun, jowodo, Lee Olayvar, ErrorNoInternet, AristarhSamos, Alexey Kudinkin, GODMapper, isometra and 22 more
For the record, Intel® Core™ i5-7600K CPU @ 3.80GHz × 4, 16Gb ram, under Ubuntu, model 13B runs with acceptable response time. Note that as mentioned by previous comments, -t 4 parameter gives the best results.
main: mem per token = 22357508 bytes
main: load time = 83076.67 ms
main: sample time = 267.12 ms
main: predict time = 193441.61 ms / 367.76 ms per token
main: total time = 277980.41 ms
Great work !
Reacted by Rintu Kutum, PhD, Chris Weed, mendhak, PizBernina, delta-whiplash and Yuang (Hilbert) Chen
Fwiw, running on my M2 Macbook Air w 8GB of ram comes to a grinding halt. At first run about 2-3 minutes of completely unresponsive machine (mouse and keyboard locked), then about 10-20 seconds per response word. Didn't expect great response times, but thats a bit slower than anticipated.
Close every other app, ideally reboot to clean state. This should help. If you see unresponsive machine, then it is swapping memory to disk. 8GB is not that much, especially if you have Browsers, Slack, etc. running.
Activity
satyajitghana commentedon Mar 11, 2023
prusnak commentedon Mar 11, 2023
Since the original models are using FP16 and llama.cpp quantizes to 4-bit, the memory requirements are around 4 times smaller than the original:
cannin commentedon Mar 12, 2023
With an M1 Max 64GB with 4-bit
65B: 38.5GB, 850 ms per token
30B: 19.5GB, 450 ms per token
13B: 7.8GB, 150 ms per token
7B: 4.0GB, 75 ms per token
dbddv01 commentedon Mar 15, 2023
For the record, Intel® Core™ i5-7600K CPU @ 3.80GHz × 4, 16Gb ram, under Ubuntu, model 13B runs with acceptable response time. Note that as mentioned by previous comments, -t 4 parameter gives the best results.
main: mem per token = 22357508 bytes
main: load time = 83076.67 ms
main: sample time = 267.12 ms
main: predict time = 193441.61 ms / 367.76 ms per token
main: total time = 277980.41 ms
Great work !
ggerganov commentedon Mar 15, 2023
Should add these to readme
sinanisler commentedon Mar 18, 2023
@prusnak is that pc ram or gpu vram ?
prusnak commentedon Mar 18, 2023
llama.cpp runs on cpu not gpu, so it's the pc ram
deniaud commentedon Mar 18, 2023
Is it possible that at some point we will get a video card version?
prusnak commentedon Mar 18, 2023
I don' think so. You can use run the original Whisper model on a GPU: https://github.com/openai/whisper
mrpher commentedon Mar 18, 2023
Fwiw, running on my M2 Macbook Air w 8GB of ram comes to a grinding halt. At first run about 2-3 minutes of completely unresponsive machine (mouse and keyboard locked), then about 10-20 seconds per response word. Didn't expect great response times, but thats a bit slower than anticipated.
prusnak commentedon Mar 18, 2023
Close every other app, ideally reboot to clean state. This should help. If you see unresponsive machine, then it is swapping memory to disk. 8GB is not that much, especially if you have Browsers, Slack, etc. running.
j-f1 commentedon Mar 18, 2023
Also make sure you’re using 4 threads instead of 8 — you don’t want to be using any of the 4 efficiency cores.
mrpher commentedon Mar 18, 2023
Working well now, good recommendations @prusnak @j-f1 thank you
112 remaining items
rdna-boosts: block 12: hybrid HIP all-reduce (RDNA4-gated)