You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Did some testing on my machine (AMD 5700G with 32GB RAM on Arch Linux) and was able to run most of the models. With the 65B model, I would need 40+ GB of ram and using swap to compensate was just too slow.
(Prompt was "They" on seed 1678609319)
Quantized Model
Threads
Memory use
Time per token
llama-7b
4
4.2GB
137.10 ms
llama-7b
6
4.2GB
100.43 ms
llama-7b
8
4.2GB
112.44 ms
llama-7b
10
4.2GB
131.63 ms
llama-7b
12
4.2GB
132.73 ms
llama-13b
4
7.9GB
261.88 ms
llama-13b
6
7.9GB
190.74 ms
llama-13b
8
7.9GB
209.15 ms
llama-13b
10
7.9GB
244.64 ms
llama-13b
12
7.9GB
257.72 ms
llama-30b
4
19GB
645.15 ms
llama-30b
6
19GB
463.04 ms
llama-30b
8
19GB
476.64 ms
llama-30b
10
19GB
583.19 ms
llama-30b
12
19GB
593.75 ms
My PC has 8 cores, so it seems like with whisper.cpp keeping threads at 6/7 gives the best results.
Reacted by AB0x, Shiyu Yuan(袁诗煜), e-mon and Nuclear6
Reacted by Andrea Tupini, Andreas Burger and Raphaël Merx
@ggerganov It looks like very nice, on CPU but gives a reasonable speed. I can run 13b even on my PC, how do u think the inference speed on a more long prompt tokens?
mem per token = 70897348 bytes
load time = 68146.04 ms
sample time = 1002.82 ms
predict time = 478729.38 ms / 936.85 ms per token
total time = 550394.94 ms
It scales with real cores. Once you get to virtual cores (threads) it starts going badly.
If you have a 8 core 16 thread use 8 cores... or a 24 core 48 thread use 24 cores etc
I have 24 real cores, but if you look at the numbers above, it seems to hit a wall at around 12 threads, and barely improves when doubling the threads to 21/24
Interesting, with almost the same setup as the top comment (AMD 5700G with 32GB RAM but Linux Mint) I get about 20% slower speed per token.
Maybe prompt length had something to do with it, or my memory dimms are slower, or arch faster, combinations of all those?
Not an issue, just mildly curious.
7b (6 threads):
main: predict time = 35909.73 ms / 120.91 ms per token
13b (6 threads):
main: predict time = 67519.31 ms / 227.34 ms per token
30b (6 threads):
main: predict time = 165125.56 ms / 555.98 ms per token
Interesting, with almost the same setup as the top comment (AMD 5700G with 32GB RAM but Linux Mint) I get about 20% slower speed per token. Maybe prompt length had something to do with it, or my memory dimms are slower, or arch faster, combinations of all those? Not an issue, just mildly curious.
7b (6 threads): main: predict time = 35909.73 ms / 120.91 ms per token
13b (6 threads): main: predict time = 67519.31 ms / 227.34 ms per token
30b (6 threads): main: predict time = 165125.56 ms / 555.98 ms per token
My assumption is memory bandwidth, my per core speed should be slower than yours according to benchmarks, but when I run with 6 threads I get faster performance. My RAM is slow, but 8 memory channels vs 2 makes up for that I guess.
Interesting how the fastest runs are t=4 and t=8 with the ones between being slower.
In comparison in I'm getting around 20-25 tokens/s (40-50 ms/token) on a 3060ti with the 7B model in text-generation-webui with the same prompt (although it gets much slower with higher amounts of context). If only GPUs had cheap, expandable VRAM.
Interesting, with almost the same setup as the top comment (AMD 5700G with 32GB RAM but Linux Mint) I get about 20% slower speed per token. Maybe prompt length had something to do with it, or my memory dimms are slower, or arch faster, combinations of all those? Not an issue, just mildly curious.
7b (6 threads): main: predict time = 35909.73 ms / 120.91 ms per token
13b (6 threads): main: predict time = 67519.31 ms / 227.34 ms per token
30b (6 threads): main: predict time = 165125.56 ms / 555.98 ms per token
My assumption is memory bandwidth, my per core speed should be slower than yours according to benchmarks, but when I run with 6 threads I get faster performance. My RAM is slow, but 8 memory channels vs 2 makes up for that I guess.
Ah, yes. I have on my system Crucial 3200 MHz DDR4 (16GB x 2) kit, but all this time I had it running at 2666MHz, for whatever reason. I actually didn't expect memory to be such bottleneck on this workload, I would have blamed CPU exclusively for every millisecond.
Now after changing settings in my BIOS, at 3200 MHz, numbers still not exactly on par, but close enough:
7B:
main: predict time = 31586.56 ms / 106.35 ms per token
13B:
main: predict time = 59035.98 ms / 198.77 ms per token
30B:
main: predict time = 139936.17 ms / 484.21 ms per token
This might be a dumb question but is there any way to reduce the memory requirements even if it increases inference time? Or is this a constant based on the model architecture and weights?
This might be a dumb question but is there any way to reduce the memory requirements even if it increases inference time?
Currently no, other than adding a lot of swap space, but even with a fast NVMe drive it will be orders of magnitude slower than running fully in memory.
Activity
headmetwall commentedon Mar 12, 2023
Did some testing on my machine (AMD 5700G with 32GB RAM on Arch Linux) and was able to run most of the models. With the 65B model, I would need 40+ GB of ram and using swap to compensate was just too slow.
(Prompt was "They" on seed 1678609319)
ElRoberto538 commentedon Mar 12, 2023
Will run the same tests on an EPYC 7443P to compare, should be able to run 65B - copying to my SSD now.
lucasjinreal commentedon Mar 12, 2023
@ggerganov It looks like very nice, on CPU but gives a reasonable speed. I can run 13b even on my PC, how do u think the inference speed on a more long prompt tokens?
breakpointninja commentedon Mar 12, 2023
Ryzen 9 5900X on llama-65b eats 40GB ram.
ElRoberto538 commentedon Mar 12, 2023
AMD EPYC 7443P 24 core (in VM).
Prompt was "They" on seed 1678609319, as above.
Interestingly it doesn't seem to scale well with cores, I guess it likes a few fast cores and high memory bandwidth?
G2G2G2G commentedon Mar 12, 2023
It scales with real cores. Once you get to virtual cores (threads) it starts going badly.
If you have a 8 core 16 thread use 8 cores... or a 24 core 48 thread use 24 cores etc
ElRoberto538 commentedon Mar 12, 2023
I have 24 real cores, but if you look at the numbers above, it seems to hit a wall at around 12 threads, and barely improves when doubling the threads to 21/24
pugzly commentedon Mar 12, 2023
Interesting, with almost the same setup as the top comment (AMD 5700G with 32GB RAM but Linux Mint) I get about 20% slower speed per token.
Maybe prompt length had something to do with it, or my memory dimms are slower, or arch faster, combinations of all those?
Not an issue, just mildly curious.
7b (6 threads):
main: predict time = 35909.73 ms / 120.91 ms per token
13b (6 threads):
main: predict time = 67519.31 ms / 227.34 ms per token
30b (6 threads):
main: predict time = 165125.56 ms / 555.98 ms per token
ElRoberto538 commentedon Mar 12, 2023
My assumption is memory bandwidth, my per core speed should be slower than yours according to benchmarks, but when I run with 6 threads I get faster performance. My RAM is slow, but 8 memory channels vs 2 makes up for that I guess.
plhosk commentedon Mar 13, 2023
Speeds on an old 4c/8t intel i7 with above prompt/seed:
7B, n=128
t=4 165 ms/token
t=5 220 ms/token
t=6 188 ms/token
t=7 168 ms/token
t=8 154 ms/token
13B
t=4 314 ms/token
t=5 420 ms/token
t=6 360 ms/token
t=7 314 ms/token
t=8 293 ms/token
Interesting how the fastest runs are t=4 and t=8 with the ones between being slower.
In comparison in I'm getting around 20-25 tokens/s (40-50 ms/token) on a 3060ti with the 7B model in text-generation-webui with the same prompt (although it gets much slower with higher amounts of context). If only GPUs had cheap, expandable VRAM.
pugzly commentedon Mar 13, 2023
Ah, yes. I have on my system Crucial 3200 MHz DDR4 (16GB x 2) kit, but all this time I had it running at 2666MHz, for whatever reason. I actually didn't expect memory to be such bottleneck on this workload, I would have blamed CPU exclusively for every millisecond.
Now after changing settings in my BIOS, at 3200 MHz, numbers still not exactly on par, but close enough:
7B:
main: predict time = 31586.56 ms / 106.35 ms per token
13B:
main: predict time = 59035.98 ms / 198.77 ms per token
30B:
main: predict time = 139936.17 ms / 484.21 ms per token
dennislysenko commentedon Mar 13, 2023
This might be a dumb question but is there any way to reduce the memory requirements even if it increases inference time? Or is this a constant based on the model architecture and weights?
plhosk commentedon Mar 14, 2023
Currently no, other than adding a lot of swap space, but even with a fast NVMe drive it will be orders of magnitude slower than running fully in memory.
prusnak commentedon Mar 18, 2023
Memory/disk requirements are being added in #269
As for the inference speed, feel free to discuss here, but I am closing this issue.
3 remaining items
feat: perf opt part2 (ggml-org#39)
experiment: fused centroid decode — half cn_norm[8] also spills on Metal
experiment: bit-arithmetic centroid decode — ZERO memory access
feat: auto-detect M1/M2 and use bit-arithmetic dequant
docs: benchmark results for turbo4 fix, sparse V, multi-model validation
Add support for bf16 to iqk_mul_mat (ggml-org#39)
Merge pull request ggml-org#39 from Tongas/fix/cublas-zero-dim-specul…
Merge pull request ggml-org#39 from PrismML-Eng/feat/cuda-hopper-q1-w…
Merge ggml-org#45: Hopper wgmma review follow-up (multi-GPU correctne…
feat(l10n): patches 36-40 — Default/Custom JSON/CSV/Samplers/Import-E…
feat(l10n): patches 36-42 + ru.json 246 фраз — финальная русификация …