Repository navigation
4bit 65B model overflow 64GB of RAM #702
Description
Prerequisites
I am running the latest code. Development is very rapid so there are no tagged versions as of now.
I carefully followed the README.md.
I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
I reviewed the Discussions, and have a new bug or useful enhancement to share.
Expected Behavior
During inference, there should be no or minimum disk activities going on, and disk should not be a bottleneck once pass the model loading stage.
Current Behavior
My disk should have a continuous reading speed of over 100MB/s, however, during the loading of the model, it only loads at around 40MB/s. After this very slow loading of Llama 65b model (converted from GPTQ with group size of 128), llama.cpp start to inference, however during the inference the programme continue to occupy the disk and reads at 40MB/s. The generation speed is also extremely slow, at around 10 minutes per token.
However, if it's 30b model or smaller, llama.cpp work as expected.
Environment and Context
Note: My interfercing were done using oobabooga's text-generation-webui's implementation of llama.cpp, as I have no idea how to use llama.cpp by itself...
- Physical (or virtual) hardware you are using, e.g. for Linux:
CPU: Ryzen 5500
Flags:
fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mc
a cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall n
x mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_go
od nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl p
ni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2api
c movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_le
gacy svm extapic cr8_legacy abm sse4a misalignsse 3dnow
prefetch osvw ibs skinit wdt tce topoext perfctr_core p
erfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw
_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1
avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap c
lflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cq
m_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero
irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm
_lock nrip_save tsc_scale vmcb_clean flushbyasid decode
assists pausefilter pfthreshold avic v_vmsave_vmload vg
if v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid ove
rflow_recov succor smca fsrm
RAM: 64GB of DDR4 running at 3000MHz
Disk where I stored my model file: 2 Barraccuda 1TB HDD in Raid 1 configuration
System SSD: NV2 500GB
- Operating System, e.g. for Linux:
Linux fgdfgfthgr-MS-7C95 5.15.0-69-generic #76-Ubuntu SMP Fri Mar 17 17:19:29 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
- SDK version, e.g. for Linux:
Python 3.9.13
GNU Make 4.3
g++ (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Failure Information (for bugs)
Not sure what other information is there to provide.
Steps to Reproduce
- Load a 65b model using oobabooga's text-generation-webui's implementation of llama.cpp.
- Use
iostat -y -d 5to monitor disk activity during loading and inference.
Failure Logs
Llama.cpp version:
https://pypi.org/project/llamacpp/
0.1.11
Pip environment:
accelerate 0.18.0
aiofiles 23.1.0
aiohttp 3.8.4
aiosignal 1.3.1
altair 4.2.2
anyio 3.6.2
async-timeout 4.0.2
attrs 22.2.0
bitsandbytes 0.37.2
certifi 2022.12.7
charset-normalizer 3.1.0
click 8.1.3
cmake 3.26.1
contourpy 1.0.7
cycler 0.11.0
datasets 2.11.0
dill 0.3.6
entrypoints 0.4
fastapi 0.95.0
ffmpy 0.3.0
filelock 3.10.7
flexgen 0.1.7
fonttools 4.39.3
frozenlist 1.3.3
fsspec 2023.3.0
gradio 3.24.0
gradio_client 0.0.5
h11 0.14.0
httpcore 0.16.3
httpx 0.23.3
huggingface-hub 0.13.3
idna 3.4
Jinja2 3.1.2
jsonschema 4.17.3
kiwisolver 1.4.4
linkify-it-py 2.0.0
lit 16.0.0
llamacpp 0.1.11
Markdown 3.4.3
markdown-it-py 2.2.0
MarkupSafe 2.1.2
matplotlib 3.7.1
mdit-py-plugins 0.3.3
mdurl 0.1.2
mpmath 1.3.0
multidict 6.0.4
multiprocess 0.70.14
networkx 3.0
numpy 1.24.2
nvidia-cublas-cu11 11.10.3.66
nvidia-cuda-cupti-cu11 11.7.101
nvidia-cuda-nvrtc-cu11 11.7.99
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cudnn-cu11 8.5.0.96
nvidia-cufft-cu11 10.9.0.58
nvidia-curand-cu11 10.2.10.91
nvidia-cusolver-cu11 11.4.0.1
nvidia-cusparse-cu11 11.7.4.91
nvidia-nccl-cu11 2.14.3
nvidia-nvtx-cu11 11.7.91
orjson 3.8.9
packaging 23.0
pandas 1.5.3
peft 0.2.0
Pillow 9.4.0
pip 23.0.1
psutil 5.9.4
PuLP 2.7.0
pyarrow 11.0.0
pydantic 1.10.7
pydub 0.25.1
pyparsing 3.0.9
pyrsistent 0.19.3
python-dateutil 2.8.2
python-multipart 0.0.6
pytz 2023.3
PyYAML 6.0
quant-cuda 0.0.0
regex 2023.3.23
requests 2.28.2
responses 0.18.0
rfc3986 1.5.0
rwkv 0.7.1
safetensors 0.3.0
semantic-version 2.10.0
sentencepiece 0.1.97
setuptools 65.6.3
six 1.16.0
sniffio 1.3.0
starlette 0.26.1
sympy 1.11.1
tokenizers 0.13.2
toolz 0.12.0
torch 2.0.0
torchaudio 2.0.1
torchvision 0.15.1
tqdm 4.65.0
transformers 4.28.0.dev0
triton 2.0.0
typing_extensions 4.5.0
uc-micro-py 1.0.1
urllib3 1.26.15
uvicorn 0.21.1
websockets 10.4
wheel 0.38.4
xxhash 3.2.0
yarl 1.8.2
(textgen) fgdfgfthgr@fgdfgfthgr-MS-7C95:/mnt/7018F20D48B6C548/text-generation-webui$ y-it-py 2.0.0
lit 16.0.0
llamacpp 0.1.11
Markdown 3.4.3
markdown-it-py 2.2.0
MarkupSafe 2.1.2
matplotlib 3.7.1
mdit-py-plugins 0.3.3
mdurl 0.1.2
mpmath 1.3.0
md5sum ggml-model-q4_0.bin
3073a8eedd1252063ad9b440af7c90cc ggml-model-q4_1.bin
Activity
jart commentedon Apr 2, 2023
There's two issues you're reporting here:
Suboptimal disk throughput during first run loading time. The Linux Kernel sometimes isn't very good at making major page faults go fast in my experience on commodity machines undergoing memory pressure, like for instance, loading 65B on a system with 64GB of RAM. Especially if you have things like X and Chrome open at the same time. You're cutting it pretty tight. It's sort of like if you were asked to dance inside a cage that's just big enough for your body. It wouldn't be a stunning performance.
Inference performance time regression, possibly due to swapping and memory pressure. In this case you may want to consider using the
--mlockflag which will force the memory to not be swapped. You have to have root privileges if you do this. Please note that, rather than making LLaMA load slower, this might just nuke your system instead if the memory pressure is indeed that high.Let me know if any of the above suggestions work for you! We should ideally be able to stretch our RAM budgets as far as possible. Being able to operate in tight constraints is a hallmark of good engineering. So I'd like to see us be able to do as much as possible for you. I just don't know how much we can do.
diimdeep commentedon Apr 2, 2023
I observe similar behavior with 78ca983 with 7B model on 8 GB RAM macOS machine (Haswell, 2 core).
If there is nearly not enough free space for model to fully load, performance drops from 600 ms/token down to 3 minutes/token. Instead of CPU bound performance with 0 disk activity, I observe disk bound performance (30-90% CPU instead of ~190% and constant 160MB/s read from disk)
Details
and here is when it is fully loaded
Details
jart commentedon Apr 2, 2023
Thank you for sharing such rich technical details @diimdeep. Have you evaluated our
--mlockflag under these conditions? Under memory pressure, obviously something on the system is going to have to pay, and using--mlockcan help ensure it isn't LLaMA. But it's an unusual situation to both test and find oneself in. So it'd be great to hear feedback on whether or not that flag is helping.diimdeep commentedon Apr 2, 2023
Yeah
--mlockhelps in that regard, when you using other apps while model is running and memory is scarce. But had to tweak system to make it workfgdfgfthgr-fox commentedon Apr 3, 2023
Just checked, the system did running out of RAM when loading 65B model, and hence the bad performance. However, 65B model should only require around 38.5 GB of RAM, and I got 64GB here. 25.5GB of extra RAM should be plenty for any reasonable overhead + firefox + linux kernel.
note: there were no background programmes running in the screenshot above.
fgdfgfthgr-fox commentedon Apr 3, 2023
And I doubt adding --mlock would help in that regard, as adding it doesn't change the fact that most RAM is used up by buffer than the cache. Maybe there is a bug in webui's code or llama.cpp's code causing the buffer usage?
[-]Disk bottleneck in 65B model[/-][+]65B model overflow 64GB of RAM[/+][-]65B model overflow 64GB of RAM[/-][+]4bit 65B model overflow 64GB of RAM[/+]fgdfgfthgr-fox commentedon Apr 19, 2023
Seems to be fixed at this point.
Fix issue with Literal and Optional cli arguments not working. Closes g…