Repository navigation
Windows 64-bit, Microsoft Visual Studio - it works like a charm after those fixes! #22
Closed as not planned
Description
First of all thremendous work Georgi! I managed to run your project with a small adjustments on:
- Intel(R) Core(TM) i7-10700T CPU @ 2.00GHz / 16GB as x64 bit app, it takes around 5GB of RAM.
Here is the list of those small fixes:
- main.cpp: added ggml_time_init() at start of main (division by zero otherwise)
- quantize.cpp: same as above at start of main (division by zero otherwise)
- ggml.c: #define QK 32 moved to dedicated define.h (should not be in .c)
- ggml.c: replace fopen with fopen_s (VS secure error message)
- ggml.c: below changes due to 'expression must be a pointer or complete object type':
- 2x
(uint8_t*)(yto:((uint8_t*)y - 4x
(const uint8_t*)(xto((const uint8_t*)x - 2x
(const uint8_t*)(yto((const uint8_t*)y
- quantize.cpp: removed qk in ggml_quantize_q4_0 & ggml_quantize_q4_1 calls
- utils.cpp: use of QK value instead of parameter value (VS raise error for:
uint8_t pp[qk / 2];)
It would be really great if you could incorporate those small fixes.
Activity
etra0 commentedon Mar 11, 2023
Interesting, doing these changes (and a couple of more hacks) I was able to run the 13B model on my HW (AMD Ryzen 7 3700X 8-Core Processor, 3593 Mhz, 8 Core(s), 16 Logical Processor(s), 32gb RAM) and I was able to get 268ms per token, with around 8GB of ram usage!
I forced the usage of AVX2 and that gave a huge speed up.
Apply fixes suggested to build on windows
bsiminski commentedon Mar 11, 2023
@etra0 here are my 13B model tests, based on number of threads & AVX2 (thanks!):
4: 3809.57 ms per token (default settings)
8: 3617.09 ms per token (default settings)
12: 2967.79 ms per token (default settings)
4: 495.08 ms per token (with AVX2)
8: 519.78 ms per token (with AVX2)
12: 490.53 ms per token (with AVX2)
Clearly AVX2 gives a huge boost. I see however that you are still way ahead with your 268 ms. What other optimizations do you have?
ggerganov commentedon Mar 11, 2023
Yes, AVX2 flags are very important for high performance.
Could you wrap these changes in a PR?
This is not very desirable - I don't want an extra file added. Although the QK constants everywhere are indeed problematic.
Some other fix?
etra0 commentedon Mar 11, 2023
I could do that, but I'm unsure whether to create a Solution, or move the project to CMake, because Windows doesn't support Make by default, sadly.
I always try to avoid Solutions because they're not multiplatform, but from looking at the makefile, rewriting it to CMake would take a bit more time. In the meantime I could do a PR to fix the things that won't compile.
ggerganov commentedon Mar 11, 2023
CMake is better than Solutions. The https://github.com/ggerganov/whisper.cpp project has a CMake build system that is compatible for Windows and the project is very similar. It should be easy to adapt
Apply fixes suggested to build on windows
kamyker commentedon Mar 12, 2023
Great! These changes finally fixed compilation for me using VS cl command (#2) and also cmake with @etra0 repo.
I get 140ms per token on i9900k and about 5gb ram usage with 7B.
Unfortunately bigger prompts are kind of unusable. Dno if it's windows issue or this library isn't yet optimized in this case. Making hardcoded 512 token limit a parameter was easy to change but it's too slow as it repeats all prompt tokens.
ggerganov commentedon Mar 12, 2023
@kamyker
Maybe the context size has to be increased - it's currently hardcoded to 512:
https://github.com/ggerganov/llama.cpp/blob/da1a4ff01f42d058cfa59806dd5679c0fe5a8604/main.cpp#L768
Haven't tested if it works with other values
jaykrell commentedon Mar 12, 2023
I didn't see your PR when I read the issue so went ahead and made one, very similar.
I made the existing Makefile work on Unix and Microsoft nmake.
#36
0xbitches commentedon Mar 12, 2023
Using the fix in #31, however, the results from 4 bit models are still repetitive nonsense. FP16 works but the results are also very bad.
Relevant spec: Intel-13700k, 240ms/token
Built make.exe with mwing64
kamyker commentedon Mar 12, 2023
As I said, I made parameter out of it and it fixes longer prompts but they are still slow. What I'm saying is that without some kind of quicker prompt loading/caching this is very far from ChatGPT.
How's let's say 300 tokens prompt for you?
38 remaining items
docs: expand remaining optimization opportunities
feat: LMCache Hash 索引优化 - Chunk-Level Bitmap Index (Task ggml-org#22)
register TURBO dequant pipelines + GET_ROWS supports_op in Vulkan bac…
CPU1 op-fusion Phase 2: MUL_MAT+residual at graph construction (qwen3…
AVX2 quantization for Q8_K (ggml-org#22)
Windows fixes (#31)
readme : adopt the strix-llama.cpp README for this fork (ggml-org#22)
readme : adopt the strix-llama.cpp README for this fork (ggml-org#22)
qwen4exp : gather the QSA row tiles from one device copy (ggml-org#22)
Merge branch 'rc/next-qsamem' into rc/release-next