Clearer windows instructions.. please? #490

Activity

1. Waloumi commented on Mar 25, 2023

   

   Waloumi

   on Mar 25, 2023

   Author

   More actions

   ok i built the code with cmake and now in visual studio... i try to run it and it says access denied for ALL_BUILD

   ;_;

   😕React with 😕1Reacted by Tahar Amin Touzi
2. aicoat commented on Mar 25, 2023

   

   aicoat

   on Mar 25, 2023

   Last edited by aicoat

   More actions

   First you need to have all the additional files in LLaMa folder like tokenizer.model and tokenizer_checklist.chk and in that folder you shoud have 7B folder and inside of that checklist.chk , consolidated.00.pth , params.json

   A- you don't need to compile the repo yourself anymore , download windows release.

   B- Now you need to download the repo zip file in order to use python files!

   extract it in your ssd ,then:
   1- install python ( im using 3.6.10) and check add the system path during installation.
   2- run python -m pip install torch numpy sentencepiece and wait until it finishes.
   3- go to your extracted folder (step B) , clear windows address bar and type cmd then hit the enter to open up cmd in that directory.
   4- type python convert-pth-to-ggml.py D:\LLama\7B\ 1 ( note that you shoud change D:\llama... to your 7B location in your pc. but leave the "1" intact.
   5- after it is done you wil have a file named ggml-model-f16.bin in your model\7B directory
   6- now open your compiled release that you downloaded in the step A .
   7- once again open that folder and type cmd in the address bar and press enter
   8- now type quantize D:\LLama\7B\ggml-model-f16.bin D:\LLama\7B\ggml-model-q4_0.bin 2 ( again all the addresses should be change to your own data location!)
   9- it is done , all you need to do is run cmd in compiled folder (A) then run your prompts , like:
   main -m D:\LLaMA\7B\ggml-model-q4_0.bin -p "What is love?"

   main: prompt: ' What is love?'
   main: number of tokens in prompt = 5
        1 -> ''
     1724 -> ' What'
      338 -> ' is'
     5360 -> ' love'
    29973 -> '?'
   
   sampling parameters: temp = 0.800000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.100000
   
   
    What is love? Is it just a feeling or is it something more than that? Is there more to it than meets the eye?
   I think it's something more. It certainly can be felt, but I don't believe it's as simple as that. Love isn't simply feelings; it has to do with actions and reactions as well as emotions. Love is not a feeling of comfort or warmth like you might feel around your best friend or someone you care about.
   Love is something more than that.
   When I think of love, I can't help but picture flowers blooming in springtime.
   

   10- if you want to use larger model like 13B,30B,65B ,the method is the same BUT you have to run quantize command more , for example if step 4 gives you multiple files like ggml-model-f16.bin ,ggml-model-f16.bin.1 ggml-model-f16.bin.2 ... ggml-model-f16.bin.7 , you should run quantize one by one for each of them like:
   quantize D:\LLama\65B\ggml-model-f16.bin D:\LLama\65B\ggml-model-q4_0.bin 2
   quantize D:\LLama\65B\ggml-model-f16.bin.1 D:\LLama\65B\ggml-model-q4_0.bin.1 2
   quantize D:\LLama\65B\ggml-model-f16.bin.2 D:\LLama\65B\ggml-model-q4_0.bin.2 2
   .
   .
   .
   quantize D:\LLama\65B\ggml-model-f16.bin.7 D:\LLama\65B\ggml-model-q4_0.bin.7 2

   i think there are other ways too but it is how i do it in windows!
   correct me if there are better ways!

   Alternatively you can download ubuntu via wsl2 in windows and run it on linux .

   👍React with 👍2Reacted by unreal79 and Gary Mulder
3. anzz1 commented on Mar 25, 2023

   

   anzz1

   on Mar 25, 2023

   Last edited by anzz1

   Contributor

   More actions

   Python versioning is painful in general, and especially so in Windows.

   Exactly why I made this https://github.com/anzz1/python-windows/
   Just extract that to C:\python and click install.reg and you'll have a working multi-version python installation with none of the crap that comes with it using the official installer. (If you choose to use this, remove all your existing Python installations first to start fresh)

   The official installers can also completely mess everything up if you do things like installing Python 2 after installing Python 3, as some of the installer versions (iirc python2.x 32-bit) do not understand the concept of multiversioning and just overwrites the shit out of everything.

   The easiest option though is to uninstall all other versions of python and just installing python 3. If you keep multiple versions in windows and especially when package managers like pip, or worse, conda come into play and virtual environments (sigh..............), I guarantee you will be in a world of pain.
4. Waloumi commented on Mar 25, 2023

   

   Waloumi

   on Mar 25, 2023

   Author

   More actions

   Thank you..! 😄 Ill try those later when my head stops hurting...

   why are instructions so cryptic in general...

   👍React with 👍1Reacted by Tahar Amin Touzi
5. gjmulder commented on Mar 26, 2023

   

   gjmulder

   on Mar 26, 2023

   Contributor

   More actions

   Because we often spend a lot of time answering issues that don't use the issue template instead of spending time writing documentation.

   😄React with 😄1Reacted by Erik Scholz
6. gjmulder

   closed this as completedon Mar 26, 2023
7. Deadsg

   added a commit that references this issue on Dec 19, 2023

   Update development docs for scikit-build-core. Closes ggml-org#490

   This issue will close once commit 57db1f9 is merged into the 'master' branch.

   57db1f9
8. ddvnguyen

   added a commit that references this issue on Sep 13, 2026

   fix(moe-cache): stop a look-ahead prefetch from evicting a warm resident

   ...

   ec23a59