You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hey, I was reading your Readme.md and I saw that your repo was hacked. I want to ask what this means and wanted to check if the users like me also get the impact of hacking. Or, this is not the thing I should worry about?
This was hacked in an evening - I have no idea if it works correctly.
"hack" was more often used in a more positive sense by enthusiasts in the early days of computing before it took on the additional negative connotation in modern times.
The computer senses date back to at least 1955 when it initially referred to creative problem solving.
4. (computing) To make a quick code change to patch a computer program, often one that, while being effective, is inelegant or makes the program harder to maintain.
Synonyms: frob, tweak
I hacked in a fix for this bug, but we'll still have to do a real fix later.
5. (computing) To accomplish a difficult programming task.
He can hack like no one else and make the program work as expected.
7. (computing, slang, transitive) To work with something on an intimately technical level.
8. (transitive, colloquial, by extension) To apply a trick, shortcut, skill, or novelty method to something to increase productivity, efficiency or ease.
It can still mean the act of compromising a system or obtaining unauthorized access, of course, but as with any word, context matters. (Like how you might've heard the phrase "life hack")
Reacted by Marius Rugan
Reacted by Zollner Robert, Nolan, Jakub Tarnawski, binarytrails, Reelix, Javel Rowe, shane1preston, Sridaran Thoniyil, Gabriel Lencioni Lovate, Gong Zhang and 52 more
Reacted by Christopher Madan, Ahmet Temiz, Joe Greener, July Hata, Hrvoje and Cyd0n1a
Ohh,, my bad. My primary language is not English lol
Reacted by Gong Zhang, Yugander Krishan Singh, D A V I D, Sahil Soni, Yifan Bao, Ron Wolf, Simone Fidanza, Kashif, Mahesh Chandran. G, Vinícius K-Max and 20 more
Reacted by Austin, Leonard Tan, Charlie Gadd and Eon
Reacted by jumango, itsme, wuhongsheng, c4big, EapRules-Revolt, reasv, Gabriel Lencioni Lovate, Uddeshya Singh, Chandan Routray, LiJX and 14 more
Reacted by Christopher Madan, Jakub Tarnawski, Pdonor, EapRules-Revolt, Anna, Tally Tarik, Lance J., Louanès, Baptiste Fontaine, sang . gnas and 62 more
Here is a short summary of the implementation (a.k.a. "hacking") process if anyone is interested - might be useful for porting other models:
Started out with the GPT-J example from the ggml repo
Used the 4-bit branch of ggml since it has initial quantization support that we want
The LLaMA model has a very similar architecture to GPT-J. It uses the same positional encoding (RoPE), similar activation function (SiLU instead of GELU). The main differences are:
no bias tensors
some new normalization layers
extra tensor in the feed-forward part
a slightly different order of the operations
seems context size is not fixed? (if I understand correctly the code)
All these are trivial changes that can be applied to the GPT-J example just by looking at the original Python LLaMA code
Modified the Python conversion script to read the .pth file of 7B model and dump it to ggml format as usual
The tokenizer was obviously more complex and problematic, but made a quick hack to at least support it partially
This was enough to get the LLaMA-7B running. Later, the rest of the models became supported by figuring out how to merge the original parts of the model thanks to some references from community
Through this process, there was no need to even run the original Python code. The downside is that I haven't had the chance to compare the outputs at different stages of the inference, so I have doubts about the correctness of this implementation. However, looking at the generated outputs, I guess it has to be correct.
Reacted by future10se, Felipe Novaes F Rocha, Ye Lin Aung, Hojin Yang, thakurudit, Umar Rudi, maurges, lin72h, jjyuhub, G2G2G2G and 369 more
Reacted by sharpe5, Sophie Bremer, Aditya Kane, jaytxrx, XYNTAXIS, Le Quang Tri, Tesla, YU_Yeqing, RomainShen, Yuantao Yang(Ayuan) and 13 more
Reacted by Nờ Lờ Pê, Ilya Baryshnikov, Serge Gotsuliak, Mahesh Madhav, noonething, Santiago Marruffo, tpoisonooo, Greggor's Joe, Rounak Datta, Nicolas Mäkelä and 89 more
Reacted by Ethan Steininger, Tones, Vinicius, Oskar Boëthius Lissheim, Kotaro MATSUOKA, Jeong-in Hwang, noonething, tpoisonooo, Javel Rowe, Greggor's Joe and 64 more
Activity
future10se commentedon Mar 12, 2023
If you're referring to the sentence:
"hack" was more often used in a more positive sense by enthusiasts in the early days of computing before it took on the additional negative connotation in modern times.
From wiktionary.org:
It can still mean the act of compromising a system or obtaining unauthorized access, of course, but as with any word, context matters. (Like how you might've heard the phrase "life hack")
effortprogrammer commentedon Mar 12, 2023
Ohh,, my bad. My primary language is not English lol
ggerganov commentedon Mar 12, 2023
Here is a short summary of the implementation (a.k.a. "hacking") process if anyone is interested - might be useful for porting other models:
ggmlrepoggmlsince it has initial quantization support that we want.pthfile of 7B model and dump it toggmlformat as usualHere is the LLaMA WIP branch in the
ggmlrepo that I then migrated to becomellama.cpp:https://github.com/ggerganov/ggml/tree/llama
Through this process, there was no need to even run the original Python code. The downside is that I haven't had the chance to compare the outputs at different stages of the inference, so I have doubts about the correctness of this implementation. However, looking at the generated outputs, I guess it has to be correct.
suprasteel commentedon Apr 1, 2023
meganfox-cmd commentedon Jan 6, 2025
Merge pull request ggml-org#33 from zoq/vulkan_tq2_0_type
Revert "Merge pull request ggml-org#33 from zoq/vulkan_tq2_0_type"
perf: fp16 centroid LUT — decode +6-14% at long context (#33)
perf: fp16 centroid LUT — decode +6-14% at long context (#33)
perf: fp16 centroid LUT — decode +6-14% at long context (ggml-org#33)
perf: fp16 centroid LUT — decode +6-14% at long context (#33)
perf: fp16 centroid LUT — decode +6-14% at long context (ggml-org#33)
55 remaining items