You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Yes, this works. But now I realised this completely defeats the purpose of the remove flag. The remove flag is there to save disk space after each conversion has been done. So this means the remove flag only makes sense when processing the files one after each other.
@ggerganov Do you think it makes sense to run the script in parallel by default and switch to serial processing when --remove-f16 is provided or do we want to have a separate orthogonal flag for parallel/serial processing?
I think it is better to multi-thread the quantize.cpp program.
Each tensor is divided in n parts and each of the n threads quantizes the corresponding part.
This way, even when quantizing the 7B model which has only 1 part, we will utilize all available CPU resources and still gain performance.
If you agree, either reformulate this issue and add "good first issue" tag or create a new one and close this.
Activity
prusnak commentedon Mar 14, 2023
The fix need to be more elaborate, because if you pass
--remove-f16then thermcommand is called before./quantizehas finished.Can you come up with a solution that does not have this issue?
prusnak commentedon Mar 14, 2023
Yes, this works. But now I realised this completely defeats the purpose of the remove flag. The remove flag is there to save disk space after each conversion has been done. So this means the remove flag only makes sense when processing the files one after each other.
@ggerganov Do you think it makes sense to run the script in parallel by default and switch to serial processing when
--remove-f16is provided or do we want to have a separate orthogonal flag for parallel/serial processing?tljstewart commentedon Mar 14, 2023
ah I see what you mean, swapping disk resources
ggerganov commentedon Mar 14, 2023
I think it is better to multi-thread the
quantize.cppprogram.Each tensor is divided in
nparts and each of thenthreads quantizes the corresponding part.This way, even when quantizing the 7B model which has only 1 part, we will utilize all available CPU resources and still gain performance.
If you agree, either reformulate this issue and add "good first issue" tag or create a new one and close this.
prusnak commentedon Mar 14, 2023
I agree. This makes sense especially for this reason:
ACK
FWIW, I really respect your shell skills @tljstewart 👍
prusnak commentedon Mar 19, 2023
Done another way (rewrite to python) in #222
toggle DLLAMA BUILD EXAMPLES to ON to included llama-finetune binaries (
Bitnet changes (ggml-org#106)
rdna-boosts: block 08: fused-core prefill kernels and GPU bit-identical
rdna-boosts: block 14: qwen4exp support
43 remaining items