Repository navigation
flux crashes with latest ggml #553
Description
I'm using ggml @ c8bd0fe and sd @ dcf91f9, and using the vulkan backend. Trying to run a flux model using this command line:
sd --diffusion-model flux1-dev-Q2_K.gguf --vae ae.safetensors --clip_l clip_l.safetensors --t5xxl t5xxl_fp16.safetensors -p "a lovely cat holding a sign says 'flux.cpp'" --cfg-scale 1.0 --sampling-method euler -v
I get a divide-by-zero crash in ggml_row_size because GGML_TYPE_Q4_0_4_8 is no longer supported after ggml-org/llama.cpp#10446. Is there a way to repack this, or do I need to use a different model or something? I'm generally just trying to run anything using flux to look at the performance.
Activity
stduhpf commentedon Jan 7, 2025
LostRuins commentedon Jan 7, 2025
@jeffbolznv according to the tensor list, q2k should not contain any GGML_TYPE_Q4_0_4_8 inside.
Did you check if the enums for
sd_type_tmatch exactly the enums forggml_type? There should be exactly the same number, withTYPE_COUNTfor both being equal to 39.jeffbolznv commentedon Jan 7, 2025
Ah, this was it. I'm not sure where I got Q4_0_4_8 from, running it in the debugger again I see the invalid format was 36 (equal to SD_TYPE_COUNT). It would be nice to be more robust to these mismatches, or at least static_assert that the COUNT values are the same.
stduhpf commentedon Jan 7, 2025
@LostRuins Good catch, that's exactly what's going on. I wrongly assumed that by "lastest ggml" jeffbolznv meant commit 6fcbd60, without paying attention to the details of the message.
stduhpf commentedon Jan 7, 2025
By the way @jeffbolznv, I know this is completely off topic, but do you know how we could make LoRAs work with quantized models on Vulkan? Right now it crashes with
Missing CPY op for types: f32 [quant_type]. I tried looking into it myself, but I can't grasp how the vulkan backend fundamentally works, so I had no luck.jeffbolznv commentedon Jan 7, 2025
I'm surprised it crashes since ggml_backend_vk_device_supports_op should return false for this. I thought ggml was supposed to fallback to the host for that, but maybe you have to use ggml a certain way to allow for the fallback.
I looked at cpy.cu, seems like it should not be terribly hard to implement similar shaders in Vulkan. Can you file an issue to track this? I may be able to do it soon.
stduhpf commentedon Jan 7, 2025
Here, or on the ggml repo?
jeffbolznv commentedon Jan 7, 2025
Maybe on llama.cpp, since that's where most work is happening.
leejet commentedon Nov 13, 2025
This should be fixed.