Repository navigation
[BUG] EXL3 crashes on model split on single GPU user - please allow exllamav2 again! #497
Description
OS
Linux
GPU Library
CUDA 12.x
Python version
3.12
Describe the bug
I have gpu_split_auto on false (shouldn't split anyways on single GPU user) and gpu_split with different values, empty [], 0, 11GB, 15GB, 16GB. No matter what it always crashes with the same error.
I tried to lower the cache down to 4096 and it still crashes.
VRAM gets from zero up to 12758MiB / 16380MiB and then crashes.
Reproduction steps
On a 4060 TI with 16GB VRAM try to load a 12B EXL3 at 5.0bpw
Python 3.14
Cuda 13.x
`Thu Oct 8 14:57:04 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4060 Ti Off | 00000000:01:00.0 On | N/A |
| 0% 48C P5 12W / 165W | 18MiB / 16380MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
`
Expected behavior
The bug is most likely not your fault but an [exllamav3 issue](turboderp-org/exllamav3#50 however turboderp closed this on Sep 5 (one month ago).
Alternatively please allow to use exllamav2 where this issue doesn't exist. You deprecated exl2 while the exl3 version you include is still buggy.
Logs
14:47:42.637 INFO: Response parsing: tool format mistral_old (tokenizer, architecture), no reasoning parsing
14:47:42.639 INFO: Loading model /mnt/nvmepci/llms/Violet_Magcap-12B-EXL3 (manual GPU split)
14:47:44.249 ERROR: Error Traceback (most recent call last):
File "/mnt/nvmepci/tabbyAPI/endpoints/OAI/utils/common_.py", line 214, in load_inline_model
await model.load_model(
...<2 lines>...
)
File "/mnt/nvmepci/tabbyAPI/common/model.py", line 265, in load_model
async for _ in load_model_gen(model_path, **kwargs):
pass
File "/mnt/nvmepci/tabbyAPI/common/model.py", line 233, in load_model_gen
async for module, modules in load_status:
...<17 lines>...
progress.stop()
File "/mnt/nvmepci/tabbyAPI/backends/exllamav3/model.py", line 908, in load_gen
async for value in iterate_in_threadpool(generator):
yield value
File "/mnt/nvmepci/tabbyAPI/common/concurrency.py", line 30, in iterate_in_threadpool
yield await asyncio.to_thread(gen_next, generator)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.14/asyncio/threads.py", line 26, in to_thread
return await loop.run_in_executor(None, func_call)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.14/concurrent/futures/thread.py", line 86, in run
result = ctx.run(self.task)
File "/usr/lib/python3.14/concurrent/futures/thread.py", line 73, in run
return fn(*args, **kwargs)
File "/mnt/nvmepci/tabbyAPI/common/concurrency.py", line 20, in gen_next
return next(generator)
File "/mnt/nvmepci/tabbyAPI/venv/lib/python3.14/site-packages/torch/utils/_contextlib.py", line 61, in generator_context
response = gen.send(request)
File "/mnt/nvmepci/tabbyAPI/backends/exllamav3/model.py", line 1037, in load_model_sync
for value in self.model.load_gen(
~~~~~~~~~~~~~~~~~~~^
tensor_p=self.use_tp,
^^^^^^^^^^^^^^^^^^^^^
...<5 lines>...
max_batch_size=self.max_batch_size,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
):
^
File "/mnt/nvmepci/tabbyAPI/venv/lib/python3.14/site-packages/exllamav3/model/model.py", line 639, in load_gen
yield from self._load_autosplit(
...<15 lines>...
)
File "/mnt/nvmepci/tabbyAPI/venv/lib/python3.14/site-packages/exllamav3/model/model_ls.py", line 301, in _load_autosplit
raise RuntimeError("Insufficient VRAM in split for model and cache")
RuntimeError: Insufficient VRAM in split for model and cache
14:47:44.254 ERROR: Sent to request: Model Violet_Magcap-12B-EXL3 failed to load: Insufficient VRAM in split for model and cache
Additional context
config.yml
model:
model_dir: "..."
current_model: ""
gpu_split_auto: false
gpu_split: []
max_seq_len: 4096
override_base_seq_len: 4096
cache_mode: Q8
cache_size: 4096
chunk_size: 512
fastlayers: true
draft_model: ""
gpu_peer_copy: false
rope_scale: 1.0
alpha_value: 1.0
Acknowledgements
- I have looked for similar issues before submitting this one.
- I have read the disclaimer, and this issue is related to a code bug. If I have a question, I will use the Discord server.
- I understand that the developers have lives and my issue will be answered when possible.
- I understand the developers of this program are human, and I will ask my questions politely.
Activity
turboderp commentedon Oct 9, 2026
I believe what's happening here is:
use_as_default. Otherwise your main-configcache_sizeandcache_modeare ignored.I've added some warnings to the output to make this a little clearer. But the fix should be either:
model:etc. sections you want to apply to that model specifically, or:cache_size,cache_modeetc. underuse_as_defaultin the main config, or:inline_model_loading: falseand load the model at startup with the main config optionsFor the last option, here is what your config could look like (Everything else would be the defaults from config_sample.yml):
If you still want to support inline loading, it could be, instead:
A couple of notes about the particular model:
The quant you seem to be using defines a linear RoPE scale in its config.json. The base model it's converted from doesn't, so that's likely to result in weird and broken behavior. I tested it at KLD=1.05 which is a huge deviation from the source model. I recommend removing these lines if you're having issues:
It's also an old quant. Without the section above it still seems to work fine (KLD=0.01), but it is still using an old codebook so performance may be less than you'd see from a more recent quant. It's a small model. so I made a 4.00 bpw and a 5.00 bpw version you can test with.
ExLlamaV2 is sadly too much of a maintenance burden to keep supporting. You can switch over to the
exl2_checkpointbranch, which contains the latest version that has EXL2 support. I don't recommend it, though, since ExLlamaV3 should ultimately use less memory than V2, run faster, and support many more architectures. It has fewer issues overall, receives updates, and it allows you to use smaller, higher-quality EXL3 quants.