Skip to content

Multi-GPU: a --layer-split stage holds only its own layers' weights, and every card's leftover VRAM spills into a helper expert tier (4-way IQ3_S: 9,269 -> 14,172 expert slots, prefill chunk 2,048 -> 4,096) - #390

Closed
gopinath87607 wants to merge 17 commits into
Niko1221:mainfrom
gopinath87607:layer-split-weight-carve
Closed

Multi-GPU: a --layer-split stage holds only its own layers' weights, and every card's leftover VRAM spills into a helper expert tier (4-way IQ3_S: 9,269 -> 14,172 expert slots, prefill chunk 2,048 -> 4,096)#390
gopinath87607 wants to merge 17 commits into
Niko1221:mainfrom
gopinath87607:layer-split-weight-carve

configs: the 4-way IQ3_S launch config, the pair to the Q8_0 one

aa504eb
Select commit
Loading
Failed to load commit list.

Workflow runs completed with no jobs