Skip to content

Elastic K/V (--kv-grow, opt-in): the K/V takes VRAM as the context grows, the expert cache holds the rest (+68% cache on a 16 GB card at 262K) - #378

Closed
sergqwer wants to merge 3 commits into
Niko1221:mainfrom
sergqwer:pr/kv-grow
Closed

Elastic K/V (--kv-grow, opt-in): the K/V takes VRAM as the context grows, the expert cache holds the rest (+68% cache on a 16 GB card at 262K)#378
sergqwer wants to merge 3 commits into
Niko1221:mainfrom
sergqwer:pr/kv-grow

Elastic K/V: a run that cannot lend cache slots maps the whole window…

d5ee0cc
Select commit
Loading
Failed to load commit list.

Workflow runs completed with no jobs