Repository navigation
Spill kv-cache to nvme #47
Description
(hi first thx for this project, i'd not dreamed I could run such a model on a 2yo GPU!)
I've had some success with the ninfer 4090 project tiering kv cache to nvme, and on my system, there's no enough main RAM for keeping full kv cache, so I'm wondering if it would make sense to enable spilling kv cache out to an nvme ssd, so that across many different long agent sessions, refills stay rare. I'm thinking of taking a stab at it and sending a PR, but if you have initial thoughts I'd be happy to have them.
Activity
Niko1221 commentedon Sep 28, 2026
I will leave this opened. First I need to do some kernel magic and after that, I'll try this as well.
Niko1221 commentedon Sep 28, 2026
Keeping this open to track the NVMe tier: it will build on the shared snapshot core from #57 once that's merged (see #52).
Niko1221 commentedon Sep 30, 2026
The NVMe tier is part of the unified plan in #57 now (spill on eviction on the shared snapshot core). Closing in favour of #57.