[Feature]: degrade the conversation cache across levels instead of deleting the oldest (RAM -> fast M.2 -> cold disk, each level switchable) #2597

Activity

1. destroyor commented on Oct 17, 2026

   

   destroyor

   on Oct 17, 2026

   More actions

   Same need here, from a single-card box. Adding measured numbers, and a small separate change that does not need the cascade to land.

   Numbers

   GPU / CPU / RAM	RX 7900 XTX (gfx1100) / Ryzen 9 9950X / 96 GB DDR5-6400
   Engine	0.1.45, HIP source build, --kv k8v4 --kv-resident 32768
   Model	Qwen3.8-Flash-Next IQ3_S native pack
   Config	--max-context 262144 --prefill auto:32768 --conversation-cache-mib 16384 --conversation-cache-slots 8 --conversation-cache-min-tokens 32768

   • A parked 130K-token conversation is ~3.6 GiB in RAM. Four of them fill 14.5 GiB of a 16 GiB
     budget and the counter reaches evictions=10.
   • Losing one costs a 95 s cold read:
     prompt 161271 tokens = 0 reused + 161271 read in 95469 ms (1689.3 tok/s).
   • SAVE / RESTORE on the session-file path: 528 ms / 815 ms for a 1.45 GB file (0.1.43.1, same
     box). File size is about 1.2 GB per 63K tokens.

   So RAM is roughly 4x the cost per conversation and 5x faster than the disk path. A 2-hot + 4-8-cold
   split would bound RAM at ~7 GiB with every switch under ~1.5 s - the same shape as the cascade
   asked for above.

   A step that can land before the cascade

   The RAM level already exists, but it is not visible in the web UI. serve/runconfig.py's EDITABLE
   list drives Settings, its ("arg", flag) kind already reads and writes the config's args, and
   app.js renders whatever GET /config returns - so no front-end change is needed:

   --- a/serve/runconfig.py
   +++ b/serve/runconfig.py
   @@ -36,6 +36,15 @@ EDITABLE = [
        ("open_browser", "bool", "Open the chat page in the browser when the model is ready"),
        ("vram_reserve_mib", ("arg", "--vram-reserve-mib"),
         "VRAM in MiB the engine leaves free for other programs (engine default 700)"),
   +    ("conversation_cache_mib", ("arg", "--conversation-cache-mib"),
   +     "RAM in MiB the engine keeps parked conversations in, so several chats stay switchable "
   +     "without a re-read (0 = off; needs a prompt cache)"),
   +    ("conversation_cache_slots", ("arg", "--conversation-cache-slots"),
   +     "How many conversations that RAM cache holds (engine default 4)"),
   +    ("conversation_cache_min_tokens", ("arg", "--conversation-cache-min-tokens"),
   +     "Park only conversations of at least this many tokens, so short side requests do not take "
   +     "the slots a long one needs (0 = park any)"),
    ]

   That puts level 1 (--conversation-cache-mib) next to the VRAM reserve, which is the knob most
   people need while a cascade does not exist yet.

   Note on the disk level

   I could not find a spill or archive path in 0.1.45: --help lists only five --conversation-cache-*
   flags, origin/main is at the same commit as the v0.1.45 tag, and
   grep -rn "spill\|conversation-cache-archive" src include serve finds nothing. If the
   spill dir ready line quoted above comes from a build I cannot see, ignore this note - otherwise the
   disk level may still be unbuilt.

   Not tested

   The diff was written by reading the 0.1.45 tree and has not been built or run. The numbers are
   one box, one quantization, one GPU, one request at a time.

   👍React with 👍1Reacted by Shahrokh
2. shahrokhzargarpour commented on Oct 17, 2026

   

   shahrokhzargarpour

   on Oct 17, 2026

   Author

   More actions

   Thanks for the detailed analysis and measurements, destroyor! You're absolutely right that waiting for the full cascade to test the RAM cache is too much. I think your proposal to expose those flags in the UI as a preliminary step is excellent. I'm going to break up the work so you can test it as soon as possible.
3. havok2-htwo commented on Oct 18, 2026

   

   havok2-htwo

   on Oct 18, 2026

   Contributor

   More actions

   The issue lists the reload cost of an eviction as not measured; destroyor's comment above gives one cold re-read (161K tokens, 95 s). We have not measured that cost per eviction either. We do have aggregate data from our fork (tier code written on 0.1.43 and carried into our 0.1.44.1-based builds; not upstream code): how much a single disk level under the RAM cache saves when RAM overflows, compared with reading the prompt again, and how much it writes. It may help with the one disk-persistence design to be picked for 0.1.46 (the maintainer's comment on #1090).

   Setup: Qwen3.8-Flash-Next UD-IQ4_XS, 4x RTX 3090, ext4 on 3 NVMe in RAID 0. When the RAM cache evicts a conversation for its size or slot limit, the conversation is written to disk instead of dropped, if it is long enough and the day's write budget is not used up: 16K+ tokens and 200 GB/day in the benchmark runs below and in production on 10-12 (including the soak), 8K+ and 300 GB/day in production since 10-13. Evictions forced by the free-RAM floor are dropped without a write (about 1,100 in the production log). Each conversation is written once and rewritten only after 8K new tokens. Any I/O, checksum or identity mismatch means the prompt is read again.

   • Forced overflow (48 users x 4 turns, 10K-token documents, 6 GiB RAM cache): 119.7 / 116.0 aggregate tok/s re-reading (two runs), 149.6 with the disk level (+25 % over the faster run, +27 % over their mean).
   • Restore of ~620 MB (about 20K tokens; 4 parallel O_DIRECT reads, one per pipeline stage): 104 ms p50, 134 ms p95, plus 39 ms p50 to copy it to the GPUs. Spill: 86 ms p50 on a writer thread, not in the request path. In production (log below, plus this morning's restart), each of 48 disk restores (10.7K-79.5K tokens, 0.49-1.99 GB) took 130-444 ms to read into RAM, GPU copy not included; the largest, 79,535 tokens (1.8 GB), took 444 ms.
   • With enough RAM (nothing evicted), tier on vs off at 16 users: -0.02 % / +0.00 % (short / long prompts), replies identical 32/32. With read-back checks on, all 64 writes and 59 restores matched. Greedy and T=0.7 (seed 1234) outputs stayed byte-identical with 15 disk restores each.
   • Files are keyed to model, binary, flags and env. A restart with the same binary restored 4/4 conversations (10-13); after a restart this morning (10-18 08:22), 6 conversations came back from the files written overnight (8 restores by 08:37). A new build starts with an empty disk level: the 42 sets (52 GB) of our two builds before the current one stay on disk unused until their TTL (168 h after last use).
   • Writes (see [Feature]: ~10x less SSD writing when switching long conversations — a small addition to #1480 #2448): ~380 GB/h in that overload. Our production log, 2026-10-12 18:27 to 10-18 06:00, shows 1,075 spills (969 GB written) against 40 disk restores (40.2 GB read), 0 misses. A one-hour soak test on 10-12 accounts for 221 of the spills (169 GB) and 11 of the restores. The 300 GB/day budget ran out on 10-14; for the rest of that day 439 evictions were dropped and counted (none on other days). These were 0.1.43- and 0.1.44.1-based builds, with a RAM cache of 48-96 GiB until 10-14 06:43 and 128 GiB after; since then: 242 spills (270 GB), 7 restores.

   In our traffic most of what we wrote never came back (40 of 969 GB read), so we would keep a write budget in any design; the 1,294 restores in a week reported in the issue show that this depends on the workload. We have not tried predicting which conversations will come back.

   Not tested: a second, slower level, SATA or single-NVMe disks, HIP/SYCL/Metal, Windows, other models.
4. shahrokhzargarpour

   closed this as not plannedon Oct 18, 2026