core: elastic expert cache via CUDA VMM and on-demand GPU vision lifecycle

[code-martin]

Summary

This PR implements elastic resizing for the MoE expert cache using the CUDA Virtual Memory Management (VMM) API and introduces an on-demand lifecycle for GPU Vision models.

Key Changes

• CUDA VMM Elastic Expert Cache:
  • Replaces fixed monolithic cudaMalloc allocations with cuMemAddressReserve, cuMemCreate, and cuMemMap / cuMemSetAccess.
  • Supports dynamic online shrink (RESIZE) and expansion (GROW) without reallocating virtual addresses or invalidating captured CUDA graphs.
  • Automatically falls back to standard cudaMalloc if VMM is unsupported on older architectures.
• On-Demand GPU Vision Lifecycle:
  • Vision encoder is dynamically loaded into VRAM only when image embeddings are required.
  • Temporarily shrinks the expert cache to release VRAM, performs vision encoding, offloads the vision model, and instantly grows the expert cache back to full capacity.
  • Reclaims over 600 MoE slots (~830 MB) of VRAM permanently during text-only generation.
• Client Validation Tool:
  • Adds ools/test_vision_client.py for end-to-end verification of image encoding, VRAM contraction, and cache restoration.

Testing

• Verified on Windows with MSVC C++20 and CUDA 12.6 (RTX 3080 10GB).
• Text-only generation runs with 2,459 expert slots (up from 1,840). Vision requests encode in ~7.5s and immediately restore the full 2,459 cache slots.

[Niko1221]

Thanks for the work and the write-ups. I went through all three carefully, and I'm not merging them as they are - here is why, so the useful parts can come back in a shape that fits:

#37 (VMM cache + on-demand vision):

• The cache can only shrink and grow back to its startup size, and the slot gain in the description comes from loading vision after the engine rather than from VMM.
• RESIZE/GROW copy the cache's startup residency table over the live one (xcache.residency_table()), while adaptation (on by default) has already swapped slots - after a resize those slots are read as the wrong expert.
• Since 0.1.13 the prompt path borrows the top of the cache for its buffers; shrinking unmaps exactly that region, and a partial GROW leaves those pointers on unmapped memory.
• The 5.4 GB / 10 GB targets are hard-coded, and every image pays a full encoder load (~7.5 s).

#38 (adaptive eviction): it mostly retunes the adaptation the engine already has (0.7 was already exponential decay), the new ADAPT command blocks after every request, and it reads usage even when adaptation is off (--adapt-swaps 0), which crashes. A per-swap stderr line floods the log.

#39 (telemetry): it changes adapt_every 4 -> 1 and adapt_swaps 96 -> 1 without saying so, checkpoints at every turn boundary (for a 40-turn agent prompt that is ~40 borrow/refill cycles and ~40 x 118 MB saves, which undoes much of 0.1.13's prompt speedup), and /api/stats skips the API-key check.

What I'd gladly take as small separate PRs against current main: the per-request decode hit rate (the DONE fields, the Monitor column), keeping the root checkpoint (only the first turn boundary), the Vision.close() temp-dir cleanup, and flags for the adaptation thresholds - each with a before/after measurement. Also please drop tools/test_vision_client.py and the CMakeLists BOM change from them.

[code-martin]

Thank you for the detailed and constructive review, Niko! You're completely right on the architectural conflicts (especially regarding the 0.1.13 borrow path, the prefill overhead with per-turn checkpoints, and the existing adaptation engine). I will close these consolidated PRs and prepare small, clean, isolated PRs addressing exactly the parts you outlined against current main.