[j-luwierski]

Problem and result

On current main (0.2.1, 9076725), a profile-filled IQ3_XXS native pack with --prefill auto and --no-token-graph could not borrow the expert cache. The residency table was only built for the token-graph hit path, so the prompt path tried to allocate its own 8192-token buffers and failed with 23 MiB free while requesting an 80 MiB buffer.

The fix builds the residency table whenever prefill borrowing is available, independently of token-graph capture. On the same RTX 4070 Ti SUPER, the updated build borrowed 2489 cache slots, refilled them after the prompt, processed 12,287 tokens, and generated one token. A token-graph control also completed and produced the same token.

Related discussion

• Original report and maintainer scope agreement: generate: the prompt loan keeps its residency bookkeeping without the token graph (#796 part B) #1131.
• Parent context: generate: one startup VRAM plan before the expert cache is committed (#765) #796.

Approach

The residency decision covers borrowing availability and a positive prefill chunk. The table is staged for either reader: the token-graph hit path or the prompt loan. Graph hit wiring remains gated by graph capture. A borrowing-only staging failure warns and disables borrowing for that run; graph staging failures retain their existing fatal behavior. Loan/refill trace lines report the slot range and any slots still lent.

Validation

Tested revision: 70660fa on 9076725. Environment: Linux, NVIDIA RTX 4070 Ti SUPER 16 GB, CUDA 13.4.92, CUDA arch 89, Ryzen 9 5900X, 125 GiB RAM. Model: Qwen3.8-Flash-Next IQ3_XXS native pack, int8 KV, 32K context, MTP spec 4. The prompt file contained 12,288 copies of token ID 198.

Passed:

• cmake --build build-repro --target strata -j 12 — CUDA Release build passed.
• cmake --build build-repro --target borrow_residency_test -j 12 and ctest --test-dir build-repro -R '^borrow_residency_test$' --output-on-failure — passed.
• Baseline --prefill auto --no-token-graph reproduction on 9076725: exited 1 at prefill initialization with 23 MiB free while the 8192-token chunk requested an 80 MiB buffer.
• Same run on 70660fa: completed the 12,287-token prompt and generated token 198; the log reported an 8192-token chunk, 2489 borrowed cache slots, and all 2489 slots refilled.
• Token-graph control on 70660fa: completed the same prompt and generated token 198.
• git diff --check — passed.

Runtime prompt setup: python3 -c 'from pathlib import Path; Path("/tmp/strata-pr1131-prompt.txt").write_text("198 " * 12288)'. The engine command used --pack $DATA/packs/iq3_xxs, the IQ3_XXS native GGUF shard pair, --expert-profile data/expert-profile.bin, --expert-cache auto, --prefill auto, --spec 4, --mtp $DATA/mtp/rt, --max-context 32768, --max-new 1, --kv int8, --no-token-graph, and --tokens-file /tmp/strata-pr1131-prompt.txt.

Failed:

• Baseline reproduction on 9076725 failed as described above; no failure was observed on the tested final revision for the supported run.

Not run:

• HIP and SYCL builds: hipcc, icpx and dpcpp are unavailable, so this PR is a draft.
• --no-capture: current main rejects this native-pack configuration because session_token has no CPU expert-pool hook; the suggested --no-pool combination was not tested.
• Multi-GPU, two-request serve run, and full CTest suite.

Limits

The hardware run covers one CUDA GPU and one IQ3_XXS pack. It makes no performance claim and does not establish HIP, SYCL, multi-GPU, or repeated-serve behavior.