[As7656431]

Problem and result

In the low-RAM mode with a pack's experts.bin read through the mapping (--mmap-experts, no RAM budget), FileExpertSource::prefetch returns at once: staged() is false (no unbuffered tier, and the io prefetch is Linux only). So each expert the decode's CPU pool computes is faulted in page by page. With STRATA_MMAP_PREFETCH the layer's CPU experts go to the OS as one batch (PrefetchVirtualMemory on Windows, madvise(MADV_WILLNEED) elsewhere) before the pool reads them; =2 issues the batch on a helper thread. With the variable unset nothing changes.

It does on Windows what 0.2.1 turned on by default for Linux experts.bin packs with STRATA_IO_PREFETCH (decode x1.21-1.38 measured on an RTX 3060 with 32 GB); that path is Linux only (set_io_prefetch is under #if defined(__linux__)).

On 0.2.3.1, same PC as #3063 with its daily settings (those of #3063 plus the optional control vector that the report leaves out, and STRATA_WATCHDOG_S=180), this branch with the variable unset against STRATA_MMAP_PREFETCH=2, four interleaved rounds (each setting first twice), a restart before each run: a short question, another, a 20K-token English document, a short question:

	unset	STRATA_MMAP_PREFETCH=2	rounds faster
first request after a start	32.2 tok/s	43.4 tok/s (+35%)	4 of 4
warm short chat	60.2 tok/s	67.9 tok/s (+13%)	4 of 4
reading the 20K document	186.1 s	184.5 s (unchanged)	1 of 4
decode right after the document	14.7 tok/s	23.9 tok/s (+62%)	4 of 4
a short question after the document	34.0 tok/s	46.2 tok/s (+36%)	4 of 4

Averages of the four rounds; decode speeds are the engine's own timings. The document hides a code word, and all 8 answers found it.

Related discussion

#3065 has the problem, the measurements and the reproduction steps. There is no maintainer reply on scope yet; the switch is opt-in and no default changes. This PR was closed in the 0.2.0-beta reset; it is now rebased on main (7ddc17a, v0.2.3.1) and re-tested there. It is independent of #3072; the two merge cleanly together.

Approach

• prefetch(): when the source is not staged and the variable is set, the layer's experts go to prefetch_mapped (on the calling thread) or prefetch_async (=2). Otherwise prefetch() is unchanged.
• prefetch_mapped: one PrefetchVirtualMemory call for the layer's blobs on Windows, madvise(MADV_WILLNEED) per blob elsewhere. Experts in the RAM copy are skipped. It only asks for pages ahead of time; what blob returns is unchanged.
• prefetch_async: one helper thread; the newest request replaces one that has not started, so the decode never waits for it (PrefetchVirtualMemory costs about 0.2 ms a call even when every page is resident). The destructor stops the thread before the mapping is closed.
• The variable is read once, when the source is constructed.
• New test test_mmap_prefetch in tests/core/file_expert_source_test.cpp: a synthetic experts.bin read with =1 and =2, every blob compared byte for byte with the file, and a request left queued when the source goes away.

Validation

Passed:

• Build, Windows 10 22H2, MSVC 2022, CUDA 13.0, -DCMAKE_CUDA_ARCHITECTURES=120 -DSTRATA_PORTABLE=ON: cmake --build build --target strata on 1087731 (this branch before the test commit; that commit touches only tests/), and cmake --build build --target file_expert_source_test with -DSTRATA_BUILD_TESTS=ON on 4eefa15 (the head). No new kind of compiler warning (one more C4996 getenv).
• file_expert_source_test on 4eefa15: file_expert_source_test: PASS, with mmap prefetch (STRATA_MMAP_PREFETCH=1): bytes PASS and mmap prefetch (STRATA_MMAP_PREFETCH=2): bytes PASS. The test also passes on main 7ddc17a, and on this branch merged with [Engine] Windows low-RAM mode: STRATA_PROMPT_DIRECT=1 for a mapped experts.bin without a RAM budget (#3065) #3072.
• Same computation. STRATA_DUMP_FIRST_LOGITS after an 8K document and a 300-token prompt, two runs per setting, in one round with main: on the 8K document all four runs of this branch (unset twice, set twice) were bit-identical to both of main's runs. On the 300-token prompt main's two runs gave two outputs (KL 1.9e-4 between them); every run of this branch was bit-identical to one of them. The first choice was the same in every run.
• Speed, the table above: my own bench script, the one behind the 20K-document measurements in [Engine] Windows low-RAM mode (--mmap-experts): prompt reads are page-fault bound - opt-in changes in #3072 and #3086 #3065. It restarts serve/server.py of 7ddc17a for each run and reads the engine's own per-request timings (STRATA_DECODE_TIMING=1). The same day, tools/strata_bench.py of 7ddc17a against main (the run in [Benchmark] RTX 5070 12 GB + 32 GB RAM on Windows 10, Flash-Next IQ2_XS low-RAM mode (0.2.3.1) #3063), medians of 3: short-chat decode 55.8 to 75.2 tok/s (the first run right after the warm-up 14.6-21.0 to 21.9-31.0), 4K prompt 87.0 to 112.6 tok/s, 16K prompt 101.3 to 94.5 tok/s (ranges 66.7-105.3 and 90.2-128.2 overlap), agent follow-up turn first token 62.8 to 47.1 s, agent decode 5.7 to 5.1 tok/s (ranges 0.7-14.4 and 0.7-24.6).

Failed: none observed.

Not run:

• HIP, SYCL and Metal builds: there is no toolchain for them on this PC. The change is host code in expert_source.cpp/hpp, which every backend compiles. I would be glad if someone with a HIP or SYCL setup could build this branch; until then I am leaving it as a draft.
• The non-Windows branch (madvise): not built or run on Linux or macOS. On Linux the default io prefetch of 0.2.1 probably covers this case better.
• A RAM budget whose file tier stays on the file cache (the RAM copy's experts are skipped, the rest are prefetched; not measured), other models. The GGUF read in place and an unbuffered tier are staged, so this PR does not change them.

Limits

• The gain comes when the mapped pages are not in RAM: after a start, and after a long prompt has pushed the cached experts out. With [Engine] Windows low-RAM mode: STRATA_PROMPT_DIRECT=1 for a mapped experts.bin without a RAM budget (#3065) #3072 the second case mostly goes away, so the two together gain less than the sum.
• =1 (on the calling thread) was not measured here; =2 is the setting I measured.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

[As7656431]

/reopen

[github-actions]

Reopened, thank you. A maintainer will pick it up from here. If it conflicts with main, rebase it first; that speeds up review.