[As7656431]

Problem and result

On a Windows PC whose RAM cannot hold the experts, the low-RAM mode (--mmap-experts, no --resident-budget-gib) reads each prompt chunk's experts by copying them out of the mapped experts.bin. Those copies are page faults. On this PC (RTX 5070 12 GB, 31.8 GiB RAM, a 35.5 GB IQ2_XS experts.bin on a drive of about 1.5 GB/s) the drive read only 0.2-0.5 GB/s during a long prompt, and the faulted pages push the decode's cached experts out of RAM. STRATA_PROMPT_DIRECT=1 already reads the prompt path's blobs unbuffered, but only on Linux and only with a RAM budget.

This PR, opt-in and Windows only:

• STRATA_PROMPT_DIRECT=1 also works on Windows and in the low-RAM mode without a budget: the stager's blobs are read with unbuffered, overlapped reads from a second handle on experts.bin, while the decode keeps the mapping.
• STRATA_PREFILL_DIRECT_RAM (on that path only): a blob whose pages are already in the working set is copied from the mapping; =2 reads the rest through a cached handle (what the standby list still holds), =3 as 2 with neighbouring blobs of a layer merged into one read.

With both unset nothing changes. On 0.2.3.1, same PC and settings, tools/strata_bench.py, medians of 3:

	main (7ddc17a)	this PR, STRATA_PROMPT_DIRECT=1 STRATA_PREFILL_DIRECT_RAM=3
4K prompt	87.0 tok/s, first token 46.3 s	414.6 tok/s, first token 9.7 s
16K prompt	101.3 tok/s, first token 158.7 s	846.6 tok/s, first token 19.0 s
agent follow-up turn, first token	62.8 s	7.2 s
agent decode	5.7 tok/s	48.7 tok/s
short-chat decode	55.8 tok/s	55.9 tok/s

Related discussion

#3065 has the problem, the measurements and the reproduction steps. There is no maintainer reply on scope yet; both switches are opt-in and no default changes. This PR was closed in the 0.2.0-beta reset; it is now rebased on main (7ddc17a, v0.2.3.1) and re-tested there. It is independent of #3086; the two merge cleanly together.

Approach

• open_prompt_direct gets a Windows branch: each expert file is opened again with FILE_FLAG_NO_BUFFERING | FILE_FLAG_OVERLAPPED (experts.bin only, and not when the file tier is already unbuffered). With STRATA_PREFILL_DIRECT_RAM >= 2 it also opens a cached overlapped handle.
• read_direct uses these handles when the tier itself is not unbuffered. copy_blobs already reads through pdirect_, as it does on Linux.
• ExpertSource::prompt_reads() (default false; true on Windows for experts.bin with those handles open): the stager in prefill.cpp then hands the unpinned blobs to copy_blobs, 8 at a time. transient() is not touched, so the decode and the prompt path's CPU share behave as before; only where the bytes come from changes.
• copy_blobs with STRATA_PREFILL_DIRECT_RAM (Windows, prompt_reads() only) asks QueryWorkingSetEx which blobs are already in RAM.
• generate.cpp reads the switch in the mmap mode without a budget on Windows only, so Linux keeps its behaviour also with the switch set.
• An earlier approach to the same faults, prefill: release copied Windows expert pages after host staging #1520, releases the copied pages after the copy; this PR avoids the faults themselves.
• New test test_prompt_direct_mapped in tests/core/file_expert_source_test.cpp: a synthetic experts.bin, open_prompt_direct, then copy_blobs of 8 blobs, read unbuffered, copied from the working set, and read through the file cache in merged runs, compared byte for byte with the file.

Validation

Passed:

• Build, Windows 10 22H2, MSVC 2022, CUDA 13.0, -DCMAKE_CUDA_ARCHITECTURES=120 -DSTRATA_PORTABLE=ON: cmake --build build --target strata on 857b57b (this branch before the test commit; that commit touches only tests/), and cmake --build build --target file_expert_source_test with -DSTRATA_BUILD_TESTS=ON on 237d6cb (the head). No new kind of compiler warning; a few more C4996 (getenv), as in the code around it.
• file_expert_source_test on 237d6cb: file_expert_source_test: PASS. The new test prints prompt direct (unbuffered): 7 of 8 blobs read from the file: bytes PASS and prompt direct (file cache, merged): 7 of 8 blobs read from the file: bytes PASS (the eighth blob was copied from the working set). The test also passes on main 7ddc17a, and on this branch merged with [Engine] Windows low-RAM mode: batch-prefetch a layer's CPU experts for a mapped experts.bin (STRATA_MMAP_PREFETCH, #3065) #3086.
• Same computation. STRATA_DUMP_FIRST_LOGITS after an 8K document and a 300-token prompt, two runs per build and setting, in one round with main: for each prompt there were two outputs in all, the first choice was the same in every run, and main itself gave both outputs for the 300-token prompt (KL 1.9e-4 between them). This branch gave each output once with the switches unset and once with them set, for both prompts. On the 8K document main's two runs gave the first output; the second differs from it by KL 1.24e-3, which is what the 0.1.45 release showed between two of its own runs of this document.
• Speed, the table above: tools/strata_bench.py of 7ddc17a as the client, against serve/server.py of the same commit started separately (--url, and PYTHONUTF8=1 for tools/strata_bench.py: UnicodeDecodeError reading CMakeLists.txt on non-UTF-8 Windows (GBK/cp932), before any measurement #2792), on the same day with the same settings; only the engine and the two variables differ. The main run is the report in [Benchmark] RTX 5070 12 GB + 32 GB RAM on Windows 10, Flash-Next IQ2_XS low-RAM mode (0.2.3.1) #3063.
• With the switches set the start-up log says the prompt path's expert reads go unbuffered (STRATA_PROMPT_DIRECT=1, mmap mode).

Failed: none observed.

Not run:

• HIP, SYCL and Metal builds: there is no toolchain for them on this PC. The new code is host-side Windows file I/O (#if defined(_WIN32)) in files that every backend compiles, so the HIP build on Windows compiles it too. I would be glad if someone with a HIP or SYCL setup could build this branch; until then I am leaving it as a draft.
• Linux: the Linux code path is unchanged, and the new test also runs there (with O_DIRECT, skipping without it), but I have not built or run it on Linux.
• A fast NVMe drive (bench/results: RTX 5070 Ti (16 GiB) with 31.6 GiB of RAM — IQ2_XS at 128K in low-RAM mode #2791 suggests the default path is much faster there), a RAM budget together with STRATA_PROMPT_DIRECT=1 on Windows (that combination now opens the handles; before, it was refused as Linux only), the GGUF read in place, other models, contexts above 32K.

Limits

• The gain needs Windows, experts that do not fit in RAM, the low-RAM mode and a drive that is not very fast. Elsewhere the switches change little or nothing.
• The tool's prompts are generated word lists. With a real 20K-token English document the gain was smaller ([Engine] Windows low-RAM mode (--mmap-experts): prompt reads are page-fault bound - opt-in changes in #3072 and #3086 #3065, on 0.1.44.1 with all switches: 162.5 s down to 33.1 s).
• NTFS serves unbuffered reads of a mapped file one at a time (the note in read_direct); on this drive they still run near its speed.

[As7656431]

I added a second commit, STRATA_PREFILL_DIRECT_RAM (the second change described in #3065). It only works on top of the first one, so it belongs in this PR rather than a separate one, and it has its own switch, so it can be judged or dropped on its own.

With STRATA_PROMPT_DIRECT=1 on Windows every unpinned blob of a prompt chunk is read from the drive, also the ones the decode has just read and still holds mapped in. With STRATA_PREFILL_DIRECT_RAM (Windows, only on that path):

• >= 1: a blob whose pages are all in the process working set (QueryWorkingSetEx) is copied from the mapping, with no fault and no drive read; the rest are read unbuffered as before
• 2: the rest through a cached overlapped handle on experts.bin, which serves what the standby list still holds
• 3: as 2, with a run of neighbouring blobs of a layer read as one request

Unset or 0, nothing changes. src/core/expert_source.cpp +113 lines, expert_source.hpp +1.

Same computation. STRATA_DUMP_FIRST_LOGITS after an 8K document and a 300-token prompt, two runs each, against the 0.1.45 release: with both switches unset, and with STRATA_PROMPT_DIRECT=1 STRATA_PREFILL_DIRECT_RAM=3, three of the four 8K runs were bit-identical to a run of the release; the fourth differed from the two release runs by KL 2.4e-3 and 1.2e-3 (the release's two runs of this document differed by 2.4e-3 in #3065 and 1.25e-3 this time). Of the four 300-token runs, two matched the release; the other two gave one other output, KL 9.7e-5 from it (#3065: the release itself gave three outputs at this level in seven runs). The first choice was the same in every run.

Speed, same PC and settings as before, tools/strata_bench.py, medians of 3:

	0.1.45 (#3063)	first commit, STRATA_PROMPT_DIRECT=1	both commits, + STRATA_PREFILL_DIRECT_RAM=3
4K prompt	58.3 tok/s, first token 69.0 s	160.6 tok/s, 25.0 s	437.1 tok/s, 9.2 s
16K prompt	127.5 tok/s, first token 126.0 s	310.8 tok/s, 51.7 s	863.6 tok/s, 18.6 s
agent follow-up turn, first token	48.2 s	25.0 s	7.2 s
agent decode	6.4 tok/s	51.8 tok/s	36.1 tok/s

The agent decode median is lower with the second commit than with the first alone; both vary widely across the 15 turns (15.5-53.4 and 11.3-67.2 tok/s), so I would not read much into it. With these two and #3086 together, the agent decode was 66.1 tok/s (the measurement in #3065). These runs were all on Windows with CUDA only.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

[As7656431]

/reopen

[github-actions]

Reopened, thank you. A maintainer will pick it up from here. If it conflicts with main, rebase it first; that speeds up review.