[am17an]

Overview

Without mmap, tensors in non-host buffers that can't use the async upload path (RPC, -sm row/-sm tensor split buffers, CPU repack buffers, --check-tensors) are loaded through a staging buffer. Today that buffer is allocated per tensor, so its pages are faulted again for every tensor, and with -lm dio read_raw also bounces each tensor through read_aligned_chunk.

This PR:

• keeps one staging buffer for the whole load, reallocated only when it is too small or more than 2x what the current tensor needs (tensors are already loaded biggest-first)
• with direct IO, reads the aligned range around the tensor straight into an aligned spot in that buffer and uploads from the offset inside it, so there is no bounce buffer

Split out from #29413.

Additional information

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES

[ServeurpersoCom]

Gave it a spin on my RTX PRO 6000 with gemma-4-12b and --no-mmap: with --check-tensors on CUDA the load goes from 6.5s to 2.9s, CPU repack is about twice as fast too, and RPC on localhost is a bit quicker. Same outputs as master, LGTM!