[am17an]

Overview

Improves loading time for me by 5x on 2 sparks.

DeepSeek-V4-Flash (145.6 GiB), 2x DGX Spark, both devices over RPC (RDMA), cold page cache, load_tensors time:

Config	master (s)	PR (s)	Speedup
-sm layer -lm dio	450	57	7.9x
-sm tensor -lm dio	1484	166	8.9x
-sm layer, default (mmap)	495	326	1.5x
-sm tensor, default (mmap)	1589	487	3.3x

Changes:

• Skip hashing when the server has no cache: the hello response's padding byte becomes a server-flags byte (RPC_SERVER_FLAG_CACHE, set by rpc-server -c). The client hashes tensors for SET_TENSOR_HASH only when the server has a cache.
• No copy on the client: SET_TENSOR sends the tensor data straight from the caller's pointer after the header, instead of first copying it into a zero-filled message buffer.
• Chunked receive on the server: SET_TENSOR data is read in chunks through a reused 16 MiB buffer, instead of a new full-size vector for every tensor. The -c cache-save path still reads the whole tensor so it can hash it.
• RDMA: rdma_recv can read part of a receive slot, so receive sizes no longer have to match send sizes. The chunked receive above needs this.
• RPC protocol 9.0.0: a major bump, because the hello response format changed. Older clients and servers are rejected instead of silently skipping the cache.

This also lowers peak memory during loading: the client keeps one fewer copy of each tensor, and the server's receive buffer is 16 MiB instead of the size of the tensor. Uploads that go through SET_TENSOR_2D (-sm tensor split weights) are not changed.

Additional information

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES

[ggerganov]

Generally, it would help reviewing changes if there is some detailed description in the PR about what is the thought process and what generally the changes achieve.

For example, I've looked at this PR for ~1h and now I understand most of the things, but it definitely took me a while to get there. The dispatcher now has flags with "remote capabilities" called "sever_flags". It also has a new call that allows to do sync send with extra buffer (called "payload") that is not copied into the rpc message structure. It is used for the SET_TENSOR command and it is an exception to how all other RPC messages are handled. There is also some buffer chunking that allegedly speeds up host -> device transfers.

The changes in transport.cpp are not clear to me yet - I don't know what they do and why they are needed.

[am17an]

Yeah agreed. The transport changes are for receiving partial messages. The model loader changes are for avoiding a bounce buffer when async uploads are not possible.

[mahounairo]

Beyond making uploads a bit faster, this also significantly reduces peak memory usage with large models. Uploading Qwen3.8-Flash-Next's 27GB per_layer_token_embd on my setup couldn't be done due to hitting OOM.

With my own, shallower implementation:

4GiB tensor over ggml-rpc-server, loopback:
MEM SERVER: 8196 -> 4357MiB
MEM CLIENT: 8197 -> 4357MiB
UPLOAD: 7.9-8.1s -> 6.0-6.2s

[am17an]

@ggerganov I've added a summary of changes to this PR. I think it's a big improvement over how we're currently doing RPC and it will make non-cached loads (the default) very usable (apart from lowering peak mem usage)

[ggerganov]

I think it would be worth trying to do a from-scratch implementation of ggml-rpc.cpp with an LLM (guided with the existing implementation). Some areas worth improving:

• message serialization
• improve type safety
• better logs and tests

Think such a refactor/reimplementation can improve the development flow in general of this sub-system.