Repository navigation
[stage CI] 345 at 153050e - #17
Closed
LeoBorcherding wants to merge 2 commits into
Closed
LeoBorcherding wants to merge 2 commits into
LeoBorcherding wants to merge 2 commits into
Conversation
…t's routing With experts in host memory and the MoE cache, the last large ubatch of each llama_decode call seeds the cache. When that ubatch had no earlier routing in the prompt (the first server chunk, or a whole prompt in one chunk) it read its own ids first, so its expert uploads ran on the compute stream after a per-layer readback instead of ahead on the prefetch copy queue. Keep the routing counts across prompts. A long seeding ubatch with no routing of its own now copies all experts ahead like any other large ubatch and seeds from the last prompt's routing. Later chunks of the same prompt still seed from the prompt's own routing. A small ubatch with outputs (the server's last prompt tokens) also reads the routing of the large ubatch before it, so the fallback follows the last prompt. Short chunks keep the routed-only copy: they leave many experts unrouted and have little compute to hide the copies under.
…cache ggml_backend_cuda_copy_regions_async returned false on HIP, so the prefetch copy of a large ubatch could only take long runs of cached experts from the MoE cache and uploaded the rest from host memory again. With the cache seeded from the last prompt that was every expert of every layer: 23.5 GB per 4.58k-token prompt on an R9700 at ub 8192, 16 GB of it already in VRAM. On HIP the device copies now go through the copy-regions kernel and the uploads are one hipMemcpyAsync each (there is no batched copy API), so a cached expert is copied from the cache and only the misses are uploaded.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI staging copy of unslothai#345 at
153050ec5exactly (the author's branch head), based on its merge base withmoe-offload-prefetch-cache,fd7489efb. Nothing is fixed here; review and fixes live on the mirror, #9.