Repository navigation
[stage CI] 251 at 184123b - #26
Draft
LeoBorcherding wants to merge 13 commits into
Draft
LeoBorcherding wants to merge 13 commits into
LeoBorcherding wants to merge 13 commits into
Conversation
Assisted-by: Claude
With --moe-cache-mib auto, --fit sizes the MoE cache when the experts do not fit in device memory: the memory left after the dense weights, the context and the compute buffers goes to the cache, or with --moe-cache-static F a fraction F of it to the experts of whole layers kept in device memory (first layers first). The result is measured again with the cache allocated and the cache shrinks by any overshoot. If the cache would be smaller than n_expert_used + 1 slots per group of cached layers, fit falls back to whole layers without a cache. Only for a MoE model on a single GPU, otherwise no cache is used. llama-fit-params prints the chosen --moe-cache-mib. (cherry picked from commit 0c0cd60)
- roll back a partially applied MoE cache on errors during the fit - reduce the static layers and re-measure if they overshoot - re-measure up to 4 times while shrinking the cache - require a cache of at least twice the minimum, no cache if no layer is in host memory - do not cache the NextN/MTP layers in the main context (cherry picked from commit e5b686c)
… reject a non-numeric --moe-cache-static A context that shares the main model (MTP) runs on its weights wherever the fit places them, so its memory was measured once with its own params and went stale when the fit moved experts to the host. Measure it with the main model params and again whenever their placement changes. std::stof accepted NaN and trailing junk for --moe-cache-static. (cherry picked from commit 82aa472)
ggml-org#29887 and ggml-org#30112 (the MoE cache over multiple GPUs) are now upstream, so the cache files take upstream's version. --moe-cache-mib auto keeps its NextN/MTP layer filter and min_size beside it, as an MTP constructor overload and a separate check, so every upstream line stays as it is.
Drop --moe-cache-static: every measurement found the full cache best or within 5% once experts spill, so the static share and its layer bookkeeping go, along with llama_model_n_bytes_exps. Drop the NextN/MTP layer filter and the MTP constructor overload, which leaves upstream's cache files untouched apart from the minimum size the fitter needs. common_fit_params takes a bool instead of a struct.
dmds_full can be measured at the native context while the fit has already reduced the context, so the difference of the totals counted the KV cache as expert weights and picked the cache where the layer split keeps most experts resident. Also log the last measured cache size on failure.
…PU device The fit writes its expert placement into params, but the cache size it chose stayed in the context params, so a reload from params (the server waking from sleep) kept the experts in host memory without a cache. The minimum cache size is 0 unless the device is a GPU the cache can use, and --moe-cache-mib rejects values with trailing characters.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI staging copy of unslothai#251 at
184123b59exactly (the author's branch head), based on its merge base withbase/upstream-2c192444c,2c192444c. Nothing is fixed here; review and fixes live on the mirror, #12.