Repository navigation
fix(bench): let a prefill size follow the model's context, not a fixed 300k - #349
Closed
markyu0401 wants to merge 1 commit into
Closed
markyu0401 wants to merge 1 commit into
markyu0401 wants to merge 1 commit into
Conversation
…d 300k
The custom-size field rejected what its own hint suggested: the input took
digits only and the parser ran parseInt("200k") → 200, so a typed "200k" or
"128k" never became a size. The ceiling was also fixed at 300k, so a model
with a larger context could not be asked for more.
Parse a k/m suffix in the binary sense the decode dialog already uses, and
size the ceiling from the model's own context — no constant to raise the next
time a model ships a bigger window. 300k stays only as the fallback when the
context is unknown.
The server keeps its structural guard: with a known context it accepts any
size and lets the existing fit check emit the skipped row; with an unknown
context it keeps the 300k bound, the only limit available without a model.
Not in scope: the decode bench's start-context field, which has no model
context at start; it already parses "128k".
Reported by @de-flandres in MiaAI-Lab#343
Owner
|
Ported to main as 3bcea09. No constant ceiling of its own is the right call — a model past 1M would just make it the next number to raise; the k/m parse, the explaining skip row and the dialog ceiling came over intact. Full gate green after resolving against the current dialog. CHANGELOG credits you. Closing as completed (closes #343). (Review notes prepared with Claude Code on behalf of the maintainer.) |
Owner
|
now on main as 3bcea09. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The prefill bench's custom-size field rejected what its own hint suggested, and its ceiling ignored the model.
parseContextSizeranparseInt("200k")→ 200, so a typed200k/128knever became a size.PREFILL_MAX_CONTEXT_SIZEwas a fixed 300k, so a model with a larger window could not be asked for more than 300k.Changes
parseContextSize(raw, maxTokens)accepts ak/msuffix in the binary sense the decode dialog already uses (128k= 131072,1m= 1048576) and validates against the caller's ceiling.prefillCeilingFor(contextLength)takes the ceiling from the model's own context. There is deliberately no fixed upper bound — models already ship windows past 1M, and any constant here is just the next thing to raise. 300k stays only as the fallback when the context is unknown.200kas an example.formatContextSizereads 204800 back as200kand 1048576 as1m.Not in scope
The decode bench's start-context field: it has no model context at start and already parses
128k. Also worth knowing thattimeoutMsForSizestill caps at 45 min, which is the practical ceiling for very large sizes on a slow engine.Tests
Parser/ceiling/normalise cases in
server/collectors/__tests__/PrefillBench.test.js, plus three dialog cases (600k accepted on a 1M context,1.5maccepted on a 2M context, 600k refused with the real ceiling on 128k).npm run typecheck,build,i18n:check, full vitest (808) andserver/__tests__+ PrefillBench (69) pass.Overlap
#323 (i18n of the benchmark workflows) touches
PrefillBenchDialog.tsxand is currently unmergeable; a rebase over it is mechanical (the hint/aria strings move into the catalog).Reported by @de-flandres in #343.