Skip to content

fix(bench): let a prefill size follow the model's context, not a fixed 300k - #349

Closed
markyu0401 wants to merge 1 commit into
MiaAI-Lab:mainfrom
markyu0401:fix/prefill-context-ceiling
Closed

markyu0401 wants to merge 1 commit into
MiaAI-Lab:mainfrom
markyu0401:fix/prefill-context-ceiling

Conversation

@markyu0401

Copy link
Copy Markdown
Contributor

The prefill bench's custom-size field rejected what its own hint suggested, and its ceiling ignored the model.

  • The input took digits only, and parseContextSize ran parseInt("200k") → 200, so a typed 200k/128k never became a size.
  • PREFILL_MAX_CONTEXT_SIZE was a fixed 300k, so a model with a larger window could not be asked for more than 300k.

Changes

  • parseContextSize(raw, maxTokens) accepts a k/m suffix in the binary sense the decode dialog already uses (128k = 131072, 1m = 1048576) and validates against the caller's ceiling.
  • prefillCeilingFor(contextLength) takes the ceiling from the model's own context. There is deliberately no fixed upper bound — models already ship windows past 1M, and any constant here is just the next thing to raise. 300k stays only as the fallback when the context is unknown.
  • The dialog uses that ceiling for the custom field and accepts the suffix in the input; the hint now shows 200k as an example.
  • The server keeps its structural guard: with a known context it accepts any structurally valid size and lets the existing fit check emit the skipped row; with an unknown context the 300k bound is the only limit available.
  • formatContextSize reads 204800 back as 200k and 1048576 as 1m.

Not in scope

The decode bench's start-context field: it has no model context at start and already parses 128k. Also worth knowing that timeoutMsForSize still caps at 45 min, which is the practical ceiling for very large sizes on a slow engine.

Tests

Parser/ceiling/normalise cases in server/collectors/__tests__/PrefillBench.test.js, plus three dialog cases (600k accepted on a 1M context, 1.5m accepted on a 2M context, 600k refused with the real ceiling on 128k). npm run typecheck, build, i18n:check, full vitest (808) and server/__tests__ + PrefillBench (69) pass.

Overlap

#323 (i18n of the benchmark workflows) touches PrefillBenchDialog.tsx and is currently unmergeable; a rebase over it is mechanical (the hint/aria strings move into the catalog).

Reported by @de-flandres in #343.

…d 300k

The custom-size field rejected what its own hint suggested: the input took
digits only and the parser ran parseInt("200k") → 200, so a typed "200k" or
"128k" never became a size. The ceiling was also fixed at 300k, so a model
with a larger context could not be asked for more.

Parse a k/m suffix in the binary sense the decode dialog already uses, and
size the ceiling from the model's own context — no constant to raise the next
time a model ships a bigger window. 300k stays only as the fallback when the
context is unknown.

The server keeps its structural guard: with a known context it accepts any
size and lets the existing fit check emit the skipped row; with an unknown
context it keeps the 300k bound, the only limit available without a model.

Not in scope: the decode bench's start-context field, which has no model
context at start; it already parses "128k".

Reported by @de-flandres in MiaAI-Lab#343
@MiaAI-Lab

Copy link
Copy Markdown
Owner

Ported to main as 3bcea09. No constant ceiling of its own is the right call — a model past 1M would just make it the next number to raise; the k/m parse, the explaining skip row and the dialog ceiling came over intact. Full gate green after resolving against the current dialog. CHANGELOG credits you. Closing as completed (closes #343).

(Review notes prepared with Claude Code on behalf of the maintainer.)

@MiaAI-Lab

Copy link
Copy Markdown
Owner

now on main as 3bcea09.

@MiaAI-Lab MiaAI-Lab closed this Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants