Skip to content

5080deploy: upstream #676 on-demand short prefills (+ two fixes), 1080deploy and chatdnp picks - #86

Open
gdevenyi wants to merge 19 commits into
5080/40-rebuild-9c635a15from
5080/41-r5-additions
Open

5080deploy: upstream #676 on-demand short prefills (+ two fixes), 1080deploy and chatdnp picks#86
gdevenyi wants to merge 19 commits into
5080/40-rebuild-9c635a15from
5080/41-r5-additions

Conversation

@gdevenyi

@gdevenyi gdevenyi commented Oct 13, 2026

Copy link
Copy Markdown
Owner

Stacked on #85. This is the new 5080deploy tip 8eb353b3, deployed 2026-10-13.

FlashML-org#676 (on-demand experts for short prefill chunks), merged as a replacement for the fork's FREETOKEN_MOE_SMALL_PREFILL_TOKENS path:

Two bugs in FlashML-org#676 as posted, both reported on the PR:

  • ensure_experts gets the chunk's full tokens x top_k ids. On main's LRU kernel, 64-token chunks stall and 128 or more fail to compile (numel exceeds triton maximum tensor numel). The fork's unique-id chunking avoids this.
  • 8eb353b3 fix(moe): sgl_kernel's moe_align_block_size makes an illegal memory access once num_experts + 1 > 1024 (0.4.5 and 0.4.9). The prefill over a 2100-slot cache aligns against the slot count, so the server crashed on its first short prefill. The fix routes to the Triton align past the limit, with a regression test.

Deployed with --moe-prefill-demand-tokens 512. TTFT over a cached ~28k prefix:

new tokens in the turn 0 256 512 1024
38-201 1.25-1.32 s 0.42-0.86 0.38-0.86 0.38-0.87
750 / 1229 1.38 / 1.44 s 1.45 / 1.45 1.41 / 1.44 1.44 / 1.51
2009-2568 1.44-1.53 s 1.42-1.53 1.45-1.54 1.45-1.62

Cold greedy fingerprints are identical at every threshold, and decode is unchanged.

From 1080deploy:

From chatdnpdeploy:

  • The HC combine fused with the next RMSNorm (0e847f8c, 8f8d7fbd, 081534bb). Bit-identical (same fingerprints and NLL), with no measurable decode change here.
  • response_format json_object/json_schema are soft-accepted.
  • Image-fetch error classification.
  • Penalty tests.
  • Already present, so skipped: top_k <= 0, the disconnect watch on /v1/messages and /v1/responses, the developer merge, the non-finite repetition penalty check, the n>1 cancel, and the continue-final parse.
  • Dropped: one Pascal-only sampling test.

Tests:

Launcher (run-freetoken.sh): source .venv-t214 and --moe-prefill-demand-tokens 512. The backup is run-freetoken.sh.bak-2026-10-13.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

YevheniiKotyrlo and others added 19 commits October 13, 2026 17:48
@YevheniiKotyrlo
test(moe): a short prefill chunk loads its experts through the slot c…
aae638d 
…ache, bit-identical to the stream
@YevheniiKotyrlo
feat(moe): load a short prefill chunk's experts on demand through the…
689f3d1 
… slot cache
@YevheniiKotyrlo
test(moe): a dense model drops the demand-prefill threshold
96c7310
@YevheniiKotyrlo
test(moe): the demand-prefill threshold is refused off GPU decode
a8f24b1 
OffloadMoeCache raises when --moe-prefill-demand-tokens is set for cpu or hybrid decode, whose experts are not read from the slot cache a demand-loaded chunk fills; the test drives both targets.
@gdevenyi @claude
merge FlashML-org#676: feat(moe): load a short prefill chunk's expert…
8392c72 
…s on demand through the slot cache

Resolved onto the fork's small-prefill path (FREETOKEN_MOE_SMALL_PREFILL_TOKENS), which this
supersedes: the on-demand branch now takes max(--moe-prefill-demand-tokens, the env knob),
keeps the fork's _ensure_unique (unique ids in decode-width chunks, scored-eviction pin,
no decode-state update, counts_as_decode=False) and runs FlashML-org#676's prefill GEMM over the slot
cache (n=cache_size, is_prefill=True) instead of the decode GEMM, so its output matches the
streamed layer. Tests updated for the prefill GEMM and the unique-id ensure.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt
@gdevenyi
fix(kernel): size triton's autotune flush buffer to the device l2
b880e7c 
A kernel autotunes on its first launch of each key, which can be the first request after
the cache planner has handed out nearly all VRAM; triton's flat 256 MB flush buffer was an
OOM there on an 11 GB card. Four times the L2 still evicts it between timed runs.

Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit ffbf001)
(cherry picked from commit e097977)
@gdevenyi
test(utils): drive the async coalesced-pull test with asyncio.run
47afdaf 
Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit 32007ec)
@gdevenyi
fix(engine): keep multimodal placeholder ids out of the repetition-pe…
c0190aa 
…nalty prompt mask

Fork #47's sampler plan scattered the raw prompt ids into a [rows, vocab+1] mask; image
tokens carry pseudo-ids >= 1,000,000, so repetition_penalty with an image indexed out of
bounds -- a device-side assert that takes the server down. They now land in the scratch
column.

Also keeps presence/frequency out of the LogitsPlan: with FlashML-org#527 device counts and
FlashML-org#393 plan both applying them, a request with a penalty plus any plan processor was
penalized twice.

(cherry picked from commit 66053de)

Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit b988390)
@gdevenyi
fix(engine): apply the repetition penalty from device state, not a pe…
3af6e25 
…r-step prompt upload

(up to 262K ids) and a synchronous pageable copy, which stalled the overlap loop and saw
the output one token late. The prompt mask is now built once per request, the output
side reuses FlashML-org#527's device counts, and the order is vLLM's (repetition, then presence and
frequency).

Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit 343d742)
(cherry picked from commit 3df811d)
Siarhei Berdachuk @gdevenyi
fix(server): clarify unsupported response_format error text
1e32ddb 
After soft-accepting json_object/json_schema, the old "no constrained decoding"
message was misleading for remaining unknown types.

(cherry picked from commit 51e17e6)
(cherry picked from commit e2df92f)
@Avicennasis @gdevenyi
fix(mm): classify image fetch errors and re-check redirect targets
9586885 
(cherry picked from commit 8952c05)
(cherry picked from commit 5eeb480)
@gdevenyi
test(engine): cover presence/frequency penalties on the greedy and sa…
cbe217b 
…mpled paths

Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit 30b8181)
(cherry picked from commit dc7cfc1)
Eva (port) @gdevenyi
perf(model): fuse hc combine with the next block's RMSNorm via hc_com…
d8d261b 
…bine_norm

the vendored combine+norm kernel was exported but unwired; mix() takes a
pre-computed rn, the model loop carries the fused pair across the layer
boundary, skipped before PLE layers where ple(hidden) lands between.

(cherry picked from commit 7636dd7)
(cherry picked from commit 0e847f8)
Eva (port) @gdevenyi
fix(kernel): bit-exact rmsnorm reduction tree in hc_combine_norm
7ec5a11 
the fused kernel summed squares over a two-axis padded tile; the standalone
grouped RMSNorm reduces one flat next_pow2 vector - different fp32 tree,
ulp-level drift caught by the greedy gate. Reshape to the identical tree.

(cherry picked from commit c1c5e76)
(cherry picked from commit 8f8d7fb)
Eva (port) @gdevenyi
test(kernel): bitwise pin for hc_combine_norm against the split path
c039107 
(cherry picked from commit 7cba55e)
(cherry picked from commit 081534b)
Siarhei Berdachuk @gdevenyi
fix(server): soft-accept OpenAI response_format json_object/json_schema
0aa00e4 
Rejecting response_format with "no constrained decoding" breaks OpenAI-compatible
clients that rely on json_object/json_schema even when the model can emit valid
JSON from a prompt. Accept those wire types and prepend a best-effort system hint;
schema is not token-level enforced. Unknown response_format types still return 400.

(cherry picked from commit ca7d704)
(cherry picked from commit db64d8e)
Siarhei Berdachuk @gdevenyi
refactor(server): drop response_format system-hint injection
97758b6 
Soft-accepting json_object/json_schema is enough when clients already
request JSON in the prompt. The injected schema wrapper was redundant
and could distort the output shape.

(cherry picked from commit 65907c8)
(cherry picked from commit c82625f)
@gdevenyi @claude
test(engine): drop the pre-volta sampling penalty test, whose torch f…
941fe3b 
…allback this branch does not carry

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt
@gdevenyi @claude
fix(moe): align on triton past sgl_kernel's 1024-expert limit
8eb353b 
sgl_kernel's moe_align_block_size (0.4.5 and 0.4.9) reads out of bounds once num_experts + 1
exceeds 1024: an illegal memory access that poisons the CUDA context. A prefill over the
offload slot cache (FlashML-org#676) aligns against the slot count, 2100 here, so
the first short prefill crashed the server; the bf16 and NVFP4 prefill paths both go through
it. The Triton align has no such limit and matches sgl_kernel's output below it.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt
@gdevenyi

gdevenyi commented Oct 13, 2026

Copy link
Copy Markdown
Owner Author

Rollback to 1639600b (torch 2.11). The main checkout's extensions are now built against torch 2.14, so a rollback has to rebuild them:

  1. git checkout -B 5080deploy 5080deploy-pre-r5
  2. .venv/bin/python setup.py build_ext --inplace
  3. cp ~/models/run-freetoken.sh.bak-2026-10-13 ~/models/run-freetoken.sh
  4. systemctl --user stop freetoken; ~/models/start-freetoken.sh

@gdevenyi

gdevenyi commented Oct 13, 2026

Copy link
Copy Markdown
Owner Author

Side calls during a long conversation (255K prompt; one side call lands mid-prefill and three during decode; 512-token decode):

decode side calls after the prefill
threshold 0, 2 reps 53.2 / 52.9 tok/s 1.63-1.69 s
threshold 512, no pre-warm, 2 reps 41.6 / 53.0 tok/s 0.86-1.31 s
threshold 512, pre-warmed, 2 reps 52.3 / 52.6 tok/s 0.87-0.98 s
  • In a fresh server process, the first side call that takes the on-demand path autotunes its kernels and stalls the main decode for ~4 s. The serve log shows one 4 s gap at 11.7 tok/s, and that rep drops to 41.6 tok/s.
  • start-freetoken.sh now sends five short prompts (1-300 words) after the cache rebuild. With them, no stall and no measurable decode cost.
  • Misses per token run ~9% higher after side calls with on-demand on, but that does not show up in decode.

@gdevenyi gdevenyi mentioned this pull request Oct 15, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@gdevenyi @YevheniiKotyrlo @Avicennasis