Repository navigation
Conversation
Adds the engine half of OpenAI logprobs support. SamplingParams gains logprobs/top_logprobs; the sampler computes log_softmax over the PRE-temperature logits (raw model distribution, so temperature/top-k/top-p do not change reported values), gathers the chosen token and batch-max top-k, and ships CPU copies covered by the existing copy_done_event. The scheduler attaches per-request values (cut to each request's own top_logprobs) to DetokenizeMsg; the detokenizer builds a neutral entry (token text via single-id decode + UTF-8 bytes, so clients can reassemble partial-UTF-8 pieces) onto UserReply.logprobs. Zero cost when off: no row asked -> no mask tensor, no log_softmax, ForwardOutput carries None. Message fields default to None, so old and new peers interoperate. Stop-string trimming can hide final visible text; entries still cover every sampled token.
API half of logprobs support. Chat: logprobs + top_logprobs (0..20) yield
choice.logprobs.content entries {token, logprob, bytes, top_logprobs[]},
streaming and non-streaming. Completions: the legacy integer field (0..5) yields
{tokens, token_logprobs, top_logprobs, text_offset} with absolute offsets across
the stream; echo+logprobs stays rejected (prompt logprobs need prefill logits
and are out of scope here).
The protocol-neutral event layer carries entries on ContentDelta (a list -- parser
buffering can release several tokens' text in one delta); entries always
accumulate on GenResult for the non-streaming path, and reasoning/tool-call
buffering carries pending entries onto the next content delta. Formatting lives
in server/logprobs.py; wire compatibility follows the OpenAI shapes.
Review finding on FlashML-org#224 (HaileyStorm): streaming chat carried logprob entries of hidden reasoning tokens onto the next visible content delta -- pending entries accumulated before the reasoning parser classified the text, and _content_delta() drained the whole list into whatever visible chunk came next. That broke token<->content alignment and leaked hidden reasoning token strings through logprobs entries; non-streaming meanwhile kept every sampled entry and end-of-stream dropped undrained ones -- three behaviors for one surface. Close it at both layers: - /v1/chat/completions rejects logprobs=true with a 400 when the server runs a reasoning parser or the request enables tool parsing: the semantic layer can hide or reclassify tokens, so entries cannot be aligned 1:1 with visible content tokens. tool_choice="none" keeps logprobs available; the raw /v1/completions path (no semantic layer) is unchanged. - _generate_events_impl now collects entries only on the passthrough path (no reasoning parser, no tool parsing), so a hidden-token entry can never ride a later visible delta even for a caller that skips API validation. Replaces test_reasoning_logprob_is_carried_to_next_content_delta (which enshrined the carry) with fail-close coverage for both conflict cases, the tool_choice="none" pass-through, and a no-carry guard test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The logprobs change moved the drain from positional unpacking to named ForwardOutput attributes (next_tokens_cpu, copy_done_event, the *_cpu logprob columns), which the bare-tuple fake in the in-flight-abort scheduler tests predates: three of them died with 'tuple' object has no attribute 'next_tokens_cpu' on a GPU run. Caught by gdevenyi on the PR thread; a macOS run cannot see it (the file already fails on the missing kernel packages there, identically on both sides of the series). A namespace mirroring the drain-facing fields keeps the test off the engine import chain, whose kernel packages a scheduler unit test should not require. Assisted-by: Claude
Two conflicts: - python/freetoken/server/openai_api.py: upstream threads the configurable default output budget (default_max_tokens, FlashML-org#411) into the resolve_sampling call that this branch had extracted into a variable to attach the logprobs fields. Kept the variable; its construction now passes default_max_tokens. - python/freetoken/tokenizer/server.py: import conflict between upstream's get_mm_processor (image input, FlashML-org#454) and this branch's build_logprobs_entry. Kept both. Assisted-by: Claude
…hat and legacy completions # Conflicts: # python/freetoken/core.py # python/freetoken/engine/engine.py # python/freetoken/engine/sample.py # python/freetoken/message/frontend.py # python/freetoken/message/tokenizer.py # python/freetoken/scheduler/scheduler.py # python/freetoken/server/generation.py # python/freetoken/server/openai_api.py # python/freetoken/tokenizer/detokenize.py # python/freetoken/tokenizer/server.py
…lag the sampler plan reads Fork #47's sampler plan asks each request whether it needs logits processing; the upstream FlashML-org#224 test fakes predate it. Assisted-by: Claude Opus 5.5
This was referenced Sep 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #62. Merges upstream FlashML-org#224 (head
9a48a4b): OpenAI-compatiblelogprobs-- raw (pre-temperature) sampled-token logprobs plus top alternatives. Before it,/v1/completionsrejectedlogprobsand chat returned"logprobs": null.What this deployment gets:
/v1/completionswithlogprobs: 0..5, streaming and not, withn > 1. Chat logprobs are refused on this server by the PR's design: it fails closed with a 400 whenever a reasoning parser is active (hidden reasoning tokens cannot be aligned with visible content), and production always runs theqwen3reasoning parser. The error message points clients to/v1/completions.Merge: conflicts in 10 files against #47 (sampler logits-processor plan,
n > 1fan-out, echo/suffix,logit_bias, detokenizer reasoning-token counts), upstream FlashML-org#471 (greedy mask) and FlashML-org#504. Resolutions to review:engine/sample.py:BatchSamplingArgscarries the plan, greedy mask and logprob rows together;max_top_logprobsis computed once (clamped to the vocab) before the all-greedy early return.compute_logprobsreads the raw batch logits --apply_logits_processorsworks on a copy -- so penalties andlogit_biasdo not change the reported values (the PR's stated semantics).openai_api.py: feat(server): OpenAI-compatible logprobs for chat and legacy completions FlashML-org/FreeToken#224's "logit_bias / echo / suffix is not supported" rejections are dropped (feat(server): OpenAI protocol coverage -- sampling extras, n, echo/suffix, usage details, tokenize/detokenize/metrics #47 implements all three); logprobs are attached per choice in then > 1chat loop and in the streaming chat/completion chunks, which keep feat(server): OpenAI protocol coverage -- sampling extras, n, echo/suffix, usage details, tokenize/detokenize/metrics #47'sindex.tokenizer/server.py,detokenize.py,core.py,engine.py, the message types: unions (reasoning-token counts, prefill span and logprob entries all ride the same reply).77d8d5c: the PR's sampler test fakes get theneeds_logits_processingflag feat(server): OpenAI protocol coverage -- sampling extras, n, echo/suffix, usage details, tokenize/detokenize/metrics #47's plan reads.Testing (RTX 5080, production flags):
logprobs: 3-> tokens[' Paris', '.', ' The'], logprobs[-0.578, -0.538, -1.232], offsets[0, 6, 7], three top alternatives per token; the streaming variant returns the same tokens;n: 2returns a logprob list per choice;logprobs: 9-> 400; chat +logprobs-> 400 with the redirect message; plain chat unchanged (nologprobskey).tests/server tests/scheduler tests/engine tests/tokenizer: 1207 passed.pytest tests/ -m "not slow"at the top of this stack (77d8d5c): 2446 passed, 206 skipped; the 4 failures are the known environment-only ones (3xtest_muse_glimmer_vision.pyon transformers 5.15.1,test_async_pull_coalescedneedingpytest-asyncio).🤖 Generated with Claude Code
https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt