seed and chat logprobs are silently ignored (200 with no effect); proposal for per-request seeded sampling #693

What happened

Two OpenAI request fields are ignored without an error, so the client gets a 200 response that does not do what it asked for:

temperature: 0 gives the same output each time in my tests (single short requests, thinking off). Other sampling fields work: top_p, top_k, min_p, the penalties, stop and max_tokens.

Minimal reproduction:

for i in 1 2; do curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "Qwen3.8-Flash-Next", "max_tokens": 20, "temperature": 1.0, "seed": 1234,
  "chat_template_kwargs": {"enable_thinking": false},
  "messages": [{"role": "user", "content": "Invent a name for a cat."}]}' | jq -r '.choices[0].message.content'; done

Cause (current main, 0bca88f)

• server/api_models.py: ChatCompletionRequest has model_config = ConfigDict(extra="allow") and no seed or logprobs field. Both are accepted and then dropped. CompletionRequest also has no seed, but it rejects logprobs (openai_api.py around line 658).
• SamplingParams has no seed, and engine/sample.py sample_impl() calls the sampling functions without seed / offset. The Triton sampler (kernel/triton/sampling.py, _gen_u) already accepts seed and offset, but only one value for the whole batch.

Proposed fix

seed:

1. Add seed: int | None to ChatCompletionRequest, CompletionRequest and SamplingParams, and carry it to the scheduler like the other sampling fields.
2. In Sampler.prepare(), build a per-row (seed, offset) pair for the rows that have a seed. Use the number of generated tokens as the offset, so each step draws a new number but a replay draws the same one.
3. Make the uniform draw per row: in the Triton path, tl.rand(seed_row, offset_row) (Philox) inside the kernel, or one generator per seeded row in _gen_u. Rows without a seed keep the current unseeded draw. A batch with no seeded rows keeps exactly the current path. If the FlashInfer sampling path cannot take per-row seeds, send batches that contain a seeded row to the Triton path.
4. Document the limit: a seed makes the draw reproducible, but the logits can still differ between batch compositions (see temperature 0 is not reproducible with thinking on (identical requests diverge on near-ties) #307 for the near-tie case at temperature 0). So the guarantee is "best effort", like OpenAI's own seed.

logprobs:

#224 implements logprobs / top_logprobs for both routes, with raw pre-temperature values and no cost when not requested. It has been rebased and tested by several people. Merging it closes this part. Until then, /v1/chat/completions should return 400 for logprobs: true, like /v1/completions already does, and not null.

In general: extra="allow" on the request models turns every unsupported OpenAI field into a silent no-op. A short deny-list of fields that change the result (seed, logprobs, top_logprobs, logit_bias, n (see #682), response_format) that returns 400 until each one is implemented would make these gaps visible to clients.

Environment

• Built from source: upstream main 2d8b9cc plus a local deploy stack. I checked each code path above on current main 0bca88f, and none of it has changed since.
• Ubuntu 24.04, Python 3.12, 2x RTX 6000 Ada 48 GB (driver 595.91), TP=2, 503 GB RAM.
• Checkpoint nvidia/Qwen3.8-Flash-Next-NVFP4.
• All results above are reproducible directly against ft serve, with no proxy.