Repository navigation
server: add realtime transcription API - #30636
Conversation
github-actions
Bot
added
the
documentation
|
/bot review |
Automated code reviewReview of PR #30636 (server: add realtime transcription API). Checklists run: scope/quick-reject gate, security, general, server ( Blocking (correctness) (point 1) First-audio race kills live sessions. (point 2) Waiting realtime slots allow the server to sleep underneath them. The Will slow the review (point 3) Batch-capacity abort on the first step. The first (point 4) Unbounded audio backlog. (point 5) No tests. A whole new public websocket API lands with zero test coverage. The e2e suite in Nits (point 6) Auth parity: the ws handler only accepts (point 7) Spurious empty (point 8) (point 9) Session-length limit is undocumented: with ctx shift disabled for multimodal, the session stops with Design-wise the approach is reasonable: reusing the control-task path for audio transport, per-connection reader thread with a mutex-guarded This review was generated automatically by pi coding agent using |
|
@ServeurpersoCom let's prioritize reviewing this and maybe merge it today or tmr, wdyt? |
Two first thing, Then, I dig deeper by conducting tests and comparisons using my minimal WebSocket protocol, a stripped-down version of the OpenAI standard from my s2s.cpp voice assistant.
|
|
Ran it in a container with an RTX PRO 6000, using the Voxtral RT Q8_0 GGUF and a small websocket client. The happy path is really nice, clean transcript and way faster than real time. Both points above reproduce: the router served a full transcript to a client with no key, and the first append race hits even on an idle server with a local client that appends right after session.created. Testing also found a third blocking one: an open realtime session freezes the other slots. A realtime slot waiting for audio is still in GENERATING, so it falls through the prompt loop and grabs I'll push commits on the branch fixing these three, so we can keep the merge on track for tonight. A few OAI compat points too, checked against the current Realtime docs:
|
httplib runs the pre-routing handler before the websocket upgrade, so the API key middleware answers 401 at handshake like any other route. This covers the router proxy, whose children run keyless, and accepts the same key forms as HTTP (Bearer, raw key, X-Api-Key).
A realtime slot waiting for audio is GENERATING with nothing to decode, so it fell through the prompt loop and took slot_batched. A request on a higher slot then failed can_batch_with on the task type and starved for the whole session. Realtime slots have no prompt: the loop sets them up once and always leaves them.
|
nice, thanks. let's merged once the CI is ok |
|
A few commits are still missing; as soon as I’m done, I’ll add my approval! |
The audio stream was created in the prompt loop, which skips a slot that cannot batch with the one generating, and session.created went out before the task was posted: early appends failed and the reader quit, leaving the socket open and the slot held until the HTTP timeout. The slot now sets up its stream at launch and signals it with the begin result, session.created waits for it, and a reader that stops closes the socket so the session ends with it.
|
Pushed three commits fixing the blocking issues: the API key middleware now guards the websocket (401 at handshake, router included), realtime slots stay out of the prompt loop so an open session no longer freezes the other slots, and the session only starts once its slot is live, so early appends work even under load or when the task is queued, and a client that leaves frees its slot right away. Tested in the same container: nominal, immediate appends on an idle and on a busy server, a queued session with -np 1, a plain completion next to a waiting session, and auth in single and router mode. What's left is : OAI compat polish, nothing blocking technically: 24 kHz input with server side resampling, the swapped ids in input_audio_buffer.committed, sending completed once the flush has drained, and the GA session shape I’m looking at what’s worth commit, and what we’ll leave for a follow-up. |
Per the Realtime spec, item_id is the item made from the committed audio and previous_item_id the one committed before it, null for the first. committed now comes before the transcript that completes that item, and both go out with the item switch under one lock, so a delta cannot land between them.
The session object follows the GA Realtime schema: type transcription, with the input format, transcription model, turn detection and noise reduction under audio.input. The rate stays the model sample rate, which is what the client must send for now.
ServeurpersoCom
left a comment
ServeurpersoCom
left a comment
There was a problem hiding this comment.
LGTM! Everything above is fixed and tested in the same container: auth in single and router mode, realtime slots alongside regular requests, early and queued sessions, committed ids, and the GA session shape.
Possible follow-ups, none blocking:
- 24 kHz input with server side resampling, so any OAI client works without knowing the model rate (separate PR).
- Send completed once the commit flush has drained, so the words the commit releases stay in the committed item instead of the next one.
* server: add realtime transcription API * drop transcription_sessions * proxy ws * add docs * server: let the API key middleware guard the websocket httplib runs the pre-routing handler before the websocket upgrade, so the API key middleware answers 401 at handshake like any other route. This covers the router proxy, whose children run keyless, and accepts the same key forms as HTTP (Bearer, raw key, X-Api-Key). * server: keep realtime slots out of the prompt loop A realtime slot waiting for audio is GENERATING with nothing to decode, so it fell through the prompt loop and took slot_batched. A request on a higher slot then failed can_batch_with on the task type and starved for the whole session. Realtime slots have no prompt: the loop sets them up once and always leaves them. * server: start the realtime session once its slot is live The audio stream was created in the prompt loop, which skips a slot that cannot batch with the one generating, and session.created went out before the task was posted: early appends failed and the reader quit, leaving the socket open and the slot held until the HTTP timeout. The slot now sets up its stream at launch and signals it with the begin result, session.created waits for it, and a reader that stops closes the socket so the session ends with it. * server: name the committed item in input_audio_buffer.committed Per the Realtime spec, item_id is the item made from the committed audio and previous_item_id the one committed before it, null for the first. committed now comes before the transcript that completes that item, and both go out with the item switch under one lock, so a delta cannot land between them. * server: use the GA transcription session shape The session object follows the GA Realtime schema: type transcription, with the input format, transcription model, turn detection and noise reduction under audio.input. The rate stays the model sample rate, which is what the client must send for now. --------- Co-authored-by: Pascal <admin@serveurperso.com>
Overview
Follow up #30629
Add
/v1/realtimewebsocket API, OAI-compatibleGGUF for testing: https://huggingface.co/ggml-org/Voxtral-Mini-4B-Realtime-2602-GGUF
TODO:
Requirements