Repository navigation
fix(deepgram): recover Nova streaming STT from a dead socket without losing speech - #2757
cosmin-staicu wants to merge 1 commit into
Conversation
…losing speech When the Nova socket dropped mid-turn the stream kept its speaking state, so the next SpeechStarted was ignored and the turn never ended. Audio sent to a socket that died without closing was lost, and a turn that stopped producing results on a socket that still answered pings waited forever. - commit the open turn from its last interim and end speech when a connection drops - keep the audio Deepgram has not transcribed (trimmed to each result's window) and send it first on the next socket - reconnect when an open turn gets no message for turnStallTimeoutMs - stop after maxRetry consecutive drops without a finished turn, with a non-retryable APIConnectionError that run() rethrows instead of redialing - skip usage reports once the event queue is closed - ignore messages from a connection that has been given up on Signed-off-by: Cosmin Staicu <cosmin.staicu@uipath.com>
🦋 Changeset detectedLatest commit: d28c63c The changes in this PR will be included in the next version bump. This PR includes changesets to release 41 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 3 potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
| this.#sentSeconds = 0; | ||
| for (const entry of this.#replay) { | ||
| this.#sentSeconds += entry.seconds; | ||
| entry.end = this.#sentSeconds; | ||
| ws.send(entry.data); | ||
| } |
There was a problem hiding this comment.
🔴 Flushed speech remains unfinished after recovery
When a socket drops after flush(), #replay resends audio without the consumed Finalize command. With no more input, Deepgram never receives the boundary needed to finalize that speech.
Learn more
A flush places a sentinel in the input queue; the sender consumes it and sends Deepgram a Finalize message. If the socket fails before returning a transcript, the next connection replays the audio but the sentinel has already been consumed. Deepgram therefore receives audio without the explicit boundary the caller requested, and an idle caller can wait indefinitely for its final transcript.
Example: The caller pushes 100 ms of speech and calls flush(). The first connection sends audio and Finalize but disconnects before a result. The new connection receives the replayed 100 ms and no Finalize; no further caller input arrives.
Recommended fix: Track outstanding flush boundaries alongside replayed audio and resend Finalize after the corresponding replayed segment. Clear each outstanding boundary only when its result is acknowledged; preserve boundaries across consecutive drops.
Was this helpful? React with 👍 or 👎 to provide feedback.
| const armTurnStallTimer = () => { | ||
| clearTurnStallTimer(); | ||
| if (!this.#speaking) return; | ||
| turnStallTimer = setTimeout(() => { | ||
| this.#logger.warn( | ||
| { turnStallTimeoutMs: this.#opts.turnStallTimeoutMs }, | ||
| 'Deepgram stopped producing mid-turn, committing the open turn and reconnecting', | ||
| ); | ||
| failConnection('Deepgram stopped producing mid-turn'); | ||
| ws.close(); | ||
| }, this.#opts.turnStallTimeoutMs); |
There was a problem hiding this comment.
🔴 Long speech ends when interim results are off
With interimResults: false, turnStallTimeoutMs can fire during a long utterance before Deepgram sends a final result. It commits the active turn and reconnects despite continued audio.
Learn more
The timer starts when SpeechStarted arrives and is rearmed only on inbound messages. With interim results disabled, Deepgram need not produce another result until a speech segment is finalized. A long uninterrupted utterance can therefore exceed the default ten-second timeout while audio continues to flow, creating a false end-of-speech boundary.
Example: With interimResults: false, a caller talks continuously for 15 seconds. Deepgram sends SpeechStarted immediately but has not finalized a segment by ten seconds. The timer synthesizes END_OF_SPEECH while the caller is still speaking.
Recommended fix: Make the stall policy account for interimResults: false and active outbound audio, or require evidence that a result is overdue rather than using elapsed time since any inbound message alone.
Was this helpful? React with 👍 or 👎 to provide feedback.
| // The interim's words are committed here, so only the audio past its window is replayed. | ||
| this.#forgetTranscribed(this.#lastInterimEnd); |
There was a problem hiding this comment.
🔴 Trailing speech lost after interim commit
If an interim window extends beyond its last recognized word, #forgetTranscribed discards that trailing audio on a drop. The replacement never hears speech not represented by the committed transcript.
Learn more
Deepgram's start + duration describes the result's audio window, while the returned words have their own end times. A nonfinal result can include unrecognized audio after its last word. Recovery publishes its provisional words as final, then deletes replay through the end of the whole window. Any trailing sound omitted from the words disappears from both the emitted transcript and the replay.
Example: An interim result for 0–1.0 seconds contains a word ending at 0.7 seconds; the next word begins at 0.8 seconds but is not yet recognized. The socket drops. Recovery commits the first word and trims replay through 1.0 seconds, so the new socket never hears the start of the next word.
Recommended fix: Trim only audio known to correspond to committed text, retaining a suitable overlap after the last recognized word; handle windows with no word timing conservatively.
Was this helpful? React with 👍 or 👎 to provide feedback.
Problem
The Nova stream (
plugins/deepgram/src/stt.ts) already redials when its socket closes, and the heartbeat added in #2470 terminates a socket that stops answering pings. What happens to the turn and to the audio around a drop is still wrong:#speakingset. The nextSpeechStartedon the new socket is ignored (if (this.#speaking) return), and the open turn never ends until Deepgram happens to send an endpoint.END_OF_SPEECH.APIErrormarked non-retryable is retried too.onAudioDurationReportcallsqueue.putwithout checking the queue. After the queue closes, the next report throwsQueue is closedinside the send loop, which looks like a connection failure and redials.Change
FINAL_TRANSCRIPT, thenEND_OF_SPEECH) so the session is not left mid-turn. A pending keyterm update is applied at that point.start + durationtrims what is kept, and the last half second of reported silence stays because Deepgram can attribute it to the speech that follows. The buffer is capped at the heartbeat's worst-case blind window plus a margin.turnStallTimeoutMs(default 10 s). If an open turn gets no message for that long, the stream commits it and reconnects.connOptions.maxRetryof them the error is a non-retryableAPIConnectionError, andrun()rethrows non-retryableAPIErrors instead of redialing. Errors stayAPIErrorsubclasses, soSpeechStream.mainTaskemits them as non-recoverable and the session handles them as it does other STT failures.AudioByteStreamfor the stream's lifetime so the partial chunk it holds survives a reconnect.stt_v2.tsis not touched.Tests
New cases in
stt_recovery.test.tsrun against a localwsserver. Five fail on currentmainand pass with this change: usage after the queue closed, commit on a mid-turn drop and a freshSTART_OF_SPEECHafterwards, replay of unacknowledged audio only, the stall timeout, and the drop budget. Two more pin behavior that must not regress: silence between turns is not a stall, and drops are counted per unfinished turn rather than over the life of the stream.Not covered here: timestamps after a reconnect inside
run(). The plugin's inner loop does not advancestartTimeOffset, which was already the case.