[davidzhao]

most STT plugins shared a very similar strategy (websocket, push audio, process server-side events) but duplicated a lot of code on connection lifecycle and error handling.

this created a lot of inconsistencies. some plugins completely ignored error frames, causing them to not handle fallback correctly when remotes stop responding. others had inconsistent timeouts.

since we now have a common monitor in #8123, this change improves the handling of timeouts & errors so plugins no longer have to worry about them.

also added a comprehensive test suite that describes each provider's protocol, so our plugins can run unit tests against that protocol against the full suite.

[iamdgarcia]

One datum for this monitor, from the soniox/simplismart half-open repro in #8031: the dangerous case never produces an ERROR frame until the pong timeout fires. A peer that vanishes without a FIN or RST accepts the upgrade and then answers nothing, and the plugin's keepalive writes keep succeeding into the half-open TCP connection (ten writes over one cycle in our capture, one every 5 s), because a write into that socket only fails once the send buffer fills. First detection was 51.5 s (ServerTimeoutError: No PONG received after 15.0 seconds), the reconnect landed 8 ms later into the same silent server, the cycle repeated every 46.0 s, and no exception ever reached the consumer.

So the thing worth centralizing is not only the handling but the reporting: the plugins that did notice did it via the pong timeout and then told nobody, so stt.FallbackAdapter never heard about it and the caller stayed unheard for the rest of the call. If WSMonitor owns reconnects as well as errors, the rule that covers this shape is the one from the #8031 thread: a reconnect that produces no traffic is a failure. It needs no tunable number and it fails safe both ways; the measured shape (51.5 s first detection, 8 ms reconnect, 46.0 s cycle, 10 keepalive writes succeeding) is there to test against.

[davidzhao]

One datum for this monitor, from the soniox/simplismart half-open repro in #8031: the dangerous case never produces an ERROR frame until the pong timeout fires. A peer that vanishes without a FIN or RST accepts the upgrade and then answers nothing, and the plugin's keepalive writes keep succeeding into the half-open TCP connection (ten writes over one cycle in our capture, one every 5 s), because a write into that socket only fails once the send buffer fills. First detection was 51.5 s (ServerTimeoutError: No PONG received after 15.0 seconds), the reconnect landed 8 ms later into the same silent server, the cycle repeated every 46.0 s, and no exception ever reached the consumer.

@iamdgarcia the latest should be addressing this in a consistent way across all plugins. could you try this branch and let me know how it works for you?

[iamdgarcia]

Ran the #8031 half-open repro against 23dee5d and it resolves the reporting gap.

Setup (same shape as the #8031 repro): local aiohttp WS server that accepts the upgrade and then answers nothing, no pongs; soniox plugin from this branch, max_retry=0, conn timeout 10s.

Before (1.8.8, numbers from the #8031 thread): first detection 51.5 s (ServerTimeoutError, No PONG after 15 s), reconnect 8 ms later into the same dead server, cycle repeating every 46.0 s, ten keepalive writes "succeeding" on the half-open socket, and no exception ever reaching the consumer over 115 s of draining.

After (23dee5d): a single connection, no reconnect cycle, and APIConnectionError("WebSocket connection lost" <- ServerTimeoutError: No WebSocket messages received for 6.0 seconds) raised to the consumer at 7.1 s wall clock (6.0 s idle timeout plus ~1 s of setup). The keepalive trap is bypassed by construction: the monitor keys on received activity, not on write success, so ten "successful" writes into the dead socket can no longer pass for liveness.

That is the property we argued for in #8031: detection that reports. Happy to run the same dead-socket probe against other migrated plugins (deepgram or assemblyai) if that is useful as an end-to-end complement to the harness contract tests.

[devin-ai-integration]

Devin Review found 1 new potential issue.

[devin-ai-integration]

🔴 Failed utterances disappear without an error

When OpenAI rejects the only transcription item, pending_items drops it without raising an error. The stream then finishes successfully with no transcript, so fallback cannot handle the failed utterance.

Learn more

The receiver tracks committed audio items in pending_items and waits for them before completing a drained stream. A failed item now leaves that set just like a completed item, but emits no speech event or error. If it was the only item, the stream reports success despite losing all user speech. If other items remain, they can complete normally, so treating every failed item as an immediate fatal error would lose valid later transcripts.

Example: A caller sends one utterance, ends input, and OpenAI sends conversation.item.input_audio_transcription.failed for its item followed by input_audio_buffer.cleared. pending_items becomes empty and the stream closes without a final transcript or failure; the caller cannot invoke fallback.

Recommended fix: Track failed item errors separately. Allow remaining committed items to finish, then propagate an API error if any item failed so the stream cannot report full success after dropping speech. Account for failures that occur before input is closed as well.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[chenghao-mou]

I have a concern about this pattern where we create task in one place and a helper function handles their cancellation and gather in another place.

Claude found the following potential issues:

• double cancellation and clean up: sarvam and rtzr
• rtzr also passed a _recv_failed future instead of a task
• tasks could potentially leak if unintentionally someone added await between task creation and try block in _run_connection_tasks

and suggested this instead:

async def _run_connection_tasks(
    self,
    sender: Coroutine[Any, Any, None],
    receiver: Coroutine[Any, Any, None],
    *,
    background: Sequence[Coroutine[Any, Any, None]] = (),
    ...
) -> bool:
    # create first so no coroutine is left un-awaited if setup below raises
    sender_task = asyncio.create_task(sender)
    receiver_task = asyncio.create_task(receiver)
    bg_tasks = [asyncio.create_task(c) for c in background]
    try:
        ...
    finally:
        await aio.cancel_and_wait(sender_task, receiver_task, *bg_tasks)

[chenghao-mou]

does this mean it will not retry even if the error match the retryable error codes (1006/1011/1012/1013 etc.)?

[chenghao-mou]

I can't seem to find this contract (flush -> done) on their docs: https://docs.mistral.ai/studio/audio/speech-to-text/transcribe-in-realtime#events

[chenghao-mou]

nit: I was not able to find the contract for this from https://docs.inworld.ai/api-reference/sttAPI/speechtotext/transcribe-stream-websocket

Keep reading until the server closes the stream, including any final transcripts and result.usage. A final transcript ends a turn, not necessarily the stream. Do not close the socket immediately after sending the control message.

[chenghao-mou]

nit: also can't find contract of this from https://aistudio.google.com/api-reference/live

Optional. Indicates that the audio stream has ended, e.g. because the microphone was turned off.

This should only be sent when automatic activity detection is enabled (which is the default).

The client can reopen the stream by sending an audio message.