Repository navigation
Soniox and simplismart STT detect a dead websocket and reconnect into it forever instead of failing over #8031
Description
Summary
The soniox and simplismart STT streams detect a half-open websocket correctly and then tell nobody. Both reconnect into the same dead state and repeat indefinitely, so stt.FallbackAdapter is never informed and the caller stays unheard for the rest of the call.
This is the third category that came out of extending @DEVJHAWAR11's probe in #7900 to all nineteen streaming STT plugins. #7900 covers the plugins that never notice at all. These two do notice — which is arguably worse, because the stream looks healthy the whole time.
Measured
A local server accepts the websocket upgrade and then sends nothing and answers no pings — a provider, or a NAT in between, that vanished without a FIN or an RST. Captured today against 3c6e65124:
217ms livekit.plugins.soniox Soniox Speech-to-Text API connection established!
46898ms livekit.plugins.soniox Soniox STT WebSocket error frame: ServerTimeoutError('No PONG received after 15.0 seconds')
46899ms livekit.plugins.soniox Soniox STT WebSocket closed; requesting reconnect
46906ms livekit.plugins.soniox Soniox Speech-to-Text API connection established!
Reconnected 8 ms after detecting the dead socket, into a server that is still silent, and round again every ~46.7 s for as long as the stream lives. Nothing is raised, so the watch expires with the stream still "running". simplismart does the same at ~50.6 s, logging simplismart connection lost.
The ~46.7 s is heartbeat=30.0 plus aiohttp's 15 s pong wait plus connect. Both plugins pass a heartbeat, which is why they detect it; three other plugins pass one and fail over correctly (smallestai 46.4 s, telnyx 46.9 s, deepgram 47.0 s), so detection is clearly not the missing piece.
Mechanism
soniox (livekit-plugins-soniox/.../stt.py)
The heartbeat's unanswered ping surfaces as WSMsgType.ERROR, which is logged and then breaks out of the receive loop without raising:
if msg.type == aiohttp.WSMsgType.ERROR: # L542
logger.warning("Soniox STT WebSocket error frame: %r", self._ws.exception())
break # L548_recv_messages_task then returns normally and arms a reconnect on its way out (note this bypasses _request_reconnect, so it is not subject to that method's guards):
# Request reconnect if STT silently dies on WS drop.
if not self._reconnect_event.is_set(): # L686
logger.warning("Soniox STT WebSocket closed; requesting reconnect")
self._reconnect_event.set()_run's while True at L345 sees wait_reconnect_task complete, clears the event, and calls _connect_ws() again. No exception ever leaves _run, so the base stt.SpeechStream never learns the stream failed.
simplismart (livekit-plugins-simplismart/.../stt.py) is the same shape in one place rather than two:
if msg.type == aiohttp.WSMsgType.ERROR: # L388
if self._session.closed:
return
logger.warning("simplismart connection lost: %r", ws.exception())
self._reconnect_event.set()
returnconn_options does not bound it
I expected max_retry=0 to cap this, and it does not. The retry budget belongs to the base class, and this reconnect is the plugin's own loop, so it is invisible to it. The probe runs with APIConnectOptions(max_retry=0, timeout=10) and soniox still reconnects indefinitely.
Why I am filing this rather than sending a patch
The correct behaviour is a judgement call I do not think belongs to a contributor:
- Bound the plugin's own reconnects — after N consecutive reconnects with no traffic received, raise so the base class and
FallbackAdaptercan act. - Treat a reconnect that produces no traffic as a failure — reconnecting is right once; reconnecting forever into silence is not.
- Leave it, and document that these two plugins do not fail over so nobody puts them behind a
FallbackAdapterexpecting they will.
Option 1 needs a number, and the ~46.7 s above is already nobody's considered choice — every plugin that passes a heartbeat passes 30.0. I would rather you pick than guess in a PR. Happy to implement whichever you want.
One scoping note in fairness to both plugins: this models a server that accepts connections and then goes silent — a load balancer in front of a dead backend, or a provider bug. Against a provider that has genuinely vanished the reconnect would fail and surface an error normally.
Reproducing
Probe, per-plugin construction table and raw results: https://github.com/Rehansanjay/livekit-stt-halfopen
# from a livekit/agents checkout with the workspace installed
uv run python probe/halfopen_probe.py soniox simplismart
Activity
iamdgarcia commentedon Oct 21, 2026
Independently reproduced against 1.8.8 (
livekit-agents==1.8.8, Windows, aiohttp 3.14) with a local server that accepts the upgrade and then answers nothing, pings included.Same mechanism and same shape as your numbers: first detection at 51.5 s (
ServerTimeoutError: No PONG received after 15.0 seconds), reconnect 8 ms later into the still-silent server, cycle repeating every 46.0 s, and no exception ever reaching the consumer withmax_retry=0. Three upgrades accepted by the dead server over 115 s of draining the stream.One detail from the probe that supports option 2: the plugin's keepalive writes (one every 5 s) kept "succeeding" on the dead socket for the whole first cycle, ten writes before the pong timeout fired. A write into a half-open TCP connection doesn't fail until the send buffer fills, so on soniox the heartbeat pong timeout is the only thing that ever notices; the write-failure reconnect path (
audio write failed) won't trigger here. Reconnect-without-traffic-as-failure would catch this category and the connect-succeeds-then-silence one with the same rule.Also confirmed in code: the ERROR-frame
breakexits_recv_messages_taskand arms the reconnect event directly on the way out, without going through_request_reconnect, and the base class retry budget never sees any of it. Happy to take whichever option you pick to a PR; the probe harness is a ~70-line aiohttp server, trivial to turn into a plugin test with a fake transport.Rehansanjay commentedon Oct 22, 2026
@iamdgarcia thank you for reproducing it, and the keepalive detail is a better argument for option 2 than anything in my original report.
I had the keepalive path listed as one of the two things that can request a reconnect and did not check whether it ever fires here. It cannot, for the reason you give: a write into a half-open TCP connection sits in the send buffer and succeeds until that buffer fills, so ten keepalives can "land" on a socket with nothing at the other end. On soniox that leaves the heartbeat pong timeout as the only detector, which means the plugin has exactly one mechanism for this and it is one that detects without reporting.
That does tighten the case for option 2 over option 1. A reconnect counter needs a number, and any number is wrong for somebody; "a reconnect that produces no traffic is a failure" needs none, and as you say it covers both this shape and the connect-succeeds-then-silence shape with one rule. It also degrades correctly against a provider that has genuinely vanished, where the reconnect fails and surfaces normally today.
On the PR: please take it if you want it. The option choice is @longcw's rather than either of ours, but I would much rather this get fixed than own the fix, and you have a reproduction and the clearer framing. I will not open anything competing — I have managed to file a triplicate on this repo once already this week by trusting a truncated
gh pr list, and I would rather not repeat it from the other direction.If it helps, the harness I used is at https://github.com/Rehansanjay/livekit-tts-halfopen and https://github.com/Rehansanjay/livekit-stt-halfopen — the STT one has the soniox numbers and a per-plugin construction table, and simplismart is in the same state from the same cause if you want both in one PR. Happy to review whatever you send.
iamdgarcia commentedon Oct 23, 2026
Right, the counter needs a number and any number is wrong for someone; the traffic rule needs none and fails safe both ways. For whoever implements: the measured shape from my repro (51.5s first detection, 8ms reconnect into the same silent server, 46.0s cycle, 10 keepalive writes "succeeding" before the pong timeout fired) is in the thread above, and #8063 is now up covering the raise-on-ERROR side.
Rehansanjay commentedon Oct 27, 2026
Resolved by #8123, and not by either of the options we were weighing here.
Measured against a local server that accepts the upgrade and then sends nothing and
answers no pings,
liveness_timeoutat its default 6.0:ae5e989f8~1(before)ae5e989f8(after)APIConnectionErrorat 6.2sThe "before" row reproduces @iamdgarcia's shape independently: a 46.9s reconnect cycle
against his measured 46.0s, and the same swallow in the log just before it --
ServerTimeoutError('No PONG received after 15.0 seconds')followed bySoniox STT WebSocket closed; requesting reconnect.Both options in this thread were about how to bound the reconnect loop. Neither turns out
to be needed, because the loop is no longer reached.
WebSocketMonitor.receiveraisesAPIConnectionError("WebSocket connection lost")onceliveness_timeoutpasses with notraffic, and
_recv_messages_taskalready hasexcept APIError: raise, so the errorleaves the task instead of being logged and converted into
_reconnect_event.set().FallbackAdapteris told on the first failure rather than never.It also survives the refactor in #8147, which routes these tasks through
_run_connection_tasks: that helper doesfor task in done: task.result(), so a taskexception is re-raised rather than absorbed.
simplismart measures the same on main: 1 connection, failover at 6.2s. One small
difference, in case it is worth tidying -- it surfaces a raw
aiohttp.ClientConnectionResetErrorout of its send task rather than wrapping it, so itlands in
FallbackAdapter'sexcept Exceptionbranch and is logged as "unexpected error,switching to next STT". Failover is correct either way; only the label is off.
Closing as fixed. Thanks @iamdgarcia -- the keepalive detail and the measured shape in your
repro are what made this checkable rather than arguable.