FallbackAdapter: allow a separate attempt timeout for fallback LLMs #6916

Activity

1. Rehansanjay commented on Oct 22, 2026

   

   Rehansanjay

   on Oct 22, 2026

   Contributor

   More actions

   Picking up the aside at the end of this — "presumably the same reasoning applies to the STT/TTS fallback adapters" — because on the streaming path the problem there is worse than a shared timeout, and it bears on how attempt_timeout is specified.

   stt.FallbackAdapter does not bound an attempt itself. It passes the value down and trusts the plugin:

   main_stream = stt.stream(
       conn_options=dataclasses.replace(
           self._conn_options,
           max_retry=self._fallback_adapter._max_retry_per_stt,
           timeout=self._fallback_adapter._attempt_timeout,
           ...

   There is no asyncio.wait_for and no timeout context manager anywhere in stt/fallback_adapter.py on the streaming path — the non-streaming recognize call catches asyncio.TimeoutError, but a SpeechStream only ends when the plugin decides it has.

   That matters because a lot of plugins never spend conn_options.timeout on the read. I measured all nineteen streaming STT plugins against a local server that accepts the websocket upgrade and then sends nothing and answers no pings — a provider, or a NAT, gone without a FIN or an RST:

   fail over	smallestai 46.4s, telnyx 46.9s, deepgram 47.0s
   detect it and never report it	soniox (~46s), simplismart (~50.6s)
   never notice at all	assemblyai, baseten, elevenlabs, fireworksai, gladia, gradium, inworld, openai

   For those eight, the attempt does not end, so attempt_timeout is inert on the streaming path and the adapter never switches — not slowly, at all. ~46s for the ones that do work is heartbeat=30.0 plus aiohttp's 15s pong wait; the plugins that notice pass a heartbeat to ws_connect and the ones that do not, do not.

   The middle row is the one I would watch if you are relying on failover: soniox detects the dead socket, requests a reconnect, is connected again 8ms later, and repeats every ~46.7s indefinitely without ever raising — so the stream looks healthy while the caller is unheard for the whole call. That is #8031, independently reproduced by another contributor yesterday.

   TTS turns out to be the mirror image, which surprised me: there the instrument is a read timeout rather than a heartbeat, and seven of twenty plugins leave the read unbounded (#8035, currently a draft while I rework it — a connection-level deadline starts before any text is sent, so a slow first token would kill a healthy stream).

   Numbers, probe and per-plugin construction tables, if you want to re-run any of it against your own provider list:

   • https://github.com/Rehansanjay/livekit-stt-halfopen
   • https://github.com/Rehansanjay/livekit-tts-halfopen

   None of which is an argument against fallback_attempt_timeout — your cold-start point stands on its own, and a separate value for providers after the first looks right to me for LLM. It is more that if the same option lands on the STT/TTS adapters, it would be worth saying in the docs that the value is advisory: it reaches the plugin as conn_options.timeout, and whether anything honours it on the read is per-plugin.

   Happy to review #6915 if that is useful.