Repository navigation
feat(fishaudio): add realtime STT - #2940
rosetta-livekit-bot[bot] wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: a980d35 The changes in this PR will be included in the next version bump. This PR includes changesets to release 42 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| #onCommitted(msg: WireMessage): void { | ||
| if (this.#currentTurn(msg)) this.#turn = null; | ||
| if (msg.user_turn === false) { | ||
| this.#logger.debug( | ||
| { turnKind: msg.turn_kind, sessionId: this.#sessionId }, | ||
| "ignoring a turn that isn't the user's", | ||
| ); | ||
| this.#endSpeech([]); | ||
| return; | ||
| } | ||
| this.#startSpeech(numberValue(msg.audio_start_ms)); |
There was a problem hiding this comment.
🔴 Stale commits close the active turn
When an older committed arrives during a new turn, #onCommitted emits its transcript and ends the new speech. #currentTurn only guards clearing the turn, not processing the stale commit.
Learn more
A stream holds its active provider turn in #turn. Draft and preflight events require matching turn indexes via #currentTurn, but a committed event always emits a final transcript and ends speech even when its index differs. A delayed old commit therefore changes the active turn's output and speech boundary. The existing stale-message test only covers a non-user commit and does not assert that the active turn remains open.
Example: After user_turn_taken for index 1, a late user committed for index 0 produces index 0's final transcript and END_OF_SPEECH for index 1. The subsequent index 1 draft can no longer describe the correct open speech.
Recommended fix: Ignore committed messages whose turn_index does not match an active #turn, except for a genuinely standalone commit when no turn exists. Cover both user and non-user stale commits while another turn is speaking.
Was this helpful? React with 👍 or 👎 to provide feedback.
| this.#ws = null; | ||
| this.#reportUsage(); | ||
| closeWebSocket(ws); |
There was a problem hiding this comment.
🟡 Shutdown loses final recognition usage
On normal STT.close(), #reportUsage runs after the base stream closes its queue. #put discards the remaining audio duration, so short sessions report no recognition usage.
Learn more
The sender accumulates audio duration and reports it at five-second intervals. The remainder is reported when the provider task exits, but SpeechStream.close immediately closes the event queue. The normal STT.close path calls that close before waiting for the provider task, and #put refuses to enqueue once the queue is closed. Thus closing an active stream drops the final usage event.
Example: A caller sends three seconds of audio and calls fish.close(). No five-second usage event has fired, and the final three seconds are discarded instead of being published.
Recommended fix: Flush pending usage before the base stream closes the queue, or provide an awaited shutdown path that keeps event emission open until the provider task reports usage.
Was this helpful? React with 👍 or 👎 to provide feedback.
| #addUsage(durationSeconds: number): void { | ||
| this.#unreportedAudioDuration += durationSeconds; | ||
| if (this.#unreportedAudioDuration >= USAGE_REPORT_INTERVAL_SECONDS) this.#reportUsage(); |
There was a problem hiding this comment.
🟡 Usage duration uses unnamed seconds
#unreportedAudioDuration stores seconds without the required InS suffix. The repository requires milliseconds by default, so this field violates its time-unit contract.
Learn more
The repository's time-unit rule requires milliseconds by default and reserves seconds for names ending in InS. The usage accumulator adds seconds from the sender, compares against a five-second threshold, and emits seconds in recognitionUsage.audioDuration. Those provider-facing seconds are valid, but the local accumulator's name does not disclose its units.
Example: Twenty milliseconds of PCM adds 0.02 to #unreportedAudioDuration, not 20 as the field name implies under the project convention.
Recommended fix: Rename the seconds-valued accumulator and associated local seconds variables to end in InS, or accumulate milliseconds and convert to seconds only when emitting recognition usage.
Was this helpful? React with 👍 or 👎 to provide feedback.
| this.#logger.warn( | ||
| { errorType, requestId: msg.request_id }, | ||
| `fish audio stt error: ${message}`, | ||
| ); |
There was a problem hiding this comment.
🟥 Provider error text bypasses log redaction
For recoverable errors, #onError embeds the provider's message directly in a log message. If that message contains speech, redaction cannot remove it.
Was this helpful? React with 👍 or 👎 to provide feedback.
Summary
Ports livekit/agents#8357 to add Fish Audio realtime streaming STT (
transcribe-stream-1).Source: livekit/agents#8357
Source diff coverage
livekit-plugins/livekit-plugins-fishaudio/README.md->plugins/fishaudio/README.md: ported STT overview and usage to TypeScript, npm, and agents-js session APIs.livekit-plugins/livekit-plugins-fishaudio/livekit/plugins/fishaudio/__init__.py->plugins/fishaudio/src/index.ts: exportsSTT,SpeechStream, and their public options from the TypeScript barrel.livekit-plugins/livekit-plugins-fishaudio/livekit/plugins/fishaudio/models.py->plugins/fishaudio/src/models.ts: added TypeScriptSTTModelsandTurnDetectionModeunions.livekit-plugins/livekit-plugins-fishaudio/livekit/plugins/fishaudio/stt.py->plugins/fishaudio/src/stt.ts: ported the MessagePack WebSocket protocol, options, turn events, timestamps, context/keyterms, usage reporting, lifecycle, and reconnect behavior to agents-js STT primitives.livekit-plugins/livekit-plugins-fishaudio/pyproject.toml->plugins/fishaudio/package.json: updated package description and keywords; existing@msgpack/msgpackandwsdependencies cover the implementation.tests/test_plugin_fishaudio_stt.py->plugins/fishaudio/src/stt.test.ts: ported all 36 source fake-WebSocket/unit tests as distinct Vitest cases.tests/test_stt.py->plugins/fishaudio/src/stt.test.ts: agents-js has no central provider STT matrix, so the Fish Audio stream-only registration is an environment-gated plugin-local generic STT integration suite.agents/src/types.tsandagents/src/stt/stt.ts: the target lacked the source framework's configurable WebSocket liveness timeout and awaitable speech-stream settlement; both are required for Fish Audio half-open recovery andSTT.close()parity.Verification
pnpm exec vitest run agents/src/stt plugins/fishaudio/src/stt.test.ts: 99 passed, 1 skipped (requiresFISH_API_KEY)pnpm --filter @livekit/agents build: passedpnpm --filter @livekit/agents-plugin-fishaudio build: passedpnpm format:check: passedagentssuite: 4303 passed, 5 skipped, 4 unrelated baseline failures (three loop-monitor assertions require per-thread CPU accounting unavailable in this runtime; one existing TCP backpressure timing assertion)src/tts.test.ts:747; all STT tests passcue-cli: not available in this environment, so runtime voice-agent validation could not be runChangeset
Minor bumps for
@livekit/agentsand@livekit/agents-plugin-fishaudiofor the new public APIs.Ported from livekit/agents#8357
Original PR description
summary
Adds
fishaudio.STT: streaming STT on Fish Audio's realtime ASR (transcribe-stream-1). ReadsFISH_API_KEY; no new dependencies.manual_flush, so the session finalizes Fish Audio's turn when it ends the user's turn.turn_detection="stt", Fish Audio ends each turn. Its defaultautomode ignores speech that isn't the user's (noise, other speakers);vadandmanualmodes are available too.chat_context).test plan
tests/test_plugin_fishaudio_stt.py(unit, fake WebSocket): 36 passed.tests/test_stt.py --stt -k fishaudio: passed.turn_detection="stt".If #8147 lands first, I'll port this to
_run_connection_tasks.