Skip to content

feat: register local ASR as a pluggable stt engine - #620

Open
TSOFTP-afk wants to merge 1 commit into
lipku:mainfrom
TSOFTP-afk:feat/stt-plugin
Open

TSOFTP-afk wants to merge 1 commit into
lipku:mainfrom
TSOFTP-afk:feat/stt-plugin

Conversation

@TSOFTP-afk

Copy link
Copy Markdown

Motivation

registry.py has preset a "stt" category since the 2026-03-31 modularization refactor, but it never received an implementation: zero plugins, zero create("stt", ...) call sites. Meanwhile the local SenseVoice/FunASR engine (#609, for #604) lives hard-coded inside server/asr_server.py, so the engine cannot be selected or replaced without editing the WebSocket handler.

This PR completes that design: STT becomes a registry plugin, and FunASR is the first registered engine.

What changed

  • stt/base_stt.py (new) — BaseSTT: ensure_loaded() / transcribe(audio_bytes, sample_rate, is_final) / is_available().
  • stt/funasr.py (new) — @register("stt", "funasr") FunASRSTT. The engine is moved verbatim from server/asr_server.py: lazy singleton, asyncio.Lock + threading.Lock pair, mutex-serialized inference, iic/SenseVoiceSmall + fsmn-vad, vad_kwargs={"max_single_segment_time": 30000}, trust_remote_code, <20 ms utterances skipped, rich_transcription_postprocess.
  • server/asr_server.py — keeps the /api/asr WebSocket protocol byte-identical; the engine is now obtained through registry.create("stt", ...). Silent degradation is preserved: when funasr is not installed, the endpoint is simply not registered.
  • config.py / config.yaml — new --asr key (default funasr). Also fixes the stale tts help text (cosyvoice / fishtts / indextts2 were removed from the codebase but were still listed in the help strings).
  • registry.py — the unused "output" preset is corrected to "streamout" (the category output plugins actually register under).
  • tests/test_asr_server.py — the two behavior tests (lazy singleton constructed once under concurrency; shared-model inference serialized) are migrated to the plugin, and a new protocol contract test locks the /api/asr message flow: config JSON → binary PCM16 frames → {"is_speaking": false} → reply {"text", "mode", "is_final", "timestamp"}. Tests remain dependency-free (fake modules injected into sys.modules).
  • Docs — local ASR is now documented in README.md / README-EN.md / docs/api.md, with an explicit note that server/asr_server.py (user speech recognition) and avatars/audio_features/base_asr.py (lip-sync audio features) are unrelated despite the similar names.

Not changed

/api/asr wire protocol, the web/asr/ frontend, the avatar/TTS/LLM pipelines, and avatars/audio_features/* (lip-sync feature extractors).

Verification

$ python tests/test_asr_server.py
...
Ran 3 tests in 0.311s
OK

Engine latency, measured to document the feature (the engine itself is untouched; same environment for all runs — Windows, RTX 3060 Laptop 6 GB, torch 2.13.0+cu130, funasr 1.4.16, 8.3 s utterance @ 16 kHz, 7 timed runs after one warm-up):

session Avg Min / Max Std RTF
run 1 245 ms 236 / 254 ms 7 ms 0.029
run 2 131 ms 123 / 140 ms 6 ms 0.016
run 3 141 ms 120 / 164 ms 14 ms 0.017

The run-to-run spread is environment noise (GPU clocks/thermals), not a code difference — the engine code is identical before and after the refactor. The protocol contract test is the behavior-preservation evidence.

Refs #601, #604

- move the SenseVoice/FunASR engine out of server/asr_server.py into a
  registry plugin (stt/funasr.py); registry.py presets category "stt"
  since the 2026-03-31 modularization but had no implementation
- keep the /api/asr WebSocket protocol byte-identical and preserve the
  silent degradation when funasr is not installed
- add --asr config key to select the STT engine (default: funasr)
- fix stale tts help text (cosyvoice/fishtts/indextts2 were removed)
  and the unused "output" registry preset (actual category: streamout)
- migrate tests to the plugin layout, add a /api/asr protocol contract
  test; document local ASR in README.md / README-EN.md / docs/api.md

Refs lipku#601, lipku#604
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant