Skip to content

feat(gradium): optional VAD-based start of speech in STT - #8247

Merged
longcw merged 5 commits into
livekit:mainfrom
davencyw:gradium-stt-vad-start-of-speech
Nov 3, 2026
Merged

longcw merged 5 commits into
livekit:mainfrom
davencyw:gradium-stt-vad-start-of-speech

Conversation

@davencyw

@davencyw davencyw commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Adds an optional vad_start_threshold to the Gradium STT. When set, the server VAD in step messages starts speech as soon as the inactivity probability drops below it, instead of waiting for the first transcribed word. This allows fast barge-in without a local VAD, like Gradbot and Unmute (which use 0.4). Off by default.

  • Steps sent before any audio was processed (total_duration_s == 0) are ignored: their estimate reads as activity whatever the input, so they would open a false turn on every new connection.
  • A turn opened by VAD with no text still ends on silence as before.

Checked against the real server, speech start detected after speech onset:

clip vad_start_threshold=0.4 default (first word)
long sentence +0.38s +1.09s
"Wait, stop." +0.40s +1.27s
"mm-hmm" +0.33s +1.26s
sentence from the very first audio +0.27s +1.13s
sentence 0.3s after connecting +0.14s +1.12s

No false starts on 10s of near silence or 6s of white noise.

With vad=None and turn_detection="stt", this start of speech sets the user state to speaking right away, so an agent can barge in on it:

@session.on("user_state_changed")
def _on_user_state(ev):
    if ev.new_state == "speaking" and session.agent_state == "speaking":
        try:
            session.interrupt()
        except RuntimeError:
            pass  # current speech is not interruptible

Note: this overlaps with #7870 and #7871 (same file), will rebase on whichever lands first.

Add vad_start_threshold. When set, a step whose VAD inactivity probability
drops below it starts speech before any text is transcribed. A turn only
opens after silence was observed, since the estimate reads low before the
model has heard any audio. Off by default.
The server sends a few steps before any audio is processed
(total_duration_s == 0) whose estimate reads as activity. Ignoring those
lets a user who talks from the very start open a turn on the VAD, where
waiting for silence first could miss it and fall back to the first word.
@CLAassistant

CLAassistant commented Oct 29, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@davencyw
davencyw marked this pull request as ready for review October 29, 2026 14:49
@davencyw
davencyw requested a review from a team as a code owner October 29, 2026 14:49
The check tested vad_bucket for truthiness, so bucket 0 (the 0.5s horizon)
silently turned the feature off. Only None disables it.
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

The end-of-turn check still tested vad_bucket for truthiness, so with
bucket 0 a turn opened by the VAD start of speech never closed. Same
change as in livekit#7871.
@davencyw

Copy link
Copy Markdown
Contributor Author

@longcw @tinalenguyen could one of you take a look when you have a moment? A customer asked for this: fast barge-in on Gradium STT without adding a local VAD. It's opt-in (vad_start_threshold, default None), so default behavior doesn't change. Live results against our server are in the description. Happy to adjust anything.

@longcw

longcw commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

could you resolve the conflicts before we can merge?

…-of-speech

# Conflicts:
#	tests/test_plugin_gradium_stt.py
@davencyw

davencyw commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Done, merged main and resolved the conflicts (both sides only added tests). Thanks for the review!

@longcw
longcw merged commit 35df79f into livekit:main Nov 3, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants