Benchmark and optimize WhisperX alignment pipeline

[carochacs]

Context

Issue #6 clarified that replacing WhisperX with standalone Faster-Whisper is not the right fix for lyric/syllable-derived alignment. WhisperX already uses Faster-Whisper/CTranslate2 for transcription, and standalone Faster-Whisper would mainly remove the forced-alignment stage that provides useful word/character timing.

This issue tracks the better fix: keep the WhisperX-style transcription + forced-alignment architecture, then benchmark and optimize the slow or unreliable stages directly.

Goal

Reduce end-to-end latency while preserving alignment quality for isolated vocal stems and lyric timing workflows.

Proposed Work

1. Benchmark the current pipeline by stage

   • Measure VAD time.
   • Measure Faster-Whisper/CTranslate2 transcription time.
   • Measure forced-alignment time.
   • Measure optional diarization time, if enabled.
   • Record total runtime and real-time factor for representative stems.

2. Tune transcription throughput

   • Test batch_size values that fit the target GPU memory.
   • Compare compute_type options, e.g. float16, int8_float16, or other supported modes.
   • Compare model sizes if latency is a hard constraint.
   • Test VAD/chunking settings to avoid unnecessary segmentation overhead or poor batching.

3. Treat alignment as a separate optimization target

   • Measure whether the wav2vec2-family alignment stage is the true bottleneck.
   • Evaluate language-specific alignment models if the current aligner performs poorly.
   • For singing-specific edge cases, test stems with sustained vowels, vibrato, melisma, pitch correction, and dense backing audio.

4. Expose alignment confidence

   • Capture confidence or score data from the alignment step where available.
   • Flag low-confidence words, characters, or syllable-derived regions.
   • Route low-confidence regions to a UI fallback, manual review, or slower secondary alignment path.

5. Define acceptance metrics

   • End-to-end runtime target on the intended hardware.
   • Alignment quality target for word/character timing.
   • Low-confidence region detection behavior.
   • Regression check against representative vocal stems.

Non-goals

• Do not replace WhisperX with standalone Faster-Whisper as the default alignment pipeline.
• Do not assume Distil-Whisper is suitable without benchmarking on singing voice stems.
• Do not treat Whisper/Faster-Whisper native word timestamps as equivalent to WhisperX forced alignment.

Expected Outcome

A measured optimization path that preserves precise lyric timing while reducing avoidable latency. The final implementation should identify where time is actually spent, tune the transcription stage safely, and improve robustness by surfacing alignment confidence instead of removing the alignment stage.