refactor: Implement low-latency STT pipeline with speculative transcription
Major architectural overhaul of the speech-to-text pipeline for real-time voice chat: STT Server Rewrite: - Replaced RealtimeSTT dependency with direct Silero VAD + Faster-Whisper integration - Achieved sub-second latency by eliminating unnecessary abstractions - Uses small.en Whisper model for fast transcription (~850ms) Speculative Transcription (NEW): - Start transcribing at 150ms silence (speculative) while still listening - If speech continues, discard speculative result and keep buffering - If 400ms silence confirmed, use pre-computed speculative result immediately - Reduces latency by ~250-850ms for typical utterances with clear pauses VAD Implementation: - Silero VAD with ONNX (CPU-efficient) for 32ms chunk processing - Direct speech boundary detection without RealtimeSTT overhead - Configurable thresholds for silence detection (400ms final, 150ms speculative) Architecture: - Single Whisper model loaded once, shared across sessions - VAD runs on every 512-sample chunk for immediate speech detection - Background transcription worker thread for non-blocking processing - Greedy decoding (beam_size=1) for maximum speed Performance: - Previous: 400ms silence wait + ~850ms transcription = ~1.25s total latency - Current: 400ms silence wait + 0ms (speculative ready) = ~400ms (best case) - Single model reduces VRAM usage, prevents OOM on GTX 1660 Container Manager Updates: - Updated health check logic to work with new response format - Changed from checking 'warmed_up' flag to just 'status: ready' - Improved terminology from 'warmup' to 'models loading' Files Changed: - stt-realtime/stt_server.py: Complete rewrite with Silero VAD + speculative transcription - stt-realtime/requirements.txt: Removed RealtimeSTT, using torch.hub for Silero VAD - bot/utils/container_manager.py: Updated health check for new STT response format - bot/api.py: Updated docstring to reflect new architecture - backups/: Archived old RealtimeSTT-based implementation This addresses low latency requirements while maintaining accuracy with configurable speech detection thresholds.
This commit is contained in:
26
bot/api.py
26
bot/api.py
@@ -2541,7 +2541,7 @@ async def initiate_voice_call(user_id: str = Form(...), voice_channel_id: str =
|
||||
|
||||
Flow:
|
||||
1. Start STT and TTS containers
|
||||
2. Wait for warmup
|
||||
2. Wait for models to load (health check)
|
||||
3. Join voice channel
|
||||
4. Send DM with invite to user
|
||||
5. Wait for user to join (30min timeout)
|
||||
@@ -2642,16 +2642,10 @@ Keep it brief (1-2 sentences). Make it feel personal and enthusiastic!"""
|
||||
|
||||
sent_message = await user.send(dm_message)
|
||||
|
||||
# Log to DM logger
|
||||
await dm_logger.log_message(
|
||||
user_id=user.id,
|
||||
user_name=user.name,
|
||||
message_content=dm_message,
|
||||
direction="outgoing",
|
||||
message_id=sent_message.id,
|
||||
attachments=[],
|
||||
response_type="voice_call_invite"
|
||||
)
|
||||
# Log to DM logger (create a mock message object for logging)
|
||||
# The dm_logger.log_user_message expects a discord.Message object
|
||||
# So we need to use the actual sent_message
|
||||
dm_logger.log_user_message(user, sent_message, is_bot_message=True)
|
||||
|
||||
logger.info(f"✓ DM sent to {user.name}")
|
||||
|
||||
@@ -2701,15 +2695,7 @@ async def _voice_call_timeout_handler(voice_session: 'VoiceSession', user: disco
|
||||
sent_message = await user.send(timeout_message)
|
||||
|
||||
# Log to DM logger
|
||||
await dm_logger.log_message(
|
||||
user_id=user.id,
|
||||
user_name=user.name,
|
||||
message_content=timeout_message,
|
||||
direction="outgoing",
|
||||
message_id=sent_message.id,
|
||||
attachments=[],
|
||||
response_type="voice_call_timeout"
|
||||
)
|
||||
dm_logger.log_user_message(user, sent_message, is_bot_message=True)
|
||||
except:
|
||||
pass
|
||||
|
||||
|
||||
Reference in New Issue
Block a user