From: hu-po
Recent advancements in speech-to-speech models have focused on achieving low latency to enable more natural, real-time spoken dialogue [00:07:07]. This contrasts with traditional cascaded systems that suffer from significant delays [00:05:00].
Challenges with Traditional Pipeline Systems
Older systems for spoken dialogue rely on a pipeline of independent components:
- Voice Activity Recognition (VAR) [00:05:01]
- Speech Recognition (ASR): Converts audio to text [00:05:01].
- Textual Dialogue (LLM): Processes text and generates a text response [00:05:03].
- Speech to Speech (TTS): Converts text back into audio [00:05:04].
The primary issue with this cascaded approach is that latency compounds across the many components, leading to a typical global latency of several seconds [00:06:04]. This delay is too slow for humans to experience natural conversation [00:06:20]. Additionally, paralinguistic information like emotion and accent is often lost when audio is converted to text and then back to speech [00:06:28].
Modern Low-Latency Approaches
Two open-source papers, Moshi and Llama Omni, propose solutions to address these latency challenges by integrating these components into a more unified system [00:06:52].
Moshi
Moshi aims for real-time dialogue by processing input and output audio streams jointly into two auto-regressive token streams [00:07:22]. This removes the traditional concept of speaker turns, allowing the model to be trained on natural conversations with overlaps and interruptions [00:07:34].
- Latency: Moshi boasts a theoretical lower limit of 160 milliseconds and achieves 200 milliseconds in practice [00:20:14].
- Time Steps: Each time step or “frame” in Moshi is 80 milliseconds [00:15:18].
- Delay for Quality: A slight delay of one to two steps (80-160 milliseconds) between semantic and acoustic tokens significantly improves generation quality [00:51:16]. This minimal delay is imperceptible to humans, allowing the model more context to produce higher quality audio [00:51:51].
- Multistream Design: The model handles multiple parallel streams, including its own speech tokens, user speech tokens, and an “inner monologue” of text tokens [00:11:16], enabled by a depth Transformer [00:43:47].
Llama Omni
Llama Omni also focuses on low-latency speech-to-speech interaction [00:19:16].
- Latency: It achieves a latency as low as 226 milliseconds [00:20:08], comparable to Moshi [00:22:53].
- Architecture: It utilizes a pre-trained speech encoder (like Whisper) [00:23:27], a speech adapter, a Large Language Model (LLM) (Llama 3.1 8B Instruct) [00:30:30], and a streaming speech decoder [00:29:16].
- Simultaneous Generation: Like Moshi, it simultaneously generates text and speech responses from speech instructions [00:22:21].
Human Perception of Latency
The goal of achieving low latency is to mimic natural human conversation [00:20:20].
- Human reaction time is around 100 milliseconds [00:20:58].
- A 60 frames per second display has about 16 milliseconds per frame [00:21:02].
- While noticeable differences exist between 10 milliseconds and 100 milliseconds, going below 1 millisecond typically doesn’t matter for human perception [00:21:50].
- The current latencies of around 200 milliseconds are considered “more than good enough” for natural interaction [00:21:59].
Implications for Future AI
The pursuit of low-latency speech-to-speech models has broader implications for AI development:
- Multimodal AI: The multi-stream design seen in Moshi could be applied to other modalities like robotics, allowing simultaneous consumption of visual, force, and proprioceptive inputs while outputting visual and action tokens [01:04:50].
- Model Quantization: For models to run locally on devices like cell phones or VR headsets, efficient quantization techniques are crucial [01:33:13]. This impacts the performance and implications of quantized language models, influencing whether AI interactions are cloud-based (centralized control) or local (decentralized) [01:34:21].
- Data Generation: The reliance on synthetically generated datasets (e.g., converting text data into speech data using TTS models like Cozy Voice 300M sft [01:23:37]) suggests that user-generated data may not be a necessary “moat” for developing high-performing AI, especially given the small “sim-to-real gap” in audio [01:31:02]. This contributes to the overall efficiency of large language models by reducing reliance on costly human annotation.
- Evaluation Metrics: The use of AI (like GPT-4) to score content and style in evaluating model output has become a standard, though it raises questions about bias and independent assessment [01:15:06].