Customization and scalability in AI models

From: aidotengineer

NVIDIA’s focus in enterprise-level speech AI model deployment is on delivering highly efficient, low-latency models for conversational AI, suitable even for embedded devices [00:00:36]. Their development approach emphasizes both variety and coverage, rejecting a “one model fits all” philosophy [00:06:41].

Core Principles for Model Development

NVIDIA considers four main categories when developing models [00:01:08]:

• Robustness [00:01:13]: Ensuring models perform well in diverse environments, including noisy settings and varying sound quality (e.g., telephonic audio, environmental contamination) [00:01:15].
• Coverage [00:01:31]: Addressing specific domain demands (medical, entertainment, call centers) and language requirements (monolingual, multilingual, dialect variations, code-switching) [00:01:34].
• Personalization [00:01:58]: Tailoring models to exact customer needs, which can include target speaker AI, word boosting for uncommon vocabulary, and text normalization [00:02:00].
• Deployment Cases [00:02:21]: Balancing speed and accuracy trade-offs, and deciding between high model variety or efficiency-focused solutions [00:02:24].

Model Architectures for Diverse Needs

NVIDIA employs various model architectures to achieve its goals of efficiency, accuracy, and customization:

• CTC (Connectionist Temporal Classification) Models [00:02:51]: Utilized for high-speed inference, especially in streaming environments due to their non-auto-regressive decoding [00:02:55].
• RNT (Recurrent Neural Transducer) / TDT (Transducer Decoder Transducer) Models [00:03:24]: Used when higher accuracy is needed, incorporating auto-regressive streaming setups with an internal language model [00:03:15].
• Attention Encoder-Decoder Setups [00:03:46]: Offer the highest accuracy, suitable for non-streaming scenarios. These transformer decoders are capable of handling multiple tasks within a single model, such as speech translation, timestamp prediction, language identification, and speech recognition [00:04:06].
• Fast Conformer [00:04:33]: The fundamental architecture across all decoding platforms. It enables significant subsampling (80-millisecond compression), leading to smaller audio inputs, reduced memory load, more efficient training with quicker convergence, and faster inference [00:04:37]. This directly contributes to improving AI model efficiency.

NVIDIA Reva Offerings

The Fast Conformer architecture underpins two main model offerings:

• Reva Parakeet [00:05:31]: Focused on streaming speech recognition, using CTC and TDT models for fast and efficient recognition, including speech translation and target speaker ASR [00:05:39].
• Reva Canary [00:06:00]: Utilizes Fast Conformer models for accuracy and multitask modeling, prioritizing the best possible accuracy over speed [00:06:03].

This comprehensive toolkit allows customers to choose between fast multitasking or high-accuracy models [00:06:28].

Advanced Customization and Accuracy Features

To further enhance model performance and usability, NVIDIA offers additional models and features:

• Derization Model Former (SoftFormer) [00:07:07]: An end-to-end neural deriser that integrates speaker time stamps with speaker tokens, allowing for multi-speaker and target speaker ASR scenarios [00:07:10]. This unified architecture can be fine-tuned with simple objectives [00:07:44].
• Voice Activity Detection (VAD) [00:08:36]: Detects speech segments to improve noise robustness [00:08:39].
• External Language Models [00:08:47]: Enhance ASR transcription accuracy and customization [00:08:50].
• Text Normalization (TN) and Inverse Text Normalization (ITN) [00:09:02]: Convert spoken terms to written forms for better readability [00:09:04].
• Punctuation and Capitalization (PNC) [00:09:15]: Adds punctuation and capitalization to transcriptions for improved readability [00:09:18].
• Speaker Diarization [00:09:29]: Identifies multiple speakers in a conversation [00:09:31].
• Word Boosting [00:15:16]: Improves recognition of product names, jargon, and context-specific knowledge [00:15:19].

This comprehensive approach to customization contributes to NVIDIA’s high rankings on leaderboards, such as the Hugging Face Open ASR leaderboard [00:10:00].

Training for Robustness and Coverage

NVIDIA’s training approach focuses on fundamental data development practices:

• Robust Data Sourcing [00:11:10]: Emphasizes multilingual coverage, dialect sensitivity, and comprehensive language documentation [00:11:16].
• Data Mix [00:11:30]: Incorporates both open-source data (for variety and domain shift) and proprietary data (for high-quality entity data) [00:11:32].
• Pseudo Labeling [00:11:44]: Uses transcripts from top-of-the-line commercial models to leverage community advancements and internal releases [00:11:47].
• Nemo Research Toolkit [00:12:07]: An open-source library that provides tools for GPU maximalization, data bucketing, and high-speed data loading, enabling efficient training [00:12:11].
• Validation [00:12:52]: Rigorous testing across open-source and proprietary data to minimize bias and ensure robustness across language categories before models reach end-users [00:12:54].

Deployment for Scalability

Trained models are deployed through NVIDIA Reva via NVIDIA NIM for low latency and high throughput inference [00:13:22]. This highlights scaling AI agents in production and leveraging AI tools for efficiency and scalability.

• High-Performance Inference [00:13:31]: Powered by NVIDIA TensorRT optimizations and the NVIDIA Triton inference server [00:13:34].
• Deployment Versatility [00:13:39]: Available as gRPC-based microservices for low-latency streaming and high-throughput offline use cases [00:13:42].
• Containerization [00:13:50]: NVIDIA Reva is fully containerized, allowing it to easily scale to hundreds of streams [00:13:52].
• Flexible Deployment Environments [00:13:56]: Can be run on-prem, in any cloud, at the edge, or on embedded platforms, supporting diverse applications like contact centers, consumer apps, and video conferencing [00:14:01].
• NVIDIA NIM [00:14:14]: Offers pre-built containers and industry-standard API support for custom models and optimized inference engines [00:14:18].

Addressing Customization Pain Points

One common pain point in real-world scenarios is the need for deep customization due to domain-specific knowledge (e.g., medical terms, menu names, telephonic conditions) [00:14:29]. NVIDIA Reva addresses this by offering customization features at every stage [00:14:56]. This includes the ability to fine-tune acoustic models (Parakeet/Canary based), external language models, punctuation models, and inverse text normalization models [00:15:04].

Getting Started

NVIDIA Reva models are available in NVIDIA NIM, with resources such as quick start guides, developer forums, and fine-tuning guides for the Nemo framework provided [00:15:32]. More information can be found at build.envidia.com/explore/speech [00:15:44].