Challenges and solutions in using large language models

From: aidotengineer

Introduction to Conversational AI and Sierra

Sierra is a conversational AI platform designed for businesses, broadening its reach from primarily chat experiences and customer service to include phone interactions, sales, subscription management, and product recommendations [00:00:27]. Sierra aims to help businesses build and improve AI agents that interact with customers across various touchpoints [00:00:59].

Evolution of AI and the “AI Caves”

The current pace of AI development is rapid, with significant advancements occurring even within the last few years [00:01:17]. Reflecting on earlier periods, 2016 felt like the “AI caves” where the focus was on helping computers distinguish between objects like Chihuahuas and blueberry muffins, or dogs and bagels/mops/fried chicken [00:02:01]. This work led to the first version of Google Lens [00:02:37].

Early challenges with models like Google Lens involved inconsistency; sometimes it was accurate, sometimes it wasn’t, feeling like a “slot machine” [00:03:48]. This highlights the non-determinism of inputs and outputs, which remains a characteristic of building with AI [00:03:59]. Present-day Google Lens demonstrates mind-blowing capabilities, achieved through consistent, step-by-step iteration over a decade [00:04:07].

Looking back further to 2012, early breakthroughs like Google Brain identifying cat videos on YouTube involved models with about a billion parameters, a significant achievement then compared to today’s frontier models of a trillion parameters [00:05:21].

Challenges in Utilizing Large Language Models

The core challenge in building with large language models (LLMs) is that they are like “building on top of a foundation of jello” [00:11:34]. Unlike traditional software which is deterministic, fast, cheap, rigid, and governed by ‘if statements,’ LLMs possess distinct characteristics:

• Non-deterministic: Outputs can vary even with the same inputs [00:11:51].
• Slow and Expensive: Running LLMs can be computationally intensive and costly [00:11:53].
• Flexible and Creative: They can reason through problems [00:11:55].
• Unpredictable: LLMs remind us of ourselves in their unpredictability [00:16:51].
• Limitations: They can be slow and “not that great at math” [00:16:56].

The rapid pace of advancement in the AI space means that models are constantly being upgraded, and new paradigms like reasoning models and multimodality emerge, requiring continuous adaptation [00:14:52].

One specific area of [[challenges_in_building_ai_voice_agents | challenges in building AI voice agents]] involves handling latency, tone, and phrasing in multimodal models [00:10:17]. Building robust voice agents goes beyond simply wiring components together; it requires considering how a human would perform under similar conditions (e.g., receiving transcribed text with delay and having to respond immediately) [00:17:34].

Solutions: Sierra’s Agent Development Life Cycle (ADLC)

To address these challenges, Sierra developed the Agent Development Life Cycle (ADLC), a process for building and improving AI agents that borrows from the traditional Software Development Life Cycle (SDLC) but invents new concepts where necessary [00:11:26].

Core Principles of ADLC

• Agents as Products: Sierra believes every agent is a product, requiring a fully featured developer platform and customer experience operations platform [00:09:04]. This allows for continuous improvement similar to mobile apps or websites [00:09:19].
• Iterative Refinement: The ADLC emphasizes iterative refinement with customers in production to maximize productivity and robustness [00:12:35].

Key Components of ADLC

1. Quality Assurance (QA):

   • Experience Manager: Customers have access to Sierra’s Experience Manager, allowing them to review every conversation and high-level performance reports of the agent in real-time [00:12:45].
   • Feedback Loop: Users can provide feedback on issues (e.g., incorrect inventory information), which leads to an issue being filed [00:13:13].
   • Test Creation and Release: Each issue leads to a test being created. Once the test passes, a new release can be deployed [00:13:18]. This process allows agents to accumulate hundreds, then thousands, of tests over time, leading to significant improvement [00:13:27].
   • Opportunities for Delight: Beyond fixing mistakes, the system identifies opportunities for agents to go “above and beyond” to delight customers [00:13:36].

2. AI in the Life Cycle: While initially manual, Sierra is now integrating AI into each part of the ADLC to speed up improvements [00:14:00].

3. Scalability: The ADLC becomes more effective for larger customers, where velocity and change management are crucial for handling tens of millions of requests [00:14:26].

4. Leveraging Reasoning Models: Reasoning models act as a “force multiplier” within the ADLC, making AI application more effective in development, testing, and QA [00:15:05].

Responsive AI Agents and Empathy in Design

Sierra’s approach to voice agents is analogous to responsive web design, where a single underlying platform and agent code can adapt to different channels (chat, voice) and modalities [00:16:07]. This allows for channel-specific customization (e.g., phrasing, parallelizing requests for lower latency) while maintaining core functionality [00:16:27].

A key aspect of building with AI is developing empathy for large language models, understanding their unpredictable, slow, and mathematically challenged nature [00:16:48]. This empathy allows designers to better understand what it means to build a good experience for these agents [00:17:03]. The goal is to create robust and rich experiences by providing LLMs with similar inputs and experiences that humans have [00:18:01].

Case Studies

• Chubbies: Partnered with Sierra to deploy an AI agent named Duncan Smothers. Duncan helps customers with sizing, inventory tracking, package tracking, and refunds, demonstrating the agent’s capability to take autonomous actions beyond just answering questions [00:07:42]. This resulted in helping more customers more quickly with higher satisfaction [00:08:56].
• SiriusXM: A large customer benefiting from Sierra’s voice capabilities, enabling them to answer customer calls immediately [00:15:36].