From: aidotengineer
Sierra, a conversational AI platform for businesses, focuses on building and improving agents for various customer experience touchpoints, including sales, subscription management, and product recommendations [00:01:02]. Initially known for chat experiences and customer service, Sierra is expanding into phone interactions, which are projected to be the majority of interactions by the end of the year [00:00:53].
Evolution of AI Development
The rapid evolution of AI highlights the need for robust development processes. While AI development has roots in earlier decades, the recent surge in capabilities has occurred largely within the last decade [00:01:45]. Early examples of AI development, such as the speaker’s work on Google Lens in 2016, focused on tasks like distinguishing Chihuahuas from blueberry muffins or identifying plants [00:02:15]. This early stage often felt like a “slot machine,” with unpredictable results due to the non-determinism of inputs or outputs [00:03:48].
Today’s Google Lens, a result of consistent, step-by-step iteration over a decade, demonstrates advanced capabilities like visual shopping, language translation, and math homework assistance [00:04:07]. This progress underscores the importance of a structured process for iterative improvement, similar to the software development life cycle (SDLC) [00:04:52].
The Need for a Specialized Life Cycle
Traditional software is deterministic, fast, cheap, rigid, and governed by strict logic [00:11:43]. However, large language models (LLMs) pose unique challenges in AI agent development [00:11:32]:
- Non-deterministic: They can produce varied outputs for the same input [00:11:51].
- Slow and expensive: Running LLMs can incur significant time and cost [00:11:53].
- Flexible and creative: LLMs can reason through problems and exhibit flexibility [00:11:55].
Sierra developed the Agent Development Life Cycle (ADLC) to leverage LLMs’ strengths while integrating traditional software where beneficial [00:12:02].
Sierra’s Agent Development Life Cycle (ADLC)
The ADLC is Sierra’s process for building and improving AI agents [00:12:16]. It is designed for iterative refinement with customers in production to ensure productivity and robustness [00:12:35].
Key Components of the ADLC
-
Quality Assurance (QA):
- Experience Manager: Customers have access to Sierra’s Experience Manager, which allows them to view every conversation an agent has and high-level performance reports in real time [00:12:49].
- Feedback Mechanism: If an agent like Duncan Smothers provides incorrect information (e.g., inventory), users can report the issue [00:13:05].
- Issue to Test: Reported issues lead to the filing of a problem and the creation of a new test [00:13:15].
- Continuous Improvement: Once a test passes, a new release can be made, leading to an agent evolving from a handful of tests at launch to hundreds and then thousands over time [00:13:23].
-
Beyond Correction: The ADLC also enables agents to go “above and beyond” [00:13:37]. For instance, Chubbies agents have a budget to delight customers, potentially allowing an agent to arrange for shorts to be DoorDashed from a retail location if unavailable online [00:13:42].
-
AI-Enhanced Development: A year ago, these processes were largely manual [00:14:00]. With advancements in AI, Sierra is now able to add AI to each part of the ADLC, accelerating improvements [00:14:13].
-
Scalability: The ADLC becomes more effective with larger customers [00:14:26]. While an agent like Duncan might handle hundreds of thousands of requests, other customers manage tens of millions, making velocity and change management incredibly valuable [00:14:28].
-
Adapting to Industry Changes: The fast-moving AI space, with model upgrades, new paradigms like reasoning models, and multimodality, constantly impacts the ADLC [00:14:48]. Reasoning models, in particular, act as a force multiplier, allowing for more effective application of AI to development, testing, and QA [00:15:06].
Case Study: Chubbies and Duncan Smothers
Chubbies partnered with Sierra to create Duncan Smothers, an AI agent representing their business [00:07:37]. Duncan Smothers is capable, empathetic, and engaging, assisting with a variety of customer cases on the Chubbies website [00:07:55]:
- Sizing and Fit: Empathetically helps customers with questions, asks for waist size, and offers product recommendations [00:08:11].
- Inventory Tracking: Informs customers about stock availability and helps them choose new items [00:08:27].
- Package Tracking and Refunds: Provides multiple tracking numbers and can issue refunds, demonstrating autonomous actions [00:08:36].
These capabilities allow Chubbies to help more customers more quickly and with higher satisfaction [00:08:58].
Building and Recruiting AI Teams
Sierra employs dedicated agent engineering and agent product management functions that work closely with customers like Chubbies [00:09:32]. The importance of finding talent for these roles is highlighted by the speaker’s experience meeting Shawn at the AI Engineering World’s Fair, which led to Shawn joining Sierra [00:10:25]. This demonstrates the value of serendipitous connections in building strong AI teams.
Voice Agents and Responsive Design
Sierra launched voice capabilities in October, allowing large customers like SiriusXM to answer phone calls immediately every time [00:15:31]. Sierra’s approach to voice is akin to responsive web design: the same underlying platform and agent code can adapt to various channels and modalities (e.g., chat, phone) [00:16:13]. While customization (different phrasing, parallel requests for lower latency) is possible, the core functionality works out of the box [00:16:27].
Designing with Empathy in AI
Building with AI is fascinating because LLMs, like humans, can be unpredictable, slow, and not great at math, yet they allow designers to have empathy in a new way [00:16:46]. By understanding the limitations and characteristics of LLMs (e.g., processing transcribed text with delay), developers can design more robust and effective experiences for AI agents [00:17:10]. This perspective helps in creating a richer and more robust AI product [00:18:01].