Best practices for AI deployment and optimization

From: aidotengineer

This article outlines key insights and best practices for implementing AI, derived from hundreds of customer interactions at Anthropic. The information covers common mistakes and strategies for successful AI deployment, particularly focusing on large language models (LLMs) [01:14:00].

Anthropic’s Approach to AI Development

Anthropic is an AI safety and research company focused on building safe, large language models (LLMs) [01:26:00]. Their most recent model, Sonnet 3.5, is a leading model in the code space, performing well on evaluations like sbench for agentic coding evaluations [02:02:00].

A key differentiator for Anthropic is its focus on interpretability research, which involves reverse engineering models to understand how they “think” and why, and then steering them in desired directions [02:34:00]. This research is conducted in stages:

• Understanding Grasping AI decision-making [03:07:00].
• Detection Identifying specific behaviors and labeling them [03:10:00].
• Steering Influencing AI input (e.g., Golden Gate Claude example) [03:15:00].
• Explainability Unlocking business value from interpretability methods [03:22:00].

This research aims to improve AI safety, reliability, and usability [03:31:00].

Solving Business Problems with AI

When considering AI implementation, organizations should focus on how AI can solve core product problems, moving beyond simple chatbots and summarization [05:17:00].

Examples of Transformative AI Use Cases

Instead of basic Q&A, consider:

• Hyper-personalization Dynamically adapting course content based on individual employee context [06:18:00].
• Adaptive Learning Adjusting content difficulty dynamically if a user is breezing through material [06:26:00].
• Dynamic Content Generation Updating course material based on learning styles (e.g., creating visual content for visual learners) [06:33:00].

Companies are using AI to enhance customer experience, making products easier to use and more trustworthy, especially in critical industries like taxes, legal, and project management where hallucinations are unacceptable [07:14:00].

Anthropic’s Customer Support Model

Anthropic’s Applied AI team focuses on technical aspects of use cases, helping customers design architectures, perform evaluations, and tweak prompts to optimize model performance [09:14:00]. They also feed customer insights back into product development [09:23:00].

Their approach involves:

1. Sprints Kicking off focused Sprints when customers face niche challenges (e.g., LLM Ops, architectures, evals) [10:17:00].
2. Defining Metrics Helping customers define specific metrics for evaluating the model against their use case [10:26:00].
3. Deployment Supporting the deployment of iterative improvements into AB test environments and eventually production [10:33:00].

Case Study: Intercom’s Finn AI Agent

Intercom, an AI customer service platform, partnered with Anthropic to enhance their AI agent, Finn [10:55:00].

• Initial Sprint A two-week Sprint by the Applied AI team and Intercom’s data science team compared Finn’s hardest prompt against a Claude-optimized prompt, showing promising results [11:25:00].
• Extended Optimization This led to a two-month Sprint focused on fine-tuning and optimizing all of Intercom’s prompts for Claude [11:43:00].
• Results Anthropic’s model outperformed Intercom’s previous LLM, leading to the launch of Finn 2 [11:57:00]. Finn 2 can solve up to 86% of customer support volume (51% out-of-the-box), offers more human elements like tone adjustment and answer length, and provides strong policy awareness (e.g., refund policies) [12:20:00].

Best Practices for AI Deployment and Optimization

1. Testing and Evaluation

Evaluations are crucial and should guide the development process from the outset, not be an afterthought [13:36:00].

Common Mistakes:

• Building workflows first, then evaluations This leads to inefficient development, as evaluations are meant to direct towards the desired outcome [13:28:00].
• Struggling with data problems for evaluations LLMs like Claude can be used for data cleaning and reconciliation to design effective evaluations [13:50:00].
• “Trusting the vibes” Relying on a few queries without representative or statistically significant samples can lead to unforeseen outliers in production [13:59:00].

Best Practices:

• Empirical Knowledge The only way to truly understand model performance after changes (e.g., prompt engineering) is through empirical evaluations [15:02:00].
• Evaluations as IP Treat evaluations as core intellectual property, enabling faster navigation of the “latent space” of possible model behaviors to find optimal solutions [15:14:00].
• Set up Telemetry Invest in telemetry for back-testing architectures [15:35:00].
• Design Representative Test Cases Include diverse examples, even “silly” ones, to ensure the model responds appropriately or reroutes questions, reflecting real-world user behavior [15:43:00].

2. Identifying Metrics for AI Model Deployment

Organizations typically optimize for one or two elements of the “intelligence-cost-latency triangle” (intelligence, cost, latency), as achieving all three simultaneously is difficult [16:16:00].

Considerations:

• Define Balance in Advance Clearly define the trade-offs for your specific use case [16:32:00].
• Time Sensitivity
  • Customer Support: Latency is critical (e.g., customer expects response within 10 seconds) [16:40:00].
  • Financial Research: Accuracy and depth are more important than speed, so a 10-minute response time might be acceptable [16:55:00].
• Stakes of Decision The importance and time-sensitivity of the decision driven by the AI output should influence your optimization choices [17:10:00].
• User Experience (UX) Consider UX improvements like a “thinking box” or redirection to other pages to manage latency expectations [17:21:00].

3. Fine-Tuning

Fine-tuning is not a silver bullet and comes with costs and limitations [17:58:00].

Common Mistakes:

• Viewing fine-tuning as a quick fix It’s like “brain surgery” on the model, potentially limiting its reasoning in areas outside the fine-tuned domain [18:06:00].
• Attempting fine-tuning without clear success criteria or evaluation sets This makes it difficult to justify the effort and cost [18:19:00].

Best Practices:

• Explore Other Approaches First Prioritize other optimization methods before resorting to fine-tuning [18:16:00].
• Clear Success Criteria Have predefined success criteria, only considering fine-tuning if those cannot be met through other means [18:24:00].
• Don’t Let it Slow You Down Pursue the use case, and if fine-tuning becomes necessary, integrate it later rather than waiting for it from the start [18:56:00].
• Justify the Cost Ensure the expected performance difference justifies the effort and cost of fine-tuning [18:43:00].

4. Alternative Methods for Optimization

Beyond basic prompt engineering, various features and architectures can significantly improve use case success and help with scaling AI solutions in production [19:30:00].

Examples:

• Prompt Caching Can drastically reduce cost (e.g., 90% reduction) and increase speed (e.g., 50% increase) without sacrificing intelligence [19:47:00].
• Contextual Retrieval Improves the effectiveness of retrieval mechanisms, allowing information to be fed to the model more efficiently, reducing processing time [19:54:00].
• Citations An out-of-the-box feature that enhances reliability [20:08:00].
• Agentic Architectures Architectural decisions that can lead to significant improvements in AI agent development [20:11:00].