Leveraging large language models for customer insights

From: aidotengineer

Analyzing vast amounts of customer data, such as sales calls, support tickets, product reviews, user feedback, and social media interactions, has historically been a daunting and time-consuming task for businesses. This unstructured data often goes untouched, despite being a valuable source of insight [09:11:00]. However, modern large language models (LLMs) are transforming this challenge into an opportunity, enabling single engineers to perform analyses that previously required dedicated teams [00:55:00].

The Challenge of Manual Analysis

Manually analyzing a large database of customer interactions, such as 10,000 sales call transcripts, is practically impossible for humans. Even with extreme dedication, it would take nearly two years (625 days) of continuous work to read, take notes, identify insights, and compile reports [02:12:00]. The human brain is simply not equipped to process such a massive volume of information [02:22:00].

Traditional approaches before LLMs typically fell into two categories [02:34:00]:

• Manual analysis: High quality but completely unscalable [02:38:00].
• Keyword analysis: Fast and cheap, but often missed context and nuance [02:44:00].

This gap created a need for a solution that could handle both scale and depth.

LLMs: The Sweet Spot for Unstructured Data

The intersection of unstructured data and pattern recognition is a “sweet spot” for AI projects [02:55:00]. Large language models excel at this, making it feasible to analyze vast datasets of customer interactions [02:50:00]. For example, a project aimed at analyzing 10,000 sales calls to refine an ideal customer profile (ICP) was accomplished by a single AI engineer in about a fortnight, a task that would have been impossible two years prior [00:46:00].

Choosing the Right Model

Selecting the appropriate LLM is crucial. While smaller, cheaper models might be tempting, they often produce an alarming number of false positives and high hallucination rates [03:26:00]. For a project where data trustworthiness is paramount, investing in more intelligent models like GPT-4o or Claude 3.5 Sonnet, despite their higher cost and slower speed, is essential to ensure acceptable accuracy [03:54:00]. Claude 3.5 Sonnet was ultimately chosen for its prompt caching capabilities and accuracy [07:34:00].

Minimizing Hallucinations and Ensuring Reliability

Achieving reliable results with LLMs requires a multi-layered approach beyond simply feeding in transcripts [04:16:00]:

• Data Enrichment: Raw transcript data is enriched via retrieval augmented generation (RAG) from both third-party and internal sources [04:27:00].
• Prompt Engineering: Techniques like chain of thought prompting are employed to elicit more reliable outputs [04:38:00].
• Structured Outputs with Citations: Generating structured JSON outputs and ensuring a verifiable trail back to the original transcripts allows for confidence in the final results and extraction of accurate company details and meaningful insights [04:46:00].

Cost Optimization

Maintaining low error rates and high accuracy can drive up costs significantly due to token limits and multiple requests per analysis [05:10:00]. Experimental features can dramatically lower these costs:

• Prompt Caching: Reusing the same transcript content for repeated analysis (e.g., metadata and insight extraction) can reduce costs by up to 90% and latency by up to 85% [05:33:00].
• Extended Outputs: Accessing experimental features that double the original output context allows for complete summaries in single passes, avoiding multiple rounds of credit usage [05:53:00]. These optimizations can turn a $5,000analysisintoa$500 one, yielding results in days instead of weeks [06:14:00].

Wider Organizational Impact and Key Takeaways

What begins as a project for an executive team can have a wide-ranging impact across an organization [06:30:00]. For example:

• Marketing teams can easily pull customer data for branding and positioning exercises [06:47:00].
• Sales teams can automate transcript downloads, saving dozens of hours weekly [06:54:00].
• The ability to easily perform analysis encourages teams to ask questions that were previously too daunting to consider [07:03:00].

This transformation turns mountains of unstructured data from a liability into an asset [07:13:00].

Key Learnings:

1. Models Matter: Despite the push for open-source and cheaper models, more powerful LLMs like Claude 3.5 and GPT-4o are often necessary for complex tasks that smaller models simply cannot handle [07:22:00]. The right tool is the one that best fits specific needs [07:41:00].
2. Good Engineering Still Matters: Significant gains come from fundamental software engineering principles, such as leveraging JSON structured output, good database schemas, and proper system architecture [07:48:00]. AI engineering involves building effective systems around LLMs, ensuring they are thoughtfully integrated into existing systems rather than being an afterthought [08:03:00].
3. Consider Additional Use Cases: Don’t stop at a single report. Building a user experience (UX) around AI analysis, with features like search filters and exports, can transform a one-off project into a company-wide resource [08:21:00].

Ultimately, LLMs can transform seemingly impossible tasks into routine operations [08:42:00]. It’s not about replacing human analysis, but augmenting it and removing human bottlenecks, thereby unlocking entirely new possibilities [08:50:00]. The tools and techniques are available today to turn customer data into gold [09:29:00].