Using AI to analyze sales calls

From: aidotengineer

Analyzing large volumes of sales call data manually is an almost impossible task for humans, often taking years and requiring extensive resources [00:00:06]. For example, analyzing 10,000 sales calls would take approximately 625 days of continuous work, equivalent to nearly two years [00:02:14]. The human brain is not equipped to process such vast amounts of information [00:02:24].

Traditional approaches to analyzing sales calls included manual, high-quality but unscalable methods, or fast, cheap keyword analyses that lacked context and nuance [00:02:34]. However, modern large language models (LLMs) offer a solution for analyzing unstructured data and recognizing patterns [00:02:50]. What once required a dedicated team working for weeks, or was considered impossible, can now be accomplished by a single AI engineer in about two weeks [00:00:48].

The Challenge

A specific goal was set to analyze 10,000 sales calls within two weeks to perform a comprehensive analysis of the ideal customer profile (ICP) for Pulley, a company whose ICP was previously defined broadly as venture-backed startups [00:00:35]. To refine this, the aim was to identify specific personas, such as “CTO of an early-stage venture-backed crypto startup” [00:01:02].

The challenge lay in the sheer volume of data, consisting of thousands of hours of sales representatives speaking directly with customers [00:01:30]. A manual analysis would involve:

• Downloading and reading each transcript [00:01:47].
• Determining if the conversation matched the target persona [00:01:53].
• Scanning hundreds or thousands of lines for key insights [00:01:58].
• Remembering information while compiling notes, reports, and citations [00:02:03].
• Repeating this process 10,000 times [00:02:12].

Technical Implementation

While seemingly simple (“just use AI to analyze sales calls”), this project required addressing several interconnected technical challenges [00:03:02].

Model Selection

The first critical decision was choosing the right LLM [00:03:12]. GPT-4o and Claude 3.5 Sonnet were identified as the most intelligent options available, despite being the most expensive and slowest [00:03:17]. Experiments with smaller, cheaper models quickly revealed their limitations, as they produced an alarming number of false positives [00:03:26]. For instance, they might incorrectly classify a transcript as crypto-related due to a brief mention of blockchain features, or misidentify a prospect as a founder without supporting evidence [00:03:37]. Ultimately, Claude 3.5 Sonnet was chosen because its hallucination rate was acceptable, ensuring data reliability [00:04:01].

Reducing Hallucinations

A multi-layered approach was developed to reduce hallucinations and ensure reliable results [00:04:20]:

1. Data Enrichment: Raw transcript data was enriched using Retrieval Augmented Generation (RAG) from both third-party and internal sources [00:04:27].
2. Prompt Engineering: Techniques like chain-of-thought prompting were employed to guide the model towards more reliable outputs [00:04:38].
3. Structured Outputs: Generating structured JSON outputs where possible allowed for the creation of verifiable citations [00:04:46].

This combined approach created a system that could reliably extract accurate company details and meaningful insights, with a verifiable trail back to the original transcripts, ensuring confidence in the final results [00:04:54].

Cost Optimization

While effective, the extensive analysis and low error rate significantly drove up costs, often hitting the 4,000-token output limit of Claude 3.5 Sonnet, requiring multiple requests per transcript [00:05:10]. Two experimental features were leveraged to dramatically reduce expenses:

1. Prompt Caching: By caching transcript content, which was often reused for extracting metadata and insights, costs were reduced by up to 90% and latency by up to 85% [00:05:31].
2. Extended Outputs: An experimental feature flag in Claude allowed access to double the original output context. This enabled the generation of complete summaries in single passes, avoiding multiple turns and reducing credit consumption [00:05:51].

These optimizations transformed a $5,000analysisintoa$500 one, delivering results in days instead of weeks [00:06:14].

Impact and Key Takeaways

The project’s most surprising aspect was its wide-ranging impact across the organization [00:06:30]. What began as an executive-level project to generate insights became useful across multiple departments:

• The marketing team could easily identify customers for branding and positioning exercises [00:06:47].
• The sales team automated transcript downloads, saving dozens of hours weekly [00:06:54].
• Teams began asking questions that were previously considered too daunting for manual analysis [00:07:03].

Ultimately, mountains of unstructured data were transformed from a liability into a valuable asset [00:07:13].

Key Takeaways:

1. Models Matter: Despite the push for open-source and cheaper models, powerful LLMs like Claude 3.5 and GPT-4o demonstrated superior capabilities for complex tasks [00:07:22]. The right tool is the one that best fits specific needs, not always the most powerful [00:07:38].
2. Good Engineering Still Matters: Significant gains were achieved through solid software engineering practices, including leveraging JSON structured output, good database schemas, and proper system architecture [00:07:48]. AI engineering involves building effective systems around LLMs, ensuring AI is thoughtfully integrated, not merely bolted on [00:08:04].
3. Consider Additional Use Cases: The project evolved beyond a single report by building an entire user experience around the AI analysis, including search filters and exports [00:08:21]. This transformed a one-off project into a company-wide resource [00:08:36].

This project demonstrates how AI can transform seemingly impossible tasks into routine operations [00:08:42]. It’s not about replacing human analysis but augmenting it and removing bottlenecks, thereby unlocking entirely new possibilities [00:08:50]. Valuable customer data such as sales calls, support tickets, product reviews, user feedback, and social media interactions, often overlooked, are now highly accessible via large language models, offering significant insights [00:09:11].