From: redpointai
The development of AI models for enterprise applications involves strategic decisions regarding how models are adapted and utilized. Chris Roman, co-founder and CEO of Fireflies.ai, shared insights on their approach to leveraging large language models (LLMs), particularly their perspective on finetuning.
The Debate: To Finetune or Not to Finetune?
Fireflies.ai generally avoids finetuning models on customer data, prioritizing privacy by not training on it by default [00:27:27] [00:30:22]. Chris Roman expressed a lack of belief in finetuning, citing several reasons:
- Cost and Diminishing Returns Finetuning is expensive, and its returns diminish over time as the base models themselves improve significantly [00:44:12] [00:46:04]. The rapid evolution of models, such as the jump from GPT-3 to GPT-4, makes earlier finetuning efforts less relevant [00:44:20] [00:45:06]. A non-finetuned GPT-5 might outperform a finetuned GPT-4 [01:04:09].
- Market Volatility The AI market changes weekly, requiring constant adaptation of assumptions and strategies [00:44:50] [00:45:01]. Finetuning slows down development and flexibility [00:45:13].
- Viability of Building Own LLMs Companies attempting to build their own LLMs from scratch for end applications often struggle due to immense costs, lack of traction, and the need for expensive AI engineers and data [00:48:42] [00:48:48]. The focus should be on riding the technology wave provided by general-purpose LLMs [00:48:52].
Alternatives to Finetuning
Instead of finetuning, Fireflies.ai emphasizes:
- Prompt Engineering: Using precise and constrained prompt engineering to guide the AI’s output, ensuring it stays within the given information and doesn’t “get too creative” [00:45:55] [00:46:02].
- Contextualization: Leveraging context from meetings to improve AI performance [00:45:55].
- Dynamic Model Selection: Utilizing different LLM vendors and models for specific tasks where they excel (e.g., one model for summary overviews, another for shorthand notes, another for action items) [00:46:47] [00:47:09]. This requires flexibility [00:47:10].
- Continuous Experimentation: Building an in-house A/B experimentation platform to roll out different models, measure performance, and gather customer ratings for responses. This allows for continuous optimization based on user feedback [00:46:11] [00:46:16] [00:46:25].
Challenges in AI Model Development and Evaluation
AI models, despite their advancements, present challenges in consistency and reliability.
- Consistency: Newer models may produce different answers for the same input, posing a challenge in controlling for variance [00:42:20] [00:42:36].
- Evaluation: While internal “eyeball tests” can give an initial sense, the ultimate judge of model quality should be the customer [00:46:30] [00:46:35]. A large user base allows for quick, strong signals on model performance [00:46:37].
Strategic Considerations for AI Application Developers
For application-layer companies, the focus should shift away from foundational model development:
- Solving End-to-End Problems: The most defensible moat against large incumbents and basic LLM capabilities is solving an end-to-end customer problem deeply within their workflow [00:49:17] [00:49:52]. This includes integrating with downstream systems like Salesforce, Asana, and Slack [00:41:40] [00:42:40].
- Leveraging Cost Reduction: The decreasing cost of transcription, increased adoption of video conferencing, and the falling cost of AI intelligence are transformative factors [00:49:29] [00:51:00].
- Pricing Strategy: A hybrid pricing model, combining seat-based pricing for core value with utility-based pricing for complex, high-compute tasks, can be effective [00:52:05] [00:53:05].
- Commoditization: Companies should be willing to be the first to commoditize features as base model capabilities improve, passing the benefits to users [00:54:39] [00:54:47].
- Focus on Value: It is crucial for founders to genuinely solve deep customer problems rather than chasing hype or valuations based solely on being an “AI company” [00:55:51] [00:56:19].
Personalization and Steerability of AI Models
AI applications can be personalized without direct finetuning:
- User Profiles: Users can inform the AI about their role (e.g., “I am a person in Pharma”) to receive tailored insights and recommendations from the same general model [01:03:39] [01:03:52].
- AI for Recommendations: AI is becoming incredibly adept at recommending relevant information and actions, even from smaller datasets, similar to how large companies use recommendation algorithms [01:04:47] [01:05:00].
Future Outlook: Multimodal AI and Agentic Collaboration
The future of AI involves more intelligent and integrated systems:
- Increased Intelligence: Future models (e.g., GPT-5) are expected to reach intelligence levels comparable to a PhD, enabling more sophisticated actions and recommendations [00:47:02] [00:47:08].
- Multimodality: Models will process and act upon various types of data, including voice, screen recognition, and external data sources, leading to capabilities like real-time background checks or research during conversations [00:47:42] [00:48:10].
- Agentic Future: An “agentic future” envisions multiple specialized AI agents collaborating (e.g., a meeting agent talking to a legal agent or a search agent to fact-check information) [00:49:06] [00:51:10].
- Horizontal AI with Customization: Instead of highly specialized vertical SaaS, the trend might shift towards general horizontal products that can be customized by users or through an ecosystem of AI apps (like app stores) [01:00:58] [01:01:40].
Overcoming Incumbent Competition
Competing with large incumbents like Microsoft, Google, and Zoom requires distinct strategies:
- Deep Execution: Startups must execute better and go deeper into specific workflows, especially for features that are merely “checklist items” for larger companies [00:53:22] [00:53:33].
- AI-First Mindset: Being an “AI-first” company allows startups to build products with a new perspective, free from legacy baggage and corporate bureaucracy [00:53:50] [00:54:02].
- Velocity: Startups can adapt and innovate faster due to their lean structure, crucial in a rapidly changing AI landscape [01:06:44].
Operationalizing AI at Scale
Beyond the AI models themselves, managing the underlying infrastructure is a significant challenge:
- Speed and Latency: Reducing processing time for meetings and notes directly correlates with increased user engagement and utility [00:59:16] [00:59:30].
- Infrastructure Management: Handling millions of meetings, adhering to strict rate limits (e.g., tokens per minute), and managing email sending volumes are massive infrastructural challenges [00:59:52] [01:00:48]. This involves breaking down monolithic codebases and optimizing each component [01:00:07] [01:00:26].
- Security and Trust: Handling sensitive conversational data at scale requires robust security measures [01:00:29]. Gaining customer trust is paramount, especially for startups competing with established incumbents [00:38:41].
Overall, the approach to AI development is shifting from deep technical customization of models (like finetuning) to strategic application development that leverages powerful, rapidly evolving base models, focusing on deep workflow integration, user experience, and efficient operations.