From: aidotengineer
Many teams approach their AI strategy like ordering takeout, choosing something that looks good online and paying a premium, only to find it doesn’t meet expectations due to overly high expectations [00:00:01]. While social media often showcases “bleeding edge models” suitable for millions of customers, this can be overkill and expensive for internal company needs, akin to “paying for truffles to garnish your instant noodles” [00:00:27]. A different, more practical recipe, which delivered millions of dollars in revenue, focuses on strategic in-house development and user-centricity [00:00:42].
Build vs. Buy: The Core Dilemma
In Q1 2024, a team faced the classic “build or buy” dilemma for an AI solution [00:00:54]. They opted to build in-house with two developers and roughly 10+ sprint weeks of effort, resulting in several million dollars of Annual Recurring Revenue (ARR) and a group-level award [00:01:05]. Approaches featuring “giant evals, multi-agent systems, RF models” — often seen as cutting-edge — are costly and delay launch, making them ideal for flashy demos but overkill for internal needs [00:01:30].
Software as a Service (SaaS) is comparable to a hotel buffet: generic but safe [00:01:59]. However, internal users often desire a customized solution, or “grandma’s secret sauce,” over generic options [00:02:04].
When to Buy SaaS:
- SaaS solutions excel when you require vendor integrations or access to cross-industry best practices [00:02:16].
When to Build In-House:
- Building in-house is advantageous when your organization already owns the data [00:02:24].
- Your colleagues possess specific knowledge about workflows and can be involved in double-checking outputs [00:02:32].
- Proximity to users allows for rapid tweaks that can ship the same day [00:02:42].
- User interfaces can be tailored to speak their language [00:02:50].
- Leveraging existing infrastructure reduces costs significantly, sometimes to pennies [00:02:53].
The rule of thumb is to buy SaaS when exploring the unknown but build in-house once the workflow is established and owned internally [00:03:02].
Five Lessons for Effective AI Implementation
Effective AI implementation is built upon five interconnected lessons [00:03:19]:
1. Focus on One Painful Job
In-house development shines by allowing deep dives into a single, painful “job to be done” without chasing a broad “total addressable market” [00:03:29].
- Identify Value Event: Select a task where the dollar-based outcome, or “value event,” is easily pinpointed [00:03:42].
- User-Centricity: Start with a simple use case, like sales alerts, and expand from there [00:03:54]. Engage users to understand their needs thoroughly and keep the solution simple, avoiding complex multi-agent systems [00:04:19].
2. Revenue Impact Trumps Evaluation Metrics
While offline evaluations are important as “smoke alarms,” board meetings prioritize revenue impact over F1 scores or NDCG [00:04:36].
- Track Money: Instrument everything to directly link AI tasks to specific dollar values [00:04:59].
- Build Revenue Funnels: Create a complete revenue funnel from beginning to end, leading back to the identified value event [00:05:07].
- User as Guardrails: Users serve as guardrails, enabling ambitious experiments without overthinking evaluations [00:05:21].
- Streamlined Decisions: Linking the system to financial outcomes simplifies decision-making and prioritization [00:05:31].
- Automate Reporting: Automate team performance reports and create leaderboards to foster healthy competition, gain leadership investment, identify champions, and support struggling team members [00:05:45].
3. Push Insights Proactively
Once revenue is being tracked, become a “chef who anticipates the next dish” rather than waiting for users to come with requests [00:06:11].
- Anticipate Needs: The best UI is one users never need to use; leverage business understanding to proactively implement solutions [00:06:19].
- Daily Digests: For example, building a system to send daily digests of critical information can serve user needs before they even ask [00:06:37]. A shared UI can then serve as a fallback for unexpected tasks [00:06:45].
4. Guide Action, Not Just Deliver Information
Proactive systems should guide action, not merely deliver information [00:07:05].
- Convert Time Saved to Time Well Spent: Saving time is worthless if it’s filled with unproductive tasks [00:07:14]. The true power lies in converting saved time into high-value activities by understanding the highest value actions for users and diverting their attention there [00:07:24].
- Improved Engagement: A proactive system that surfaces unforeseen opportunities can significantly boost Net Promoter Score (NPS) and engagement compared to chat-based applications [00:07:43].
5. Invest in the Basics: Good Data Beats Great Models
When deciding where to invest limited development resources, “good data consistently beats great models” [00:08:11].
- Cost vs. Impact: Larger models like GPT-3 can be significantly more expensive and slower than smaller alternatives like GPT-4 mini, with the biggest impact in production often being on cost [00:08:22].
- Focus on Triggers and Depth: Better results often come from simply adding more triggers to alert users and going deeper into their specific needs, even if it seems “boring” [00:08:34]. Model changes may only affect costs and evaluations, not user value [00:08:45].
- User Value Over Benchmarks: Build for what users need, not what you want to experiment with [00:08:55]. Focusing on true user value instead of chasing model benchmarks initiates a “powerful flywheel” [00:09:07].
The Revenue Flywheel
Focusing on user value creates tight feedback loops, making users feel heard and prompting them to provide improvement ideas [00:09:17]. This feedback enables rapid experiments, driving more adoption, generating more data for prioritization, and leading to more ideas [00:09:27]. This accelerates the “revenue flywheel,” leading to continuous improvement and growth [00:09:38].
Key Takeaways for Effective AI Implementation
To effectively implement AI solutions, start small and follow these principles:
- Focus on one painful job: Prioritize a specific problem with clear dollar value, avoiding comprehensive “boil the ocean” solutions [00:09:52].
- Revenue impact over evaluation metrics: Track everything to the final dollar-based outcome and base decisions on this financial impact [00:10:01].
- Push insights proactively: Deliver information and guidance before users have to ask for it [00:10:09].
- Guide action effectively: Ensure time savings are channeled into the highest value activities [00:10:20].
- Invest in the basics: Prioritize good data and user needs over chasing the latest, most expensive models [00:10:29].
In summary, start small, follow the money, and let your users guide your AI strategy [00:10:38].