From: aidotengineer
Many teams approach their AI strategy by treating it like ordering takeout: picking something appealing online and paying a premium, only to find it doesn’t meet expectations [00:00:01]. This often stems from overly high expectations influenced by social media showcases of “bleeding edge models” [00:00:14]. While such advanced models might be suitable for millions of customers, they can be an expensive overkill for internal company needs [00:00:29].
A Different Recipe: Building In-House
A different approach, focusing on in-house development, has delivered millions of dollars in revenue [00:00:39]. For example, in Q1 2024, a team faced the classic “build or buy” dilemma and chose to build [00:00:54]. With just two developers and approximately 10 sprint weeks of effort, they created a system that generated several million dollars in Annual Recurring Revenue (ARR) and earned a group-level award [00:01:05]. This success demonstrated that focusing on complex features like “giant evals, multi-agent systems, or RF models” (often seen on social media) would have been detrimental, leading to high costs and delayed launches [00:01:27]. These flashy solutions are often perfect for SaaS demos but overkill for internal needs [00:01:43].
The Build vs. Buy Decision
When “Buy” (SaaS) Shines
Buying a Software as a Service (SaaS) solution can be compared to a “hotel buffet” – generic but safe [00:01:59]. It is advantageous when:
- You require vendor integrations [00:02:16].
- You need cross-industry best practices [00:02:21].
When “Build” (In-House) Wins
Building in-house, like preparing a “home kitchen” meal, is superior when:
- You already own the data [00:02:27].
- Your colleagues possess precise knowledge of workflows (e.g., exact keystrokes to close deals) [00:02:31].
- You can involve internal users in double-checking outputs [00:02:36].
- Proximity to users allows for same-day tweaks and UI that speaks their language [00:02:42].
- The system can run on existing infrastructure, significantly lowering costs [00:02:53].
The general rule of thumb is to “buy SaaS to explore the unknown, but build in-house once the workflow is yours” [00:03:02].
Five Lessons for Successful In-House AI Development
These lessons work best when applied as a set, starting with the foundation:
1. Focus on One Painful Job to Be Done (JTD)
In-house development excels at going “absurdly deep on one painful job to be done” without needing to chase a large total addressable market [00:03:29].
- Identify a Value Event: Pick something where you can easily pinpoint a clear, dollar-based outcome [00:03:42].
- Start Simple: For instance, begin with a simple sales alert use case and expand from there [00:03:54].
- Engage Users: Your users know best what’s needed; talk to them to truly nail the solution [00:04:19].
- Keep it Simple: Staying focused allows you to avoid complex, “agentic” systems [00:04:25]. This also helps with challenges in building AI applications.
2. Track Revenue Impact (Not Just Evaluation Metrics)
Offline evaluations (evals) are important as “smoke alarms,” but they don’t sign contracts [00:04:36].
- Focus on Dollars: Board meetings ask about revenue impact, not F1 scores or NDCG [00:04:40].
- Instrument Everything: Ensure you can trace an AI task’s contribution directly to revenue [00:04:59].
- Build Revenue Funnels: Map the entire process from beginning to end to the defined “value event” [00:05:07].
- Users as Guard Rails: Your users can serve as guard rails for ambitious experiments, reducing the need to overthink evaluations [00:05:18].
- Simplify Decisions: Linking your system to dollar impact streamlines decision-making and prioritization [00:05:31].
- Automate Reports and Leaderboards: Automate team performance reports and create leaderboards to foster healthy competition, engage leadership, identify champions, and support struggling users [00:05:52].
3. Be Proactive, Anticipate User Needs
Don’t wait for users to come to you; become the “chef who anticipates what the next dish should be” [00:06:11].
- The Best UI is Invisible: The most effective user interface is one users never need to use [00:06:21].
- Automate Tasks: Since you understand the business, proactively take the next steps for users rather than waiting for requests [00:06:28].
- Example: Building a motion to send daily digests (“Here is what you need to know today”) [00:06:37].
- A traditional UI can still exist as a fallback for unexpected tasks [00:06:45].
4. Guide Action, Don’t Just Deliver Information
Your AI system should guide action, not merely provide information [00:07:05].
- Convert Time Saved to Time Well Spent: Saving 30 minutes is pointless if users fill it with unproductive tasks [00:07:14]. The true value comes from converting saved time into engagement with high-value activities [00:07:23].
- Leverage Revenue Funnels: As you build revenue funnels, you gain insights into where to direct freed-up time and user attention [00:07:34].
- Proactive System Success: Proactive systems that surface unthought-of actions can achieve significantly higher NPS (20 points higher) and engagement than chat-based applications [00:07:43]. This highlights a successful AI-enhanced organization approach.
5. Invest in Good Data, Not Just Great Models
This is a secret often not found on social media: good data consistently outperforms great models [00:08:11]. This is critical for AI development and scaling AI products.
- Cost vs. Performance: More advanced models (like O3) can be 60 times more expensive and an order of magnitude slower than simpler ones (like 41 mini), primarily impacting production costs [00:08:22].
- Focus on User Needs: The best results come from adding more triggers to alert users and delving deeper into their specific needs [00:08:34]. Changes in models often only affect costs and evaluations, not user value [00:08:45].
- The Revenue Flywheel: Focusing on true user value rather than chasing model benchmarks initiates a powerful feedback loop [00:09:03]:
- Tight feedback loops make users feel heard [00:09:17].
- Users provide ideas for improvements [00:09:20].
- This enables running experiments based on feedback [00:09:27].
- Which drives more adoption [00:09:29].
- Generating more data for prioritization and new ideas [00:09:32].
- Accelerating the “revenue flywheel” [00:09:38].
Key Takeaways
To successfully implement AI in-house:
- Focus Small: Concentrate on one painful job with a clear dollar value, avoiding comprehensive “boil the ocean” solutions [00:09:52].
- Follow the Money: Prioritize revenue impact over evaluation metrics, tracking everything to the final dollar-based outcome [00:10:01].
- Be Proactive: Push insights proactively instead of waiting for user questions [00:10:13].
- Guide Action: Ensure time savings are channeled into the highest-value activities [00:10:20].
- Invest in Basics: Prioritize good data and user needs over chasing complex models; it truly pays off [00:10:29].
In essence: Start small, follow the money, and let your users guide you [00:10:37].