From: redpointai
Eric Ries, author of The Lean Startup, and Jeremy Howard are building Answer AI, aiming to create the “Bell Labs of AI” [00:00:01]. They focus on building smaller, cheaper, more affordable models and applications, particularly in the legal and education sectors [00:00:05]. Ries observes that while his Lean Startup lessons defined how many in tech approach building and innovating, the AI world sometimes presents different scenarios [00:00:50].
Unique Challenges in Applying Lean Startup to AI
Ries notes that AI makes unbelievably good demos, which can lead companies to believe they are the exception and don’t need to test with customers [00:01:32]. This often results in significant spending on models and compute long before a product touches the market [00:01:00].
Another challenge is the tendency to simply copy-paste the existing SAS (Software as a Service) stack to AI, assuming everything will be the same [00:02:23]. Many AI companies creating APIs push the product-market fit question down the “stack” to their customers, who then must define it with their customers [00:02:45]. This value chain can be two, three, or four layers deep between the model and the end product, potentially leading to “carnage” in applications if the stack is not assembled differently [00:03:05].
Ries highlights that the economics of AI are “completely different” from traditional software, drawing more parallels to physical manufacturing, deep-sea oil drilling, or nuclear power plants due to their infrastructure, operating costs, and market risk [00:03:41].
Core Lean Startup Principles Remain Essential
Despite the unique aspects of AI, fundamental Lean Startup principles still apply [00:01:51]:
- Customer-Centricity: “You can’t know in advance what customers are going to want” [00:01:56]. Instead of asking customers what they want, businesses should experiment and discover needs through “revealed actions” [00:02:12]. Eric Ries emphasizes understanding the “end, end, end customer” regardless of a company’s position in the stack [00:03:25]. As Peter Drucker stated, “a business is an entity that exists to create a customer,” and this is unchanged by AI; AI agents are not the customer, human beings are [00:04:15].
- Rapid Iteration and Pivoting: Given the high level of uncertainty in AI, especially regarding future model capabilities and consumer desires, it’s crucial to build in a way that allows for rapid iteration and the ability to pivot [00:07:39]. This means staying alert to the possibility that assumptions could be wrong [00:07:42]. The pace of change in AI makes quick adaptation and feedback more important than ever [00:08:27].
Moats and Defensibility in AI
Some entrepreneurs are “paralyzed into action” by concerns about “moats” and defensibility in the AI space [00:04:44]. Jeremy Howard suggests that the priority should be to build something customers want, and then “earn the right” to think about a moat [00:05:04].
Ries describes this as “picking up dimes in front of a steamroller” [00:05:20]. While large platforms like OpenAI could theoretically “nuke you from orbit,” they have limited focus [00:05:37]. The strategy is to quickly jump in, grab a use case (the “dime”), and jump out [00:05:51]. This requires speed and the ability to pivot if needed [00:05:51].
Answer AI’s R&D Lab Approach
Answer AI operates as a for-profit R&D lab, an unconventional model for venture-backed startups [00:14:35]. Eric Ries argues that the best research occurs when the researcher is “coupled to the application” [00:16:07]. This contrasts with the modern trend of hyper-specialization where research (R) and development (D) are separated, leading to research untethered from practical applications [00:15:35].
The Edison Approach
Answer AI aims to emulate Thomas Edison’s lab, where the iteration loop extends from the customer all the way into scientific inquiry and back [00:16:21].
For instance, in a corporate setting, Ries witnessed researchers focused on winning a Nobel Prize for energy efficiency in data centers, only to find through an MVP that customers (data center builders) cared more about physical footprint [00:16:54]. Bringing this customer feedback back to the lab immediately redirected research toward a relevant problem [00:17:58].
”Long Leash with Narrow Fences”
Inspired by historian Eric Gilliam, Answer AI adopts a “long leash with narrow fences” approach [00:25:24]. Jeremy Howard and Eric Ries establish “narrow fences” – a research thesis focusing on specific areas like efficiency and reasonable cost, countering overinvestment in large, expensive models [00:25:29]. Within these fences, team members have a “long leash” to pursue projects they believe in [00:24:43].
An example is Karam’s breakthrough in efficient fine-tuning of Llama 3 [00:24:10]. He independently pursued this project, driven by the team’s shared focus on reducing costs and increasing accessibility [00:24:43]. This focus on “the same thing but cheaper” is a major part of Answer AI’s mission to make AI more accessible, which is often dismissed as “tedious” by larger labs [00:26:59].
Efficiency and Accessibility as Breakthroughs
Ries argues that a “difference in degree becomes a difference in kind” [00:27:15]. Making inference costs cheaper doesn’t just improve margins; it enables new applications [00:27:23]. The software industry is now dealing with “actual supply chain constraints” like physical installation, power access, and HBM memory manufacturing limits [00:27:30].
Lower costs could enable continuous fine-tuning, allowing models to have “memory” and hyper-personalization, unlike current “amnesiac models” [00:28:01]. This opens up use cases requiring dedicated, customized models per customer [00:29:02].
Just like electricity’s practical applications were not always obvious but required “hundreds of thousands of individual experiments painstakingly done,” AI needs to focus on “manufacturability,” deployability, and usability [00:29:22].
Applications in Legal and Education
Answer AI specifically targets the legal and education sectors, which are heavily language-based and offer significant opportunities for societal improvement [00:34:43].
- Legal: The law is often used as a “weapon by wealthy people and organizations” [00:35:36]. Reducing the cost of high-quality legal advice can combat injustice and gatekeeping, enabling more people to pursue their ideas [00:36:14].
- Education: Jeremy Howard, a homeschooling dad, believes AI can improve education by removing constraints like standardized paths, allowing for more customized learning [00:37:06].
Both fields involve significant “language in, language out” [00:38:16].
Challenges and Strategies in Enterprise AI Deployment
Jeremy Howard expresses concern over proposed legislation like California’s SB147, which aims to ensure AI model safety through regulation [00:39:02]. His research, including interviews with over 70 experts, suggests such policies would be ineffective and could even lead to less safe situations [00:39:43].
The core issue is that AI models are “dual use technology,” like a pen, paper, or calculator [00:40:32]. A model deemed “safe” can still be fine-tuned or prompted to do unsafe things by users [00:40:55].
- Implications of Regulation: If safety must be “ensured,” it effectively means models cannot be released in their raw form [00:41:34]. Instead, only “products on top of them” can be released (e.g., ChatGPT versus downloadable Llama 3) [00:42:01].
- Centralization of Power: Raw models are “much more powerful” due to the ability to fine-tune, study weights, control, and cache [00:42:42]. Restricting their release makes them an “extremely rivalrous good,” accessible only to big states and companies, leading to “massive centralization of power” and reduced transparency [00:43:03]. This prevents widespread use for defensive purposes like cybersecurity or vaccine development [00:43:34].
- Safety through Accessibility: Eric Ries argues that focusing on unlocking the “unbelievable reservoir of applications that don’t require AGI” [00:45:14] leads to building intrinsically safe applications with smaller, properly fine-tuned models [00:46:15]. If these options aren’t available, people default to potentially unsafe uses of large frontier models [00:46:38].
Ries also points out that large foundation labs can become “schizophrenic,” with commercial teams disconnected from or uninterested in the safety agenda of the research teams, which can lead to internal “rival tensions” [00:47:08]. He suggests these labs re-establish the connection between research and the customer by affirmatively seeing their customers succeed and using the “Toyota production system Playbook” to “go and see for yourself” [00:48:06].
Overhyped and Underhyped in AI
- Overhyped: Agents [00:48:34]. Jeremy Howard believes current agent capabilities are not compatible with the mathematical foundations of language models, especially for novel planning [00:48:42]. Agents excel at “various mixes and matches of stuff that you’ve seen in the training data” (e.g., answering emails, adding CSS) but struggle with novel algorithms or research breakthroughs not in their training data [00:22:31].
- Underhyped: Resource efficiency [00:48:38].
Future Breakthroughs and Cognition
Jeremy Howard highlights two key breakthroughs that would change his mind on model capabilities:
- A significant breakthrough in energy or other resource requirements [00:51:03].
- A breakthrough in planning and reasoning capability that goes “past subgraph matching,” like Yean LeCun’s “Jeeper based models” or “diffusion models for text” [00:51:14]. Current auto-regressive models are limited by picking words sequentially [00:51:51].
Eric Ries notes that the most interesting aspect of LLMs is how they’ve changed the perception of human intelligence, suggesting “far more of human intelligence is obviously encoded in language and linguistic processing than we previously thought” [00:52:36]. He hopes for a breakthrough that reveals “the problem is not in our understanding of the scaling of LLMs, the problem is in our understanding of cognition itself in human cognition” [00:53:20]. He uses the analogy of a calculator built with Minecraft blocks: the current approach might be a brute-force, inefficient way of discovering a “critical algorithm in cognition” that could eventually be done directly [00:53:32].