From: redpointai
The “Unsupervised Learning AI Podcast,” hosted by Jacob Efron and Pat Chase, featured Omj, founder of Replit, to discuss the dynamic landscape of AI model development and utilization within the tech industry [00:00:00]. Replit, valued at over a billion dollars, is at the forefront of integrating AI into coding solutions [00:00:15].
AI in Coding Education and the Evolving Skillset
Omj emphasizes that the best way to learn coding is by “making things,” a principle that aligns with Codecademy’s approach where he was a founding engineer [01:09:00] [02:51:00]. Large Language Models (LLMs) significantly accelerate this “learn by doing” methodology, allowing users to get something running in minutes using an AI-powered editor [03:00:00]. Replit enables users to start by prompting or forking templates, quickly getting a “dopamine hit” from seeing immediate results [03:30:00].
The role of a software engineer is expected to bifurcate [04:41:00]:
- Product Creator/Entrepreneur: Focused on making things, acquiring customers and users, often through prompting and iterating on prompts [04:51:00]. This path might involve less traditional software engineering knowledge [06:17:00].
- Traditional Software Engineer: Focused on building cloud infrastructure, data pipelines, or backend systems [05:48:00]. A computer science degree remains relevant for this path [06:10:00].
AI disproportionately benefits beginners, offering an unprecedented return on investment (ROI) for learning to code [01:10:00] [01:51:00]. Studies suggest AI boosts the productivity of less experienced individuals more [01:51:00] [19:30:00]. However, advanced users who learn how to leverage AI with sophisticated prompting techniques (like Chain of Thought) will also see significant benefits [02:03:00].
Younger generations are proving much better at adapting to new AI tools, naturally building mental models and prompting effectively [02:10:00] [02:22:00]. This adaptation is likened to the introduction of calculators in education, which some teachers initially resisted but later found beneficial [02:29:00].
Replit’s Strategy: Embedding AI in the Core Product
Replit’s AI product, originally called “Ghostwriter,” has been renamed to “Replit AI” to signify its complete integration into the platform [06:40:00] [06:48:00]. This move reflects Omj’s view that AI add-ons (like “co-pilots”) are a “transitionary period,” and companies relying on that revenue should be concerned [07:37:00].
Replit’s approach is to embed AI into every interaction, making it part of the free plan [08:07:00]. This ensures designers think from an AI-first perspective [08:42:00]. Key AI features at Replit include:
- Code Suggestions: Passive “push model” where AI suggests code as the user types, similar to Gmail’s ghost text [08:51:00].
- Generate File: An active “pull model” where users can right-click and prompt to generate an entire file based on context [09:29:00].
- AI Debug Button: Appears on console errors, opening an AI chat pre-prompted with the error and relevant context [09:53:00].
Capabilities and Limitations of AI Models in Coding
Omj notes that the understanding of LLMs has evolved from a “mystical component” to a more reductive view: they are primarily a function of data, essentially a compression of vast datasets [11:01:00] [11:36:00]. The power of LLMs lies in interpolating different distributions of data, such as writing a rap song in the style of Shakespeare [11:53:00]. This new paradigm is “software 2.0,” where you program with data [12:42:00].
Data Quality and Training
To understand a model’s capabilities, one must understand the data it was fed and the post-training mechanisms used (e.g., instruction fine-tuning, RLHF, DPO) [12:51:00].
- Size and Compute: More tokens, more compute, and greater diversity/freshness of tokens lead to better models [16:11:00].
- Quality: Training on minified JavaScript, for example, can “mess up” the model [16:48:00]. Models should ideally be trained on data generated by the “best programmers” because GPTs are “emulation machines” that clone human behavior [16:58:00].
- Data Scarcity: Omj argues that the industry is “running out of open-source tokens” [14:02:00]. GPT-4, for instance, is trained on all internet code data plus hundreds of millions of dollars spent on annotated coding data [14:24:00]. Replit has an advantage with its large user base generating unique application code, which is scarcer than high-quality infrastructure code found on GitHub [14:44:00] [18:14:00].
- Diverse Data Sources: Scientific and even legal texts have been shown to improve code generation capabilities, indicating the models learn “coding adjacent reasoning” [15:17:00]. Omj predicts another 2-3 years of increased coding capabilities [15:34:00].
Organizational Structure and AI Adoption
Replit favors a horizontal organizational structure for AI integration, building platforms that touch every aspect of the software [02:41:00]. Omj expresses surprise at the slow pace of AI adoption in some areas, particularly within larger corporations, despite rapid advancements like Copilot’s spread [02:44:00]. He believes AI should move faster to kickstart economic growth, but cultural, legal, and internal forces (e.g., within big AI labs) are slowing it down [02:59:00].
Build vs. Buy: Replit’s Decision to Train Its Own Model
Replit made a strategic decision to train its own AI model (a 3-billion parameter model) for its core code suggestion feature, Ghostwriter [07:12:00] [03:05:00]. The primary reasons were:
- Latency: Commercial models couldn’t meet the low latency requirements for real-time code suggestions [02:07:00]. Omj notes that even Copilot, which had a deep partnership with OpenAI for custom models, has gotten slower [02:17:00].
- Cost: To offer AI as part of Replit’s free experience, commercial model pricing was prohibitive [02:28:00]. Training their 3B model cost around $100,000, which is not a “huge capital expenditure” [02:35:00].
- Small Model Capabilities: Replit was early to realize that small models are capable, affordable to train, and deploy effectively for specific tasks [02:41:00].
However, Replit also uses commercial models for other use cases, such as general-purpose chat features [03:10:00]. The decision to build internally or use commercial APIs should start from the customer pain point, explore solutions, and run the numbers, considering strategic goals (e.g., wanting to be an AI company) [03:24:00].
The Illusion of Open Source AI Models
Omj stirred discussion with a tweet arguing that “true open source models” don’t exist in AI today because they cannot be easily reproduced [03:18:00]. He draws an analogy to Linux: if you could only use the binary or had the source code but no compiler, it wouldn’t be considered open source [03:20:00].
Critically, if companies use open-source models like Meta’s Llama, they remain dependent on the goodwill of the creators (e.g., Mark Zuckerberg) to continue pushing out new versions [03:27:00].
From a security perspective, not having clarity on the training process and data of a model poses a huge risk [03:51:00]. Since “you’re programming with data,” the data acts as the “source code” in this analogy [03:12:00]. Omj references Ken Thompson’s “Reflections on Trusting Trust” paper from the 1970s, which describes how a backdoor can be embedded in a compiler and evade inspection, a concept with parallels in LLM training [03:51:00].
Given these dependencies and risks, Omj believes that in the long term, companies shouldn’t depend on current open-source models as primary solutions. Instead, they should treat them like commercial models for prototyping and experimentation, while having a sustainable path that avoids external dependencies [03:30:00]. He hopes for a truly open-source project that allows contributions and fosters an open-source flywheel [03:39:00].
Future Trends and Challenges
Agents
Omj believes that agents are the “next big thing” after multimodal AI [04:56:00]. While multimodal is a profound incremental improvement, agents represent a more significant shift [04:03:00].
- Cost: Recursive calls by agents (like AutoGPT) and larger models (GPT-4) can become very expensive quickly, making them cost-prohibitive for many users [04:00:00].
- Current State: LLMs often have “accidental” agent capabilities, but true agents might require “action transformers” that predict actions instead of tokens [04:11:00].
- Milestone: A key milestone for agents will be their ability to reliably follow a bulleted list of actions without “going off the rails” or requiring “insane amounts of Chain of Thought and recursive debugging” [04:49:00]. This dependability is crucial for financial or legal workflows [05:27:00].
- Timeline: Omj expects some version of agentic workflows and background agents to start emerging this year [04:06:00]. He encourages entrepreneurs to “walk through walls” and try to make agents work even with current limitations, as it’s a bet worth making [04:30:00].
Hyperreality
Omj notes that the world is rapidly moving towards “hyperreality,” where it becomes incredibly difficult to distinguish what’s fiction from what’s real due to generative AI [03:03:00]. He expresses concern that there isn’t enough focus on building technology to counteract this, like a Chrome extension to identify fake media [03:58:00].
Pricing and Business Models
For AI-native products, Omj advocates for a value-based pricing model rather than “cost-plus” [04:41:00]. He anticipates a future where AI inference costs continue to decrease and the inference stack becomes more efficient [04:11:00].
- Usage-based pricing is becoming more prevalent, especially with AI, because power users can incur significant costs on models, making a pure subscription SaaS model less viable [04:29:00]. Replit, for example, offers bundles but also allows for overages [04:51:00].
- The industry is in a period of “VC subsidized models training and tokens,” which won’t last forever [04:49:00].
Industry Consolidation vs. Specialization
The “default pessimistic assumption” is that Microsoft, with its vast install base, Enterprise relationships, and sales team, will win the entire AI coding space [05:26:00]. However, Omj is optimistic about a new crop of specialized companies [05:03:00]:
- Companies that take a holistic approach, like Replit, providing a cloud development environment with AI sitting on top of the entire stack, can build more ambitious AI products, including agentic workflows [05:38:00].
- Specialized companies, like Codium (generating tests), will also do well [05:59:00].
- The challenge for pure code generation startups is that future models (e.g., GPT-5) might leapfrog their capabilities, making their heavy training investments less competitive [05:46:00]. While Code Llama has shown strong benchmarks, the “Vibes” (actual user experience) may still fall short of proprietary models like GPT-4 [05:57:00].
Surprises and Underhyped Areas
- Biggest Surprise: How much latency matters in AI features [05:27:00]. A 2-3 second response time changes the user experience entirely compared to 300 milliseconds [05:37:00].
- Failed Features: Replit initially struggled to expose “inline actions” effectively [05:14:00]. These actions, which derive information from cursor context, are superior to chat windows but required UI prompting for user adoption [05:31:00].
- Most Exciting AI Startup (outside Replit’s space): OpenAI for its ambition and diverse ventures (education, robotics, self-driving) [05:58:00]. Perplexity AI is also highly regarded for its engineering competency, which allowed it to zoom ahead of competitors [06:04:00].
- Underhyped Area: Using LLMs as part of everyday systems and backend call chains [05:05:00].
- Overhyped Area: Chatbots [05:51:00].
Impact on Engineering Teams
Omj predicts that in 5 years, what’s done now could be achieved with 1/10th of the engineers [06:46:00]. In 10 years, there could be a “1000x component” leading to significantly smaller company sizes [06:55:00]. While the number of people doing “software creation” will grow, they might be called “software creators” rather than traditional “software engineers” [07:14:00].