AIs impact on software development

From: redpointai

Omay (omjmad), founder of Replit, discusses the transformative impact of AI on software development and the evolving role and expectations of software developers [00:00:05]. Replit is at the forefront of using AI in innovative ways for coding, having raised over $100millionatavaluationexceeding$1 billion [00:00:15].

Learning to Code in the AI Era

The traditional academic approach to learning to code, such as going to university to study basics, is not how most people learn [00:01:17]. Instead, people learn by creating things to achieve a goal [00:01:27]. Omaj believes that Large Language Models (LLMs) take this “learn by doing” concept to its extreme [00:03:00]. With LLM-powered editors like Replit, users can get something running in as little as five minutes, avoiding the “drudgery” of installation and setup [00:03:05]. This provides immediate “dopamine hits” from seeing results, encouraging further experimentation and larger projects [00:03:39].

The Evolving Role of Software Engineers

Omaj predicts a bifurcation in software engineering roles, a trend that was already present with frontend and backend engineers, but which AI will significantly widen [00:04:41].

The two main paths are:

• Product Engineer/Creator: This role involves using AI to create products for customers and users, often through prompting and iterating on prompts [00:04:51]. While debugging code is still part of it for now, it may eventually be automated [00:05:13]. This path does not necessarily require a computer science degree [00:06:10].
• Traditional Software Engineering: This path focuses on building cloud infrastructure, data pipelines, or backend systems [00:05:48]. This role, which involves thinking about components and systems, is expected to change less dramatically with AI [00:05:29]. A computer science degree remains relevant for this path [00:06:10].

AI disproportionately benefits beginners more than advanced users initially [00:19:34]. Beginners can achieve significant results with minimal coding knowledge, even starting companies and generating substantial revenue within months [00:18:53]. However, Omaj suggests that advanced users who are skilled at leveraging AI tools (e.g., prompting techniques, Chain of Thought, using multiple LLMs) will ultimately see greater benefits [00:20:12]. Younger generations, in particular, are much better at adapting to and building mental models for what LLMs can do [00:22:10].

Replit’s AI Integration

Replit’s strategy involves deeply embedding AI into its product, moving beyond the “co-pilot” add-on model [00:07:37].

• From Ghostwriter to Core Product: Initially, Replit’s AI product was called Ghostwriter, functioning as an add-on [00:06:40]. Recognizing that AI would become “table stakes,” Replit embedded AI into every interaction, making it part of the free plan [00:08:05].
• AI-Powered Workflows:
  • Co-suggestions: Replit provides AI code suggestions while users type, similar to GitHub Copilot [00:08:51]. This is a “push model” where AI passively assists [00:09:19].
  • Code Generation: Users can prompt the AI to generate entire files based on context [00:09:32]. This is a “pull model” where the user actively requests AI assistance [00:09:29].
  • AI Debugging: An “AI debug button” appears when an error occurs in the console, opening an AI chat pre-prompted with the error and relevant context [00:09:53].
  • Contextual Actions: The product can derive information from the cursor’s position, enabling highly contextual AI actions within the IDE [00:59:36].

Replit chose to build its own 3-billion parameter model for Ghostwriter, spending approximately $100,000, due to the need for low latency and cost-effectiveness [00:28:05]. This decision was crucial because commercial models at the time did not meet their latency requirements [00:28:07]. Small models were found to be capable and affordable to train and deploy [00:29:18]. Despite training their own models, Replit also utilizes commercial models for other use cases like general-purpose chat features [00:30:10].

Understanding AI Models for Coding

Omaj views LLMs reductively as a “function of data” or a “compression of the data” [00:11:38]. Their power lies in interpolating different distributions of data (e.g., writing a rap song in Shakespeare’s style) [00:11:53].

Key factors for model capabilities are:

• Data Sources: Understanding the data fed into the models is crucial [00:12:51].
• Fine-tuning/Post-training: Mechanisms like instruction fine-tuning, RLHF (Reinforcement Learning from Human Feedback), and DPO (Direct Preference Optimization) are used to improve models for specific downstream tasks [00:12:55].

For code generation models, the quality of data is paramount:

• Size and Compute: More tokens and more compute spent on training lead to better models [00:16:11].
• Diversity and Freshness: A wider variety and newer data are beneficial [00:16:23].
• Human-generated Data: Training on code generated by the “best programmers” is important because LLMs emulate human thinking [00:16:58].
• Application Code: While GitHub is rich in high-quality infrastructure code, there’s a “poverty” of high-quality application code [00:18:07]. Replit, with its user base writing applications, has an advantage in gathering this type of data [00:18:14].
• Adjacent Data: Surprisingly, scientific data and even legal text have been shown to improve code generation abilities [00:15:17].

Omaj expresses concern that “open source” models today are not truly open source because they cannot be fully reproduced or compiled without proprietary compilers or data, making users dependent on the “goodwill” of the companies releasing them [00:31:48]. Not knowing the underlying training data also poses a significant security risk, as “backdoors” can be hidden in the models, activated only under specific conditions [00:36:20].

The Future of AI in Software Development

• Agents: Omaj believes agentic workflows and background agents are the next major advancement [00:40:00]. These allow models to recursively call themselves to perform larger tasks, such as refactoring code and running tests [00:40:05]. The current limitation is cost with large models like GPT-4 [00:40:30] and reliability (e.g., models going “off the rails”) [00:49:52].
• Pricing: As AI becomes table stakes, pricing models for software will shift from “Cost Plus” (based on AI inference costs) to “Value Based” [00:42:51]. Usage-based pricing is also becoming more prevalent to account for “power users” who incur high costs [00:44:33].
• Competitive Landscape: Microsoft, with its vast install base, sales team, and commercial mindset, is seen as the default winner in the AI coding market [00:51:26]. However, there is room for specialized companies (e.g., those focused on generating tests) and platforms that offer a holistic approach to cloud development environments, integrating AI across the entire stack [00:52:03].

Omaj notes that the overall speed of AI adoption in everyday technology is surprisingly slow, attributing it to corporate inertia and forces working against faster progress [00:24:41]. Despite this, he believes AI will “kickstart another S-curve of growth” [00:26:08]. The future of software development will see fewer engineers required to achieve current output levels (e.g., 1/10th the engineers in 5 years, possibly 1/1000th in 10 years), but the number of “software creators” will continue to grow, even if they are called something else [01:01:49].

Key Insights

• Latency Matters: For AI infrastructure and developer tools, low latency is crucial; even a few seconds delay can drastically change the user experience [00:58:27].
• “Vibes” of Models: While benchmarks are important, the practical “vibes” (how a model feels to use) are also key [00:55:14].
• Hyperreality: The rapid advancement of generative AI is leading to a “hyperreality” where it’s increasingly difficult to distinguish between fiction and reality, raising concerns about lack of technology to counteract it [00:37:49].
• OpenAI’s ambition: OpenAI is admired for its ambition and diverse partnerships across various sectors, from education to robotics [01:00:00].
• Perplexity: Perplexity is highlighted for its strong engineering and ability to deliver value, despite intense competition [01:01:04].