From: redpointai
Introduction
The rise of AI has led to a provocative question about its impact on employment, particularly in coding [00:00:02]. Coding is considered one of the best use cases for AI so far, evidenced by over a million paying users for GitHub Co-pilot [00:00:09]. This article explores the current landscape of AI coding and software engineering, as well as its future potential, drawing insights from Sourcegraph’s experience [00:00:37].
Current State of AI in Coding
Current AI coding tools primarily act as “inner loop accelerants” [00:07:01]. The “inner loop” refers to the frequent actions a developer performs daily, such as writing, testing, and iterating on code [00:07:18].
Sourcegraph’s AI Products: Cody and Code Search
Sourcegraph offers two main products for developers:
- Code Search A search engine that helps human developers understand vast codebases by allowing them to search, navigate definitions, and find references, thereby acquiring necessary context [00:04:09].
- Cody An AI coding assistant that provides inline code completions, answers high-level questions about code, generates code (e.g., for XYZ functions), and offers commands for specific tasks like writing doc strings or unit tests [00:04:50]. Cody’s differentiator is its context awareness of the user’s codebase, providing more relevant suggestions than generic models [00:05:30]. This means Cody cares about using specific frameworks, libraries, or deployment environments unique to a user’s code [00:06:09].
Model Performance and Context
The advancement of models from GPT-3.5 to GPT-4 and Claude 3 has significantly improved the reliability and consistency of AI coding assistants [00:14:30]. These newer models are better at incorporating context and integrating it into working code examples [00:14:41]. Users can choose among different Frontier models like GPT-4, Claude 3, and Mixtral within Cody, with Claude 3 and GPT-4 being popular for their style, and Mixtral for its speed [00:13:21].
The size of context windows is highly beneficial, as more context allows for more informed model answers, especially for questions tying together many concepts [00:15:58]. However, even with large context windows, simply “shoving the entire codebase” into the window isn’t sufficient for complex tasks, as models struggle with composition beyond simple recollection [00:16:21]. Therefore, effective Retrieval Augmented Generation (RAG) engines, which combine context windows with tailored information retrieval, remain crucial [00:17:00].
User Segmentation and Benefits
There are observed correlations in how different levels of developers benefit from AI coding tools [00:21:09]:
- Junior Developers: Tend to benefit more from inline code completions, finding them a useful pedagogical tool that provides a starting example [00:21:38]. An AI model, being a “machine for pattern matching” and generating “correct looking answers,” is helpful for less experienced users [00:23:13].
- Senior Developers: Gain more value from chat interfaces, while some may find inline completions disruptive if they are not perfectly accurate, preferring to write the code themselves [00:21:48]. This aversion can be mitigated by improving context quality for completions [00:22:21].
Future Potential of AI in Coding
The future of software engineering with AI extends beyond current “inner loop accelerants” towards greater automation [00:30:37].
Agents and Automation
The next frontier is “multi-step, bot-driven development” where the AI agent drives the process, and the human acts more as an advisor [00:08:40]. This requires:
- Virtual Execution Environment: A way for the AI agent to “trial and error” by making code changes and observing results [00:10:51].
- Good Context Fetchers: Crucial for shortening the number of cycles needed to arrive at the correct answer, reducing cost and time [00:11:07].
- Better Models: Continued improvements in model quality and efficiency will reduce the need for surrounding tools and guardrails [01:05:07].
Currently, fully automated systems solve “zero or close to zero” classes of problems in production [01:04:01]. Milestones for the future include automated simple bug fixes, such as those derived from production logs [01:04:25].
The Role of Formal Languages
While natural language processing is advancing, formal specifications and programming languages will remain crucial. Natural language is often not specific enough for precise instructions, making formal languages an increasingly important complement to AI [01:15:59].
Local Inference and Accessibility
A growing trend is the use of local inference models, allowing users to run AI models on their own hardware using tools like Olama or LM Studio [00:43:58]. This offers benefits in:
- Availability: Enables use in environments without network connectivity, like airplanes [00:44:47].
- Privacy: Keeps sensitive code and operations within the user’s local machine [00:45:31].
- Cost: Reduces reliance on external API costs [00:45:26].
- Latency: As GPUs become faster, local inference can reduce round-trip network latency, which is critical for developers who are highly sensitive to even milliseconds of delay [00:46:10].
Evolution of the Developer’s Role
The number of engineers is expected to grow, potentially accelerating in the medium to long term, but the day-to-day experience of being a software developer will drastically change [01:06:05]. AI will “steal” the toilsome and drudgery parts of the job, allowing developers to focus more on creative, high-value tasks and the “magic part” of software creation [01:08:01]. Most developers currently spend little time actually producing software and shipping features, with much of their time consumed by understanding existing code, context acquisition, and communication overhead [01:07:09].
Sourcegraph’s Approach to AI Development
Sourcegraph’s organizational structure includes a model layer team focusing on fine-tuning models and defining quantitative benchmarks, and separate teams for code search and Cody [00:36:21]. Over time, these product teams are expected to converge due to increasing synergies [00:36:41].
Development Philosophy
Sourcegraph prioritizes delivering end-user value quickly by often starting with the “dumb thing first” [00:25:11]. This philosophy, inspired by machine learning research, suggests beginning with simple baselines (like keyword search for RAG) before attempting complex or “fancy” solutions [00:25:50]. This allows for rapid iteration and avoids the overhead of extensive data gathering and training [00:27:28].
Fine-Tuning and Context Control
While initially relying heavily on RAG, Sourcegraph is now exploring fine-tuning models for language-specific code generation, especially for languages like Rust or Ruby that perform less well with general models [00:28:06].
The “Cody Ignore” feature allows users to specify which files in their codebase Cody should ignore, initially for sensitive data, but now also for excluding “bad code” from being used as context for AI generation [00:29:19]. This provides engineering leaders with control over code quality and context for automated code generation [00:30:00].
Model Evaluation
Sourcegraph primarily evaluates models using product metrics, such as acceptance rate and overall volume for completions, and engagement for chat features [00:18:02]. They prioritize getting new models into users’ hands quickly to gather real-world feedback, as product metrics are harder to game than internal benchmarks and reflect actual value [00:18:15].
Inference Costs and Pricing
While inference costs are a consideration, Sourcegraph has not over-optimized for cheapness, believing costs will decrease over time [00:31:33]. Their pricing model for Cody and Code Search is based on “active user per month,” meaning customers only pay if a user logs in and uses the product, aligning incentives with value delivery [00:33:03]. Cody can be used independently or connected to a Sourcegraph instance for enhanced context [00:34:30].
The Future Potential and Development of AI Assistance APIs
The future potential and development of AI assistance APIs will emphasize providing high-quality context providers as reusable building blocks for an open ecosystem [01:00:12]. This allows other developers and companies to build on top of AI, fostering innovation without proprietary lock-ins [01:00:54].
Sourcegraph supports this by:
- Custom Commands: Cody allows users to create custom commands by defining desired context, prompts, and output rendering, enabling personalized AI actions [01:01:51].
- Open Models: Open-source models like StarCoder are increasingly prevalent, offering advantages like fine-tuning capabilities and access to internal model states (e.g., attention weights) for application-level insights [01:20:04].
Conclusion
The impact of AI on research, coding, and education is profound. The current state of AI coding is characterized by effective “inner loop accelerants” that streamline common development tasks, driven by advanced models and intelligent context retrieval (RAG). The future potential lies in full automation via AI agents, which will reshape the software development job, allowing developers to focus on higher-level, creative work [01:07:09]. This shift will also foster a vibrant, open ecosystem of developer tools built on shared AI assistance APIs and models, rather than proprietary platforms [01:10:50].