From: redpointai
AI coding is one of the best use cases for AI observed so far, with GitHub Copilot alone having over a million paying users [00:00:07]. This has led to the evolution of numerous AI tools in the space [00:00:14].
Current State of AI Coding Tools
Currently, AI coding tools primarily act as “inner loop accelerants” [00:07:04]. The “inner loop” of software development involves a developer’s daily routine of figuring out how to implement a feature, writing code, testing it, and then moving to the next task [00:07:16]. Key functionalities in heavy day-to-day use include:
- Inline completion: Autocompleting thoughts as code is typed [00:05:00].
- Chat interfaces: Answering high-level questions about code or generating code for specific tasks [00:05:02].
- Specific commands: Automating “toil” tasks such as writing docstrings or high-quality unit tests for functions [00:05:09].
While these tools are effective for common or boilerplate functions, fully automated, multi-step, bot-driven development, where a human primarily advises rather than guides, is still on the horizon [00:08:32]. Even tools like Devin, while promising, still require human oversight and assistance for most issues [00:08:55].
Sourcegraph’s Approach with Cody
Sourcegraph offers two main products:
- Code Search Engine: Helps human developers understand vast codebases by enabling search, navigation (e.g., go to definition, find references), and context acquisition [00:04:09].
- Cody: An AI coding assistant that provides inline completion, chat capabilities, and a menu of commands for specific development actions [00:04:50].
Cody’s differentiator is its ability to operate with context and awareness of the user’s specific codebase, moving beyond generic suggestions to provide relevant code based on internal frameworks, libraries, and production environments [00:05:30].
Impact of Model Advancements
Improvements in underlying models, such as the progression from GPT-3.5 to GPT-4 and Claude 3, have significantly enabled new use cases and improved reliability [00:13:11]. For instance, Cody’s chat function has become much more reliable, consistently leading to “wow moments” due to the models’ enhanced ability to incorporate and integrate given context into working code examples [00:14:37]. Newer models can even generate an app from scratch using specific libraries and APIs with zero-shot prompting, a feat not possible with older models [00:14:53].
Role of Context Windows and RAG
Larger context windows in models are beneficial because they allow more information to be incorporated into the model’s answer, especially for questions that tie together many different concepts [00:15:58]. However, it’s still not possible to simply “shove the entire codebase” into the context window and expect optimal results [00:16:21].
Retrieval Augmented Generation (RAG) remains critical [00:17:00]. The best application architectures combine large context windows with tailored information retrieval mechanisms [00:17:08]. Effective RAG requires:
- A way to trial and error, making code changes and observing results in a virtual execution environment [00:10:51].
- “Really, really good context fetchers” to reduce the number of iterations needed to find a correct solution [00:11:07].
Model Evaluation and User Feedback
Sourcegraph’s strategy for new model releases (e.g., GPT-5) is to enable them in Cody as quickly as possible [00:17:37]. They run internal benchmarks but prioritize getting models into users’ hands to observe product usage metrics [00:18:02]. Key product metrics for evaluation include:
- Acceptance rate for code completions [00:41:59].
- Overall volume of completions [00:42:01].
- Engagement for explicit features like chat or test generation [00:42:33].
Sourcegraph emphasizes that product metrics are the most authoritative because they reflect real user value [00:19:56]. They sometimes choose models that don’t top benchmarks but perform better in day-to-day use, trusting user feedback and “gut checking” over predefined benchmarks [00:19:11]. This “do the dumb thing first” philosophy is crucial, especially in a fast-evolving space [00:25:29].
Beneficiaries of AI Coding Tools
There’s a correlation in who benefits most from different AI coding tools:
- Junior developers tend to prefer inline completions, finding them a useful “pedagogical tool” that provides a starting point [00:23:01].
- Senior engineers often derive more value from chat features and can find inline completions disruptive if they are not accurate enough [00:21:48]. Large Language Models (LLMs) are effective at “pattern matching” or “bullshitting their way to a correct-looking answer,” which is particularly helpful for less experienced users seeking common solutions [00:23:16].
Fine-tuning and Data Quality
While initially focusing on RAG, Sourcegraph is now fine-tuning models for specific use cases, such as improving code generation in less common programming languages like Rust or Ruby [00:28:06]. They also developed “Cody Ignore,” a feature allowing users to specify files or code sections that the AI should not use as context, enabling exclusion of sensitive or low-quality code [00:29:19]. This offers engineering leaders a lever to guide code quality [00:30:14].
Inference Costs and Pricing
Inference cost is a consideration, but Sourcegraph has not prioritized over-optimizing it, anticipating that costs will significantly decrease over time [00:31:16]. Rate limiters are primarily used to counteract abuse rather than manage legitimate high usage [00:32:15]. Sourcegraph’s pricing model for both its code search engine and Cody is based on “active user per month,” meaning customers only pay if a user logs in and actively uses the product [00:33:03]. This aligns incentives with customer value and avoids over-allocation of seats [00:33:34]. Cody can be used independently (free and pro tiers) or in conjunction with Sourcegraph’s code search for enhanced context [00:34:25].
Organizational Structure for AI
Sourcegraph has a distinct team focused on the “model layer” (fine-tuning and defining quantitative benchmarks) [00:36:21]. The rest of the product engineering organization is currently divided into code search and Cody teams, but these are expected to converge over time due to significant integration opportunities and synergies [00:36:30].
Evolution of Sourcegraph with AI
The early success of incorporating AI signals into search ranking provided initial validation for Sourcegraph’s AI direction [00:38:53]. The company observed where mainstream tools like GitHub Copilot fell short (e.g., lack of awareness of surrounding code) and began experimenting with context-driven code generation [00:39:38]. The release of ChatGPT was a significant accelerant, shifting industry focus and prompting Sourcegraph to put more resources behind Cody’s development, leading to its MVP release [00:40:12]. Companies with unique access to or ways of structuring valuable data have been major beneficiaries of the current AI wave [00:41:24].
Future of AI in Software Engineering and Market Dynamics
The future of AI coding involves a progression from inner-loop accelerants to more automation and agents [00:08:29]. Meaningful milestones would include automated solutions for specific classes of problems, such as 80% of simple bug fixes becoming automatically fixable based on production logs [01:03:52].
While the number of engineers is expected to grow, the definition and day-to-day experience of software development will drastically change [01:06:08]. AI will likely “steal” the toil and drudgery of coding, allowing developers to focus on more creative and valuable tasks [01:08:06].
The development tools sector has become a “hot investment area,” a significant shift from 2013 when investors often avoided it [01:08:36]. The process of software creation is becoming a market in itself, separate from simply building platforms [01:09:31]. AI expands software literacy, enabling more people to create software, thus greatly expanding the market [01:10:39]. This large market has room for many different applications catering to various domains and coding styles [01:10:53].
Sourcegraph aims to ensure the emerging ecosystem remains open, preserving “Freedom of Choice” for individual developers and companies, avoiding a future where large players vertically integrate the market and force proprietary development platforms [01:11:08]. They believe in providing building blocks (like an API for Cody) that allow other developers to build on top of AI [01:00:54], exemplified by their custom command framework [01:01:53].
Underhyped Aspects of AI in Coding
Formal specifications and languages are underhyped. While AI will generate much code, natural language is not precise enough for complex tasks. Programming languages and formal methods will remain crucial complements to AI for defining things with precision [01:15:36].
Local Inference Models
There’s a growing community interest in running local inference models, primarily driven by the desire for offline availability (e.g., on an airplane without connectivity) and longer-term concerns about privacy and cost [00:43:50]. As GPUs and models become faster, inference time, currently the largest source of latency, will decrease, making network roundtrip latency more noticeable and further favoring local computation for latency-sensitive developer workflows [00:46:01].
Conclusion
The evolution of AI coding tools is rapid, moving towards greater automation and integration. Companies like Sourcegraph are adapting by focusing on user value, smart context retrieval, and maintaining an open ecosystem to empower developers. The future promises a transformed, more joyful coding experience by offloading toil to AI.