From: redpointai
Sourcegraph, led by CTO and co-founder Beyang Liu, has been at the forefront of building AI products for developers. Their flagship product, Cody, is an AI coding assistant designed to enhance software development by providing contextual assistance and automation [00:00:16]. This article explores Sourcegraph’s approach to building and scaling AI products, their insights into the evolving landscape of AI in coding, and their vision for the future of software engineering.
The Evolving Role of Software Developers in the Age of AI
AI is rapidly transforming software development. While some may provocatively ask if AI will “steal my job,” the reality is that coding is one of the best use cases for AI seen so far, with over a million paying users for tools like GitHub Copilot [00:00:07] [00:00:11].
Instead of replacing developers, AI is expanding the horizons of software creation, enabling the generation of high-quality software with increased leverage from technical thinking [00:02:34] [00:02:36] [00:02:40]. The essence of software creation lies in connecting the product side (delivering value to end-users) with the underlying computational pieces (bits, bytes, data structures, algorithms) [00:02:50]. A significant portion of a developer’s time is spent understanding existing codebases and acquiring context, rather than actively producing and shipping features [01:07:10] [01:07:17]. AI aims to automate this “toil and drudgery,” allowing developers to focus on more creative and high-value tasks [01:08:06] [01:08:09].
The definition of “engineering” and the day-to-day experience of a software developer are expected to change drastically, leading to a potentially better job [01:06:11] [01:06:56]. The number of engineers is projected to grow, possibly accelerating in the medium to long term [01:05:48].
Sourcegraph’s AI Offerings: Cody and Code Search
Sourcegraph offers two main products for developers:
- Code Search Engine: Designed to help human developers understand code. It allows users to search vast codebases, acquire understanding, and navigate code references (e.g., go to definition, find references) to gain context and effectiveness [00:04:09] [00:04:12].
- Cody (AI Coding Assistant): Performs functions expected of an AI coding assistant, including:
- Inline completion: Completing thoughts as code is typed in the editor [00:04:56].
- Chat: Answering high-level questions about code and generating code for specific tasks [00:05:00].
- Commands: Targeting specific actions and “toil” in the software development loop, such as writing doc strings or high-quality unit tests [00:05:11].
Cody’s differentiator is its ability to operate with contextual awareness of the user’s codebase, providing more relevant and precise suggestions than general models trained on open-source code or Stack Overflow [00:05:37] [00:06:01]. This context is supplied by Sourcegraph’s search capabilities [00:05:40].
Current State of AI in Coding
Currently, AI in coding primarily excels as “inner loop accelerants” [00:07:04]. The “inner loop” refers to the daily cycle of a developer: figuring out how to do something, writing code, testing it, and repeating [00:07:18]. Tools like code completion or code-aware chat help automate commonplace or “uninteresting” functions, allowing the AI to autocomplete or generate boilerplate code [00:08:02] [00:08:15].
The next frontier is “multi-step, bot-driven development,” where the AI drives the process and the human acts as an advisor [00:08:40] [00:08:47]. While tools like Devin show promise, human oversight and assistance are still required for the majority of issues [00:08:55] [00:09:07].
Factors Driving AI Product Improvement
Model Advancements and Evaluation
Model advancements, such as the progression from GPT-3.5 to GPT-4 and Claude 3, have significantly enabled new use cases and improved reliability in AI coding assistants [01:13:10] [01:14:00]. Sourcegraph allows users to select between different frontier models (e.g., GPT-4, Claude 3, Mixtral) in Cody’s chat view, finding that Claude 3 and GPT-4 excel at integrating context into working code examples, while Mixtral is favored for its speed [01:13:21] [01:14:01].
When a new model like GPT-5 is released, Sourcegraph aims to integrate it quickly to allow users to experiment and provide feedback [01:17:37]. This user feedback, combined with product usage metrics (like completion acceptance rate and overall volume), is often prioritized over traditional benchmarks, as real-world usage reflects true value [01:18:02] [01:19:07].
Context Windows and Retrieval Augmented Generation (RAG)
Larger context windows in models are beneficial as they allow more information to be incorporated into the model’s answer, especially for questions involving tying many concepts together [01:16:02]. However, simply stuffing an entire codebase into a context window is not yet effective for complex tasks beyond simple recollection [01:16:21]. The best application architectures combine larger context windows with tailored information retrieval mechanisms [01:17:08]. Sourcegraph is not getting rid of RAG [01:17:19].
Sourcegraph’s RAG approach prioritizes simple, effective solutions first. Initially, Cody’s context engine largely relied on “classical keyword search” combined with clever chunking strategies, proving highly effective for context quality [00:52:08] [00:52:49]. While vector databases and complex embeddings models were explored, they often performed poorly or were noisy compared to keyword search [00:53:28] [00:53:53]. The search problem is considered a “meaty” challenge, involving data-driven approaches combined with human judgment for ranking and relevance [00:56:52].
Local Inference Models
A growing trend is the use of local inference models, running on users’ local hardware (e.g., via Olama or LM Studio) [00:44:01]. This is particularly useful for scenarios without network connectivity (like on an airplane) and for addressing privacy and cost concerns [00:44:50] [00:45:22]. As GPUs become faster and models more efficient, local inference will reduce latency, which is critical for developers who are highly sensitive to even milliseconds of delay in their workflow [00:46:13] [00:46:26].
Building and Scaling Sourcegraph’s AI Product
Organizational Structure
Sourcegraph’s product engineering organization is structured into three main areas [00:36:21]:
- Model Layer Team: Focused on fine-tuning models for better code generation in specific use cases (especially for less common programming languages like Rust or Ruby) and defining quantitative benchmarks [00:28:06] [00:37:22].
- Code Search Team: Works on the core code search engine [00:36:35].
- Cody Team: Focuses on the AI coding assistant [00:36:36].
These teams are expected to integrate more over time due to synergies between their respective areas, such as using context APIs from code search to improve Cody, and using AI for better ranking mechanisms in search [00:36:41]. Cody started with a small team of two people and grew to 5-6, with the client team remaining relatively small (less than 10 people) supported by others improving context fetching [00:37:41].
Product Development Philosophy: “Do the Dumb Thing First”
Sourcegraph’s philosophy for building AI products is to “do the dumb thing first” [00:25:29]. This involves starting with the simplest possible solution to establish a baseline and quickly iterate [00:27:22]. This approach allowed them to achieve high context quality rapidly with keyword search before exploring more complex methods like fine-tuning or specialized embeddings [00:52:08]. Fine-tuning becomes important when simpler RAG approaches are no longer sufficient to achieve desired model performance [00:27:51].
Data Curation: “Cody Ignore”
Sourcegraph has a feature called “Cody ignore,” which is a file format that tells Cody to disregard certain files in a codebase when generating completions or responses [00:29:21]. Initially for sensitive files, it’s evolving to allow users and engineering leaders to control which code (e.g., “bad code”) is used as context, enabling higher quality outputs and improving overall code quality within an organization [00:29:35] [00:29:50].
Inference Cost and Pricing
While inference cost is a consideration, Sourcegraph has not prioritized over-optimizing it, believing that costs will significantly decrease over time [00:31:35]. Their focus remains on adding user value [00:31:47]. Rate limiters are primarily used to counteract abuse rather than manage legitimate user costs [00:32:15].
Sourcegraph employs an “active user per month” pricing model for both code search and Cody [00:33:03]. This is a variant of seat-based pricing where customers only pay if a user logs in and actively uses the product, aligning incentives with the value seen by the customer [00:33:11]. While usage-based pricing might be introduced for high-value but expensive features in the future, simplicity is prioritized for now [00:33:53]. Cody can be used for free or with a Pro tier without adopting Sourcegraph’s code search, but connecting it to a Sourcegraph instance enhances its value by providing more context [00:34:25].
Market and Ecosystem Vision
The AI coding market is viewed as a “hot investment area,” a stark contrast to the pre-AI era for developer tools [01:08:41] [01:09:05]. Software creation itself is becoming a significant market, expanding software literacy and putting the power of software creation into more hands [01:09:33] [01:10:28]. This implies significant room for diverse applications that cater to different domains and styles of software building [01:10:50].
Sourcegraph’s goal is to ensure the emerging ecosystem is open and preserves freedom of choice for individual developers and companies regarding models, code hosts, and technology stacks [01:11:06]. They provide not just end-user applications but also “building blocks” (like the Cody API and custom commands) for other developers to integrate and build upon [01:00:54] [01:01:10] [01:01:51]. This includes working closely with various model vendors, both open and closed source [01:13:30].
The Future of AI in Coding
Agents and Automation
Sourcegraph is actively experimenting with systems targeting full automation for specific problem sectors [00:58:00]. These systems combine existing context providers with an execution loop, aiming for recurring usage [00:58:18]. Solving these “outer loop” problems also improves “inner loop” cases, as sub-routines (e.g., test generation for verification) are reusable across both [00:58:43].
Key Milestones for AI in Coding
Meaningful milestones for AI in coding would include the ability for fully automated systems to solve classes of problems that are currently impossible in production, such as automatically fixing 80% of simple bugs identified from production logs [01:03:55] [01:04:25]. Continued advancements in model quality and controllability will also reduce the need for “hacking around” model limitations [01:04:36] [01:05:07].
Overhyped vs. Underhyped
- Overhyped: The notion of AGI (Artificial General Intelligence), particularly the “messianic vision” of it solving all problems or causing doom. Simply scaling Transformers is seen as a fundamentally flawed approach to achieving AGI [01:14:12] [01:14:35].
- Underhyped: Building complementary technologies to AI, especially formal specifications and formal languages [01:15:28]. While AI can process natural language, the precision needed for useful tasks means that programming languages and mathematical notations will remain crucial for describing intentions with precision [01:15:59] [01:16:10].
Open Source Models
Open-source models are expected to become very widespread, complementing proprietary models [01:29:30]. The increasing accessibility of key research means that the ability to train high-quality models will become more common [01:20:24]. Sourcegraph itself uses open-source models like StarCoder as their primary code completion model [01:20:44]. Advantages of open-source models include the ability to fine-tune them and peer into internal states like attention weights, which is useful at the application level [01:20:50].
Contact Information
To learn more about Cody, visit cody.dev [01:26:24]. It can be downloaded from the VS Code Marketplace or the JetBrains plugin store [01:26:31]. For code search, visit sourcegraph.com [01:26:39]. Users can also join their Discord community for discussions and feedback [01:26:41].