AI coding assistants and their uses

From: redpointai

AI coding assistants are emerging as powerful tools in software development, with over a million paying users for products like GitHub Copilot alone [00:00:11]. These tools represent one of the best use cases for AI seen to date [00:00:08].

Sourcegraph’s Cody: An AI Coding Assistant

Sourcegraph offers an AI coding assistant called Cody, which provides various functionalities to developers [00:04:50]:

• Inline completion: Completing thoughts as a developer types code in the editor [00:04:56].
• Chat: Allowing users to ask high-level questions about code, generate code snippets, or perform specific actions [00:05:00]. Examples include writing docstrings for functions or generating unit tests [00:05:20].
• Context awareness: Cody’s differentiator is its ability to integrate context and awareness from the user’s code base, similar to how Sourcegraph’s code search engine provides context to human developers [00:05:35]. This means the AI is aware of specific frameworks, libraries, and production environments, moving beyond generic open-source examples [00:06:06].
• Retrieval Augmented Generation (RAG): Cody heavily utilizes RAG to connect queries to the customer’s actual code base [00:06:17].

Development Philosophy

Sourcegraph’s approach to building AI products emphasizes simplicity and iterative improvement:

• “Dumb thing first”: The company prioritizes the simplest solution that provides user value, like leveraging classical keyword search for RAG in the early days [00:25:29], [00:52:05]. This allows for rapid iteration without the overhead of extensive data gathering or complex model training [00:27:31].
• Model selection: Users are given the freedom to choose among different models (e.g., GPT-4, Claude 3, Mixtral) in Cody’s chat view, allowing them to find what works best for their use case [00:13:21]. This also helps in gathering diverse user feedback for product improvement [00:18:11].
• Fine-tuning: While initially focusing on RAG, Sourcegraph is now implementing fine-tuning for specific use cases, particularly for language-specific code generation where general models may perform less effectively (e.g., Rust, Ruby, MATLAB) [00:28:06].
• Code quality control: Sourcegraph plans to allow engineering leaders to control which code is used as context for automated generation, enabling them to improve overall code quality [00:30:04].

Current Landscape of AI in Coding

The current state of AI in software development is focused on “inner loop accelerants” [00:07:01]. These are tools that speed up the frequent, repetitive tasks developers perform daily, such as writing common functions, auto-completing code, or generating boilerplate [00:07:18], [00:08:09].

The Shift Towards Automation

While “inner loop” tasks are well-supported, the next frontier for AI advancements in coding and software engineering is multi-step, bot-driven development, where AI agents drive the development process with humans advising [00:08:32]. This involves:

• Trial and error: AI agents trying different approaches, observing results, and learning from mistakes to refine code changes [00:10:30].
• Virtual execution environments: Tools need to support virtual environments to allow AI to make and test code changes and observe outcomes [00:11:00].
• Improved context fetching: The ability to quickly retrieve and integrate relevant code context is crucial for reducing the number of cycles an AI needs to reach a correct solution, making automation more efficient and cost-effective [00:11:07], [00:11:37].

Impact of Model Advancements and Context Windows

Advancements in large language models (LLMs) like the transition from GPT-3.5 to GPT-4 have significantly improved the reliability of AI coding assistants, making “wow moments” more consistent [00:14:15]. Larger context windows are also beneficial, as they allow more information to be incorporated into the model’s answer, especially for questions involving complex concept tying [00:15:58]. However, simply stuffing an entire codebase into a context window is not yet effective for complex reasoning tasks [00:16:21]. The best architectures combine large context windows with tailored information retrieval mechanisms like RAG [00:17:08].

Benefits and Segmentation of Users

AI coding tools offer varied benefits depending on the developer’s experience level:

• Junior developers: Tend to benefit more from inline code completions, viewing them as a “pedagogical tool” that provides a starting point or median way of doing things [00:21:36], [00:23:06]. This aligns with how LLMs are good at pattern matching and generating plausible-looking answers [00:23:16].
• Senior engineers: Often prefer chat functionalities and may find inline completions disruptive if they are not consistently high quality, as they know exactly what they want to write [00:21:48], [00:22:05]. Improvements in context quality can address some of their concerns [00:22:21].

Production and Evaluation

Sourcegraph’s approach to AI production and evaluation techniques prioritizes real-world user feedback and product metrics:

• User feedback: New models like GPT-5 are immediately made available to users, allowing Sourcegraph to observe their impact on product usage metrics and gather direct feedback [00:18:02].
• Product metrics: For completions, key metrics include acceptance rate (how often users accept suggestions) and overall volume (how much value is being added) [00:41:59]. For explicit actions like chat or test generation, engagement and overall usage are monitored [00:42:32]. These “authoritative metrics” are considered harder to game and directly reflect user value [00:19:56].
• Beyond benchmarks: Sourcegraph has sometimes chosen models not at the top of benchmark lists but that generate better results in day-to-day use, highlighting the nuance of end-user experience not captured by well-defined tests [00:19:14].
• Offline benchmarks: The company also maintains a set of offline benchmarks for new models, with an ongoing effort to build more specifically tuned benchmarks for retrieval-based models [00:48:37].

Cost Considerations

Inference cost is a factor, but Sourcegraph aims to do the “bare minimum” to control costs, believing they will decrease over time [00:31:38]. The focus remains on adding value. Rate limiters are primarily used to counteract abuse rather than as a primary cost-saving measure for legitimate users [00:32:03].

Local Inference Models

There’s a growing community around local inference models, allowing users to run models on their local hardware (e.g., with Ollama or LM Studio) [00:43:58]. The benefits of local inference include:

• Availability: Enables usage in environments without network connectivity, like on an airplane [00:44:47].
• Privacy: Keeping computation contained within the local machine addresses developer concerns about data privacy [00:45:29].
• Cost: Potentially reduces inference costs over time [00:45:26].
• Latency: As GPUs become faster and inference times decrease, local processing eliminates network round-trip latency, which is crucial for developers who are highly sensitive to even milliseconds of delay in their coding environment [00:46:16].

The Future of Software Engineering with AI

AI is set to profoundly change the role of software engineers:

• Job transformation: While AI will “take your job,” it will transform it into a different, potentially better, job [01:06:46].
• Focus on value creation: Currently, developers spend little time actually producing software and shipping features, with much time spent on understanding existing codebases, context acquisition, and communication overhead [01:07:09]. AI can automate the “toil and drudgery,” allowing engineers to focus on “magic part” – creating unique value and differentiation [01:08:04].
• Increased number of engineers: The aggregate number of engineers is expected to grow, as AI in education makes software literacy more widespread and puts the power of software creation into more hands [01:10:28].
• Formal languages remain vital: Despite the ability of LLMs to generate code from natural language, formal specifications and programming languages will remain crucial due to the need for precision that natural language lacks [01:15:50].

Market Dynamics for AI Coding Tools

The market for AI coding tools is vibrant and growing, shifting from a niche “Dev tools” area to a “market unto itself” [01:09:05], [01:09:31].

• Open ecosystem: Sourcegraph advocates for an open ecosystem that preserves freedom of choice for developers and companies regarding models, code hosts, and technology stacks [01:11:08]. This contrasts with a potential future where large players vertically integrate and force users onto proprietary platforms [01:12:13].
• Open source models: Open-source models are expected to become very widespread due to public accessibility of research and their inherent advantages like fine-tuning capabilities and deeper introspection into model attention weights [01:20:21]. StarCoder, an open-source model, is Sourcegraph’s primary code completion model [01:20:44].
• Building blocks: Beyond end-user applications, companies like Sourcegraph aim to provide reusable “building blocks” (e.g., context APIs, custom command frameworks) that enable other developers to build their own AI-enabled tools and script against them [01:00:54], [01:01:11].