From: redpointai
Logan Kilpatrick, the first AI hire at OpenAI, highlights the extensive applications of ChatGPT, particularly in coding and software development. His role at OpenAI focuses on enhancing the developer platform, a task where ChatGPT has proven indispensable [00:54:54].
Personal and Professional Use of ChatGPT in Coding
Kilpatrick, despite not having a classical computer science background as a software engineer or being a web development expert, uses ChatGPT extensively in his work [01:00:00]. He estimates that approximately 90% of the features he ships are built using ChatGPT-generated code [00:01:15]. This reliance on AI allows him to go “beyond what I would normally be able to do” without spending excessive hours on documentation for frameworks like React [00:01:20]. He finds significant value in its ability to generate “true coding things” daily [00:01:31].
The models, including GPT-4, provide significant freedom for engineers [01:18]. The ability to quickly translate ideas into functional code without deep domain expertise is a major benefit [01:07].
Impact on Developers and Software Engineering
Kilpatrick emphasizes that AI coding tools are becoming “table stakes” for developers [00:55:08]. He asserts that an average developer using AI tools can outperform some of the best in the world without them [00:55:10]. This capability amplifies a developer’s ability to build anything they can imagine, a stark contrast to two years prior [00:55:20].
He strongly advises all developers to use tools like ChatGPT and GitHub Copilot [00:54:57]. The continuous evolution of models suggests that in the coming years, individuals will have the capability to build almost any product or service due to the extensive assistance provided by these models [00:55:31].
Key Tools and Concepts:
- Code Interpreter: Six months prior to the interview, code interpreter was not available in the API, but it has since become one of the most exciting features for developers to build into their products [03:00].
- Assistance API: Kilpatrick believes the Assistance API will be a long-term significant offering, enabling many more experiences a year from now [02:10] [02:48].
- Fine-tuning: Fine-tuning models like GPT-3.5 can achieve GPT-4 level performance with prompt engineering, leading to token savings [05:58].
- Function Calling: A crucial use case that enables most interesting production applications [49:51]. Fine-tuning GPT-3.5 for function calling works very well, allowing developers to remove function tokens from prompts, reducing costs [06:49].
- Custom Models: While expensive ($2-3 million), custom models are useful for domains where base models lack sufficient data, such as legal or medical fields [07:40] [08:17]. They also allow for more compute-efficient models by removing unnecessary training data [10:06]. However, building them requires significant data (billions of tokens) and machine learning expertise [10:44]. OpenAI’s custom model program offers hand-holding from their research teams for companies lacking world-class ML teams [11:15].
- Multimodal AI: While still early for vision use cases, the future will involve significant multimodal capabilities [03:21]. The current state of vision models is comparable to GPT-3.5, and a “GPT-4” level leap is needed for more robust applications, especially for detailed understanding of positional relationships between objects [04:06].
- Prompt Engineering: Despite talk of its “death,” prompt engineering persists in being quite useful [00:28:50]. Its fundamental nature is communication, and models like DALL-E 3 demonstrate a future where the model translates a concise user prompt into a more verbose, revised prompt for better results, reducing user friction [00:29:05].
- Observability: Kilpatrick considers observability “underhyped” and crucial for developers to understand how models are functioning [00:48:18]. Many developers use third-party observability products due to OpenAI’s current lack of a detailed dashboard for API usage [00:17:37].
Comparison with Open Source Models
Kilpatrick, a proponent of open source, believes that OpenAI’s models will consistently outperform open source alternatives due to the immense cost and engineering work involved in training large models [00:15:01]. However, open source models like Llama offer greater customization and fine-tuning options, including the ability to apply techniques like Reinforcement Learning from Human Feedback (RLHF), which are not yet available in OpenAI’s standard offerings [00:15:52].
Challenges and Future Outlook
- Robustness and Reliability: Enterprises face challenges in ensuring LLM robustness and reliability, often requiring third-party orchestration frameworks and guardrails. OpenAI aims to solve many of these problems upstream [00:45:08].
- Latency: High latency remains a significant barrier for many use cases where users cannot wait several seconds for a response [00:46:11]. Improving inference speed and increasing GPU availability are key objectives for 2024 [00:46:23].
- User Experience (UX): A fundamental challenge for ChatGPT is that users often don’t know what to do next [00:56:50].
- Agents: While there was a hype cycle for agents (e.g., AutoGPT, BabyAGI), Kilpatrick emphasizes the need for significant internet infrastructure work to authenticate humans versus AI agents to prevent misuse and ensure responsible deployment [00:35:44]. The current environment is not ready for wide-ranging autonomous agents, and a gradual transition is necessary [00:36:11].
- Model Evals: The process of evaluating new models is a “huge pain” [00:22:21]. Kilpatrick is excited about startups that can solve the “eval problem,” as it’s critical for users to know how a new model will impact their specific use case [00:21:52]. Learning often happens by examining failure points, which is hard to automate [00:23:18].
OpenAI’s Strategy and Growth
OpenAI’s product footprint is vast, serving diverse users and use cases [00:11:50]. Prioritization balances clear developer demands (like API key-based usage data) with fundamental needs like reliability, which is a Northstar [00:12:17] [00:12:41]. Shipping new capabilities often takes precedence due to engineering resource constraints [00:12:57]. As the team grows, there will be more time to address “rough edges” like dashboards and monitoring, leading to a true Enterprise platform in 2024 [00:13:14].
Kilpatrick notes the fascinating growth of OpenAI, describing it as an “endless amount of work” despite rapid hiring [00:50:45]. The company has expanded from a small engineering team for the entire API to multiple specialized teams for capabilities, fine-tuning, and enterprise offerings [00:50:57].
General Advice for AI Adoption
For those “AI curious” but overwhelmed, Kilpatrick advises:
- Identify Pain Points: Audit your job or daily life for tasks you dislike or want to improve [00:54:24].
- Integrate AI into Workflows: Make using AI tools a habit by incorporating them into daily routines [00:56:03].
- Be an Ambassador: Share your experiences and insights with others [00:54:15].
Ultimately, the goal is for AI to assist users in their existing workflows, rather than requiring them to adopt new platforms [00:32:01]. This is exemplified by Microsoft’s Copilot strategy, embedding AI directly into widely used applications [00:34:45].