From: aidotengineer
Karina, an AI researcher at OpenAI, discussed the scaling paradigms in AI research over the past two to four years and the new frontiers they have unlocked in product research. She also shared insights into the design challenges encountered during the development of products like Claude and ChatGPT, and her vision for the future of AI agents as co-innovators [00:00:54].
Scaling Paradigms and Emerging Challenges
Two primary scaling paradigms have dominated AI research recently:
Next Token Prediction (Pre-training)
This paradigm, often referred to as pre-training, enables models to build a “world understanding” by predicting the next word or token in a sequence [00:01:46]. This process allows models to learn about the physics of the world and perform massive multitask learning [00:02:09]. While some tasks, like translation, are easy to learn, others present significant challenges:
- Hard-to-Learn Tasks Scaling compute in the pre-training stage is crucial for tasks that are “really, really hard to learn” [00:03:21]. These include:
- Math and Computational Tasks Requires complex reasoning, often necessitating techniques like Chain of Thought to enable the model to compute numbers and reason through problems [00:03:43].
- Creative Writing This remains an open-ended research problem and “one of the hardest AI research problems today” [00:03:38]. The difficulty arises because it’s hard to measure what constitutes good creative writing [00:04:46], and models struggle with maintaining plot coherence over long narratives, leading to rapid deterioration [00:04:22]. The goal is for models to invent new forms of writing and generate extremely creative content [00:04:51].
Post-training (RLHF/RLAF)
Following pre-training, models undergo post-training using Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning from AI Feedback (RLAF) [00:06:06]. This stage teaches models to complete specific functions like understanding docstrings, generating multi-line completions, and predicting/applying diffs [00:06:23].
Scaling Reinforcement Learning on Chain of Thought
The latest paradigm, introduced with OpenAI’s GPT-4, involves scaling reinforcement learning on Chain of Thought [00:07:01]. This allows models to learn how to “think” during training by receiving good feedback signals, leading to highly complex reasoning [00:07:24].
A key challenge in this area is:
- Faithfulness in Chain of Thought Significant scientific work is needed to measure the faithfulness of a model’s Chain of Thought [00:08:35]. This includes understanding what happens if a model pursues a wrong direction and if it can backtrack itself to correct errors [00:08:46].
Specific Design Challenges for AI Agents
As model capabilities and interaction paradigms evolve, new design challenges emerge [00:09:46]:
Human-AI Interaction Paradigms
- Managing Wait Times A significant design challenge is creating new interaction paradigms where humans don’t have to wait extended periods (e.g., 15 seconds or 30 minutes) for a model’s response [00:09:12]. One simple approach implemented is streaming model thoughts to the user [00:09:31].
- Familiar Form Factors for New Capabilities Bringing unfamiliar AI capabilities into familiar user interfaces is a design challenge [00:14:07]. For instance, the success of Claude’s 100K context was partly due to integrating it via familiar file uploads, rather than less intuitive infinite chats [00:13:41]. Product features should enable modular compositions that scale well with future model capabilities [00:15:21].
Bridging Real-time and Asynchronous Tasks
- Building Trust A major challenge is bridging real-time interaction with asynchronous task completion, where a model might research or write code for hours before returning a solution [00:15:39]. The core bottleneck here is trust [00:16:00]. This can be addressed by providing humans with new collaborative affordances to verify and edit model outputs, and by enabling real-time feedback for model self-improvement [00:16:02].
Multi-agentic and Multiplayer Collaboration
- Navigating Complex Interactions A new design challenge involves scaling interfaces to support multi-agentic and multiplayer collaboration, where multiple people can join a document or multiple AI agents (e.g., a model critic or editor) can interact simultaneously [00:17:57].
Creative Co-Innovation and Co-direction
The future vision involves AI agents becoming “co-innovators,” blending reasoning, tool use, and long context with creativity, enabled through human-AI collaboration [00:10:27]. This requires creating new affordances for humans to collaborate more effectively with AI [00:10:52]. Ultimately, the goal is for co-innovation to occur through “co-direction” with models, leading to new novels, films, games, science, and knowledge creation [00:23:31].
The ability to use highly reasoning models to distill knowledge into smaller, faster models, and to synthetically generate new data for post-training and reinforcement learning environments, creates a rapid iteration cycle for product development [00:11:30]. This also enables the creation of new classes of tasks, such as simulating different users for multiplayer collaboration [00:11:57].