From: redpointai
Bob McGrew, former Chief Research Officer at OpenAI, discusses the current state, challenges, and future potential of video models and robotics models [00:00:13]. He highlights that while progress in AI models might appear slow from the outside, significant advancements are underway internally within major labs [00:01:00].
Advancements in Video Generation Models
The modality that has “resisted” integration into main multimodal models for a long time is video [00:18:54].
Sora’s Role and Capabilities
Sora has been among the first to demonstrate advanced video generation capabilities [00:19:00]. Two key aspects distinguish video from other modalities like images:
- Extended Sequences and User Interface: Video involves an extended sequence of events, not just one prompt [00:19:49]. This necessitates a comprehensive user interface to craft a story that unfolds over time [00:19:54]. Sora’s product team has focused on developing a storyboard capability, allowing users to set checkpoints to guide the generation process [00:21:54].
- High Costs: Video generation is inherently expensive, both in terms of training and running these models [00:20:18].
Sora’s quality is high, and its distribution is broad, being available to OpenAI Plus and Pro account holders, setting a high bar for competitors [00:20:55].
Future Outlook for Video Models
The progress in video models is expected to be very direct, similar to the trajectory of LLMs [00:21:23]:
- Improved Quality: While instantaneous quality is already good, the next generations of models will focus on achieving extended coherent generations, moving from seconds of video to potentially an hour [00:21:49].
- Reduced Cost: Similar to how GPT-3 quality tokens became 100 times cheaper, the cost of generating high-quality, realistic videos with Sora is expected to become practically nothing [00:22:25].
The dream of a full-length, AI-generated movie that audiences genuinely want to watch could be realized in about two years [00:23:08]. The key will be directors leveraging video models to exercise their creative vision to produce content unachievable through traditional filming [00:23:17].
Advancements in Robotics Models
Bob McGrew initially explored robotics in 2015, believing it was five years away from broad adoption, a prediction he now corrects to “now” [00:24:50]. He anticipates widespread, though somewhat limited, robotics adoption in about five years [00:25:00].
Impact of Foundation Models on Robotics
Foundation models represent a significant breakthrough in robotics, enabling quicker setup and crucial generalization capabilities [00:25:14].
- Vision to Action: The ability to use vision and translate it into plans of action comes almost “for free” with foundation models [00:25:31].
- Natural Language Interaction: The ecosystem has developed to the point where users can simply talk to robots, making interaction much easier than typing commands [00:26:01].
Key Challenges
- Reliability: The most immediate challenge for agents, especially those taking actions in the real world (e.g., buying things, sending messages), is reliability [00:08:57]. Achieving 90% to 99% reliability requires an order of magnitude increase in compute, and going from 99% to 99.9% requires another [00:09:51].
- Simulation vs. Real World Learning:
- Simulation Advantages: Simulators are efficient for training and good at handling rigid bodies [00:26:48].
- Real-World Necessity: However, simulators struggle with “floppy” materials like cloth or cardboard [00:27:14]. For general-purpose robotics, real-world demonstrations are currently the only effective approach [00:27:31].
Future of Robotics Adoption
Widespread consumer adoption of home robots is still a distant prospect due to safety concerns (robot arms can be dangerous) [00:28:10]. However, in work environments like retail or warehouses, significant deployment of robots is expected within five years [00:28:42]. For example, Amazon warehouses already utilize robots for mobility and are working on pick-and-place tasks [00:28:48].
Convergence of AI Models
Frontier labs are expected to continue developing general purpose models that perform optimally across various data types and applications [00:29:41]. Specialization in AI models primarily offers price-performance advantages [00:29:55]. Companies can fine-tune smaller models using data generated by the best frontier models, resulting in significantly cheaper operations, though with slightly less off-script capability [00:30:21].
The overarching sentiment is that progress in AI will continue to be very exciting, dynamic, and constant, albeit with evolving forms and challenges [01:06:55].