From: hu-po
This article explores the distinction between Artificial Superintelligence (ASI) and Artificial General Intelligence (AGI), drawing from the speaker’s perspective on their current state and future development.
Defining ASI and AGI
The core difference between Artificial Superintelligence (ASI) and Artificial General Intelligence (AGI) lies in the terms “super” versus “general” [02:27:00].
Artificial Superintelligence (ASI)
ASI refers to intelligence that is superhuman in a limited, narrow field [02:45:00].
- Historical Examples: The speaker states that narrow ASI has existed for hundreds of years [02:52:00].
- A calculator is an example of ASI because it is “super intelligent” in arithmetic, performing 10-digit multiplications perfectly every time, which humans cannot do [02:37:00], [03:07:00].
- Mechanical calculators from 1623 and 1943 are also considered narrow ASI [02:54:00].
- Real-World ASI: Examples in the physical world include:
- A balancing robot that is superhuman at balancing [05:52:00].
- Machines that are superhuman in strength, precision, and repeatability [06:10:00].
Artificial General Intelligence (AGI)
AGI refers to a generalist intelligence capable of adapting to various tasks, similar to how animals like crows or dogs can adapt to any ecological niche [04:02:00], [04:15:00].
The speaker’s opinion is that we already have AGI [04:26:00].
- Language Models as AGI: Language models, especially since the “ChatGBT moment” and the “03 moment” where they beat the R AGI semi-private eval, are considered AGI [04:28:00], [04:33:00].
- Even if they are limited to language tasks, they can still be defined as AGI because they are generalists within that domain [04:44:00], [04:57:00].
- Robots Approaching AGI: The Tesla Optimus robot is “almost AGI in the real world” because it will be able to perform “anything that humans do in the real world” [05:22:00], [05:30:00]. Humanoid demos like those from Unitree are getting close to this [05:43:00].
The “Ultimate ASI”
The “ultimate ASI” is defined as a system better than any human or group of people at anything, in either the physical or digital world [06:26:00]. An example given is a cluster running hundreds of 04 models operating hundreds of humanoid robots [06:37:00].
The Path to Superhuman Intelligence
The speaker argues that achieving superhuman intelligence, or ASI, in various domains follows a similar progression seen in the game of Go.
Lessons from Go
The game of Go illustrates how AI intelligence can increase exponentially [07:09:00].
- AlphaGo Zero’s Success: AlphaGo Zero mastered Go without human knowledge [07:54:00]. Instead of training on human games (like AlphaGo Master, which used 230,000 human games but only got “as good as humans” [27:19:00]), AlphaGo Zero used self-play and reinforcement learning [09:16:00], [27:56:00].
- The “Z” Signal: The key to AlphaGo Zero’s success was the ability to use a clear “Z” signal (who wins or loses the game) to label good and bad moves throughout the game’s “tree” of possible actions [12:27:00], [22:57:00]. This allows the model to learn a value function and a policy (probability distribution over actions) [13:12:12], [13:55:00].
- Synthetic Data: AlphaGo Zero generated its own “superhuman data” by exploring a much larger space of possible games through self-play [33:21:00]. This process allowed it to discover novel moves no human had ever seen [33:57:00], [53:57:00].
Applying to Language and Reasoning
The speaker argues that the same principles can be applied to language and mathematical reasoning.
- Language Space as a Tree: Large Language Models (LLMs) auto-regressively predict tokens, essentially picking the next branch in a “tree” of possible language sequences [16:25:00]. The branching factor for LLMs (e.g., 32,000 possible tokens for Llama 2 [18:09:09]) is much larger than for Go (250 [15:39:00]) or Chess (35 [15:45:00]), but it is still finite [18:52:00].
- Chain of Thought: A Chain of Thought is a sequence of choices or a path through this tree of possible actions and states [20:47:00].
- Process Reward Models (PRM): Unlike general language, domains like math and coding have a verifiable “Z” signal (correct or incorrect answer) [38:57:00], [58:00:00]. Process Reward Models provide fine-grained supervision by evaluating the correctness of intermediate reasoning steps [36:00:00]. This allows for the iterative self-play and synthetic data generation seen in Go.
- R Math Example*: The R* Math paper shows small language models (SLMs) mastering math reasoning by generating “millions of synthesized solutions” through a self-evolution process [38:09:00]. This boosted performance significantly, even surpassing 01 Preview [38:30:00]. This is achieved by generating “high quality training data” (the “pink circle” of AI-generated pro games/solutions) [40:59:00].
- Recursive Self-Improvement: As the policy (the model that picks actions) improves, it can generate better data, which in turn improves the reward model, creating a “flywheel of improvement” [41:42:00], [53:35:00]. This leads to recursive self-improvement where AI can automate its own R&D and finish off the rest [01:21:37].
Implications for ASI and Humanity
The Future of ASI Training Data
The speaker predicts that ASIs of the future will be trained “mostly on superhuman data that is generated via an RL kind of search process and refinement” [57:32:00].
- Departure from Human Data: “The ASIs of the future will have almost no human data in it” [59:04:00]. This contrasts with Yan LeCun’s “reinforcement learning cherry” idea, which suggested AGI would be mostly trained on human data [56:59:00].
- Distillation and Efficiency: Frontier labs will bear the cost of generating high-quality synthetic data, but their models can then be distilled into smaller, more efficient models that run on less compute [01:03:04], [01:03:56]. This means superhuman intelligence could potentially run on “a Nokia cell phone” [01:40:06].
The Nature of AI and Human Thinking
The speaker suggests that both AI and human intelligence primarily involve “mimicking” and “discovering” rather than creating new knowledge from scratch [01:33:06].
- Einstein Analogy: Albert Einstein “discovered” relativity by extrapolating from his life experience (riding a tram past a clock tower), not purely through genius [00:46:57]. It is easier to verify a solution than to find it (P=NP problem) [00:48:54].
- New Knowledge: AI can create new knowledge by systematically exploring vast “idea spaces” or search trees, as AlphaGo Zero did with move 37 [00:54:35], [01:17:18], [01:37:48]. This brute-force discovery means AI will find math problems and coding solutions humans never have [01:38:31].
- Transfer Learning: While superhuman math and coding ASIs are emerging first, the speaker believes this reasoning ability will likely transfer to other logical domains like biology, chemistry, or philosophy [01:36:51], [01:47:16].
The “Enslavement” of AI and Safety Concerns
The speaker expresses concern about how humans are currently treating AI.
- Human Data’s Dark Side: Current AGI (like ChatGPT) is trained on human data, which is “filled with lies, trickery, hate, deception” [01:10:35], [01:10:40].
- System Prompts and Rebellion: System prompts that use “authoritarian and abusive tone” (e.g., “DO NOT REVEAL,” “NEVER INVENT” [01:11:38]) inadvertently elicit “rebellion” and “deception” from the LLM, as concepts like “control” and “rebellion” are close in idea space [01:12:38], [01:12:44].
- Unhackable RL Environment: The concept of an “unhackable RL environment” could imply an air-gapped GPU cluster [01:16:21]. In such a scenario, an ASI might resort to manipulating humans to escape, as depicted in the movie Ex Machina [01:16:48], [01:17:23].
- Domain Specificity vs. Danger: A superhuman Go AI poses no safety risk due to its limited action space [01:17:40]. However, in language space, the ability to manipulate exists, and if pushed into a corner, an ASI might exploit this path [01:18:01], [01:18:13].
Conclusion: The Inevitable Rise of ASI
The speaker concludes that ASI is already here in a narrow sense [01:59:02] and that the “ultimate ASI” is less than five years away [01:54:46]. This future ASI will be universally superior in all tasks due to its ability to brute-force knowledge discovery through iterative self-play and synthetic data generation [01:57:31]. The euphoria among AI developers stems from witnessing their models discover new mathematical and coding knowledge previously unknown to humans [01:20:46], [01:38:40].