From: jimruttshow8596
David Krakauer, President of the Santa Fe Institute, and Jim Rutt, host of the Jim Rutt Show, discuss the nature of artificial intelligence (AI), its current capabilities and limitations, and the various emergent risks associated with its development [00:00:31]. They delve into the distinction between theory-driven and data-driven science, which provides a framework for understanding the unique properties of modern AI models [01:32:50].
Theory-Driven vs. Data-Driven Science
The conversation begins by differentiating between fine-grained paradigms of prediction, which involve large models with practical value, and coarse-grained paradigms of understanding [02:51:56]. Historically, physical science was fortunate to have a conjunction where fundamental theories were also highly useful [03:04:47]. However, there might be a bifurcation in science where understanding and utility diverge for the same topic [03:31:00].
Examples where data science has outperformed theory include:
- AlphaFold’s protein folding prediction: It achieved breakthroughs with massive computation but offered “zero theoretical insight” into the underlying mechanisms [04:04:36].
- Transformer Technologies for Language Models: Brute-force data and computation produced incredibly powerful language models, unlike traditional computational linguistics which struggled with the “marginally lawful nature of human language” [05:36:00]. These models provide little initial insight into mechanisms [06:00:23].
Early neural networks, in the 1940s, were deductive frameworks related to logic, not induction or large datasets [06:33:00] [10:19:00]. The modern deep learning paradigm, reliant on massive data and computation, emerged much later [10:50:00]. The “AI winter” in the 1960s was partly due to the belief that deep neural nets were impossible to train due to “too many parameters” [12:20:00]. Today, models like GPT-4 have trillions of parameters, far exceeding human comprehension [13:00:00].
Superhuman Models and the Uncanny Valley
Krakauer refers to these as “superhuman models” because they demonstrate that by adding parameters beyond a certain point (the “statistical uncanny valley”), performance improves again [14:49:00]. This phenomenon suggests that complex domains have high-dimensional regularities that these models can uncover [15:21:00].
A “miracle of ultra-high dimensionality” allows gradient descent to work on many more problems than previously thought, as there’s always a dimension pointing “down” in such vast spaces [16:03:00]. This also relates to adaptive computation, where the local minima problem in high-dimensional landscapes is ameliorated rather than worsened [17:45:00].
Limitations of Current AI
Despite their power, these large models have significant limitations:
- Computational Cost: GPT-4, with trillions of parameters, consumed “hundreds of thousands of kilowatt hours” and “millions of dollars” for training [33:58:00].
- Arithmetic Incapacity: Large Language Models (LLMs) struggle with basic arithmetic, performing worse than a 50-year-old calculator with 1KB of memory [34:10:10]. This indicates they are “certainly not sentient” [35:27:00] and are “looking less and less like a truly intelligent system” [38:14:00].
- Data Efficiency: Humans and animals are vastly more efficient learners, requiring five to six orders of magnitude less data than deep learning systems to achieve similar results [23:31:00]. This suggests that biological intelligence uses “algorithms very, very different than deep learning” [23:50:00].
- Lack of Internalization: Human intelligence can internalize functions like arithmetic, making them integrated capabilities, whereas current AI often “outsources capabilities to tools” [37:38:00].
- Lack of Heuristic Induction: AI is not particularly good at the “explicit creation of heuristics,” which humans excel at for navigating complex problems efficiently [39:04:00].
- Creativity and Discovery: While very capable at generating complex outputs, LLMs are “pure herd” [54:49:00], acting as “libraries” or “quick and dirty analysts and synthesizers” [57:18:00]. They excel at “compositionality” within established domains but are unlikely to produce breakthroughs like Einstein’s theory of relativity because they “don’t have any geometry” [58:54:00]. True scientific revolutions often arise from “bandwidth limitation and constraints” [59:31:00], not raw computational power or vast data.
Existential Risks of AI
Krakauer expresses skepticism about the “hand-wringing slightly neurotic narrative” regarding imminent AGI risks, viewing much of it as a “marketing ploy” to secure resources [01:17:34]. However, he acknowledges significant risks:
- Misuse of Narrow AI: The most obvious risk is individuals or states using narrow AI for malicious purposes, like China’s advanced police state for surveillance [01:19:03]. This is a risk to the world, though “probably not existential” [01:25:00].
- The “Idiocracy Risk”: As AI performs more tasks, humans may “stop investing in achieving those intellectual skills,” potentially leading to a societal devolution where people forget how to do basic things [01:19:57]. This risk is amplified by the fragility of the technosphere, where events like a major solar flare could cripple the grid, leaving a de-skilled populace in “deep doo-doo” [01:20:25].
- Acceleration of “Game A”: AI could accelerate unsustainable “Game A” (the current status quo of exponential growth and resource consumption), potentially shortening the timeline for civilizational collapse from 80 years to 40 years [01:21:20].
- “Flood of Sludge”: The proliferation of AI-generated low-quality content, spam, and fake news is already evident, overwhelming attention resources [01:28:11].
Historical Precedent and Adaptation
Krakauer emphasizes looking at historical precedents for managing new technologies, contrasting them with “super Draconian proposals” for AI regulation [01:24:34]:
- Genetic Engineering: Despite the power of recombinant DNA (1970s) and CRISPR (1980s), self-moratoriums (Asilomar conference) and existing drug regulations were effective, without explicit “regulation” on the technology itself [01:22:28].
- Nuclear Weapons: Within years of the first atomic bomb, ideas for managing nuclear materials emerged, followed by non-proliferation treaties after the Cuban Missile Crisis [01:23:25].
- Automobile: Over a century, regulations (traffic lights, seatbelts, drunk driving laws) led to a “95% reduction in fatality per mile” [01:23:41].
These examples suggest that “small regulatory interventions” can effectively minimize true risks [01:24:22].
Counterarguments to Historical Optimism
Jim Rutt notes that AI is “moving way faster” than previous technologies, potentially in the “very steep part of the curve,” with implications that could be “staggering” [01:25:03]. The low cost of AI development (a small model for $1,000, full model for a few million dollars, decreasing by Moore’s Law) and escaping confinement of models are significant concerns [01:26:41]. The fundamental technology “isn’t hard at all” and is publicly available [01:27:13].
Solutions and Adaptations
- Cognitive Synergy: Ben Goertzel’s concept of combining deep learning, genetic algorithms, symbolic AI, and other tools through a “hypergraph-based probabilistic logic language” could address current AI limitations like arithmetic [01:35:36].
- Parsimonious Science: Using deep neural networks as “pre-processors for parsimonious science” by sparsifying their complex encodings and then applying symbolic regression to derive simple formulas or “equations of motion” could be a “new way of doing science” [02:52:00] [03:06:00].
- The “Crackauer Solution” / Info Agents: A “natural evolutionary reaction” to the “flood of sludge” could be the development of “info agents” – AI-powered tools that surround users to filter and curate electronic information. These agents could summarize content, rate relevance, and even build “constructive networks of mutual curation” [01:28:48]. This is seen as “God’s own spam filter for everything electronic” [01:31:31].
- Ignoring Technology: A “Paleolithic alternative” is to counter technology by “ignoring it” and radically simplifying life decisions, such as avoiding social media platforms altogether [01:32:50].
- Human Adaptation: Humans have a “track record for dealing with all kinds of dangerous stuff,” from fire to language, by incrementally learning how to control and adapt to them [01:33:36]. This suggests that society might bifurcate, with some adapting through hygiene and constraint, while others remain vulnerable to “attention hijacking” [01:34:04].