From: aidotengineer
Travis Fry Singer, Technical Director of AI ETHLite, discusses the “coherency trap,” explaining why prompting Large Language Models (LLMs) feels like magic but isn’t true intelligence [00:00:00]. His top-down talk aims to explain the perceived effectiveness of LLMs despite their lack of intelligence, intent, or desire [00:00:14].
Initial Encounters with LLMs
In November 2022, upon the release of GPT-3.5, the speaker was initially disappointed [00:00:37]. While it offered some advancements, such as improving emails, its understanding was brittle, its fluency often collapsed at edge cases, and it suffered from prompt sensitivity and context limits [00:00:50].
However, the release of GPT-4 in January 2023 marked a significant shift [00:01:22]. The output felt like understanding, transcending mere text generation to display something eerily similar to comprehension [00:02:17]. This experience was shared by others, including Microsoft Research, who published “Sparks of Artificial General Intelligence: Early Experiments with GPT-4” [00:01:37]. The speaker felt there was an unexplored space between “very dumb chatbots” and AGI [00:02:33].
Experimental Exploration of LLM Capabilities
To understand this shift, the speaker, an engineer and scientist, began conducting experiments [00:02:40].
Vibe Coding and Utility Development
Starting with a live stream, he engaged in “chat assisted programming” (also known as “vibe coding” or “chat oriented programming”) with ChatGPT [00:03:02]. This process, though effort-intensive for a few hundred lines of code, served as a prototype for AI pair programming [00:03:34].
From these sessions, he developed Webcat, a Python Azure function that scraped web pages for content [00:04:00]. This was crucial because early ChatGPT-4 lacked internet access, making Webcat useful for feeding web content to the model [00:04:14].
Collaborative Projects
Using Webcat and working collaboratively with AI, the speaker built his blog, AIBuddy.software [00:04:33]. The AI selected the Ghost platform and provided instructions for setting up the template [00:04:59]. This collaborative approach proved successful for generating content [00:05:36].
Expanding on creative collaboration, he used LLMs to produce a concept album titled “Mr. Fluff’s Reign of Tiny Terror,” a “feline metal album” [00:06:00]. ChatGPT was used for lyrics and music composition, and its image generation capabilities were employed for consistent visuals [00:06:24]. Despite the humorous concept, the project garnered over 3,000 views and positive feedback on YouTube, demonstrating AI’s ability to assist in creating valuable content beyond his individual capabilities [00:07:01].
The AI Decision Loop
These successful collaborative experiences led the speaker to investigate whether concepts like decision intelligence and pairing behaviors could explain the LLM’s perceived utility [00:07:33]. He built an analysis tool using his “vibe coding” skills to parse his ChatGPT history, looking for qualitative and quantitative metrics related to decision intelligence and pairing [00:08:19]. This analysis culminated in a 21-page research paper available on his AIBuddy site [00:08:50].
The findings revealed a cyclical process, which he termed the AI Decision Loop:
- Frame: Define the problem and provide context (akin to prompt engineering) [00:09:35].
- Generate: Produce outputs, potentially multiple options [00:09:45].
- Judge: Evaluate the quality and fit of the output [00:09:57].
- Validate: Optionally, ensure external requirements are met [00:10:04].
- Iterate: Refine the prompt to improve the output based on what was right or wrong [00:10:15].
A less academic version of this process is the “nudge and iterate” framework: Frame, Generate, Judge, Iterate [00:10:38]. This iterative prompting and nudging was found to be at the heart of reliable LLM interactions [00:10:55].
Coherence Theory
Despite these mechanics, the question remained: why did LLMs work so well if they weren’t intelligent? The speaker turned to the concept of coherence from natural language processing, aligning with research like Anthropic’s work on feature superposition and concept circuits [00:11:12].
What is Coherence?
Coherence is described as a system property, not a cognitive one, serving as the “infrastructure that thought navigates” [00:11:59]. It has four key properties:
- Relevant: The model’s output feels topical, connected, and purposeful [00:12:24].
- Consistent: The model maintains a singular tone, terminology, and structure across multiple turns [00:12:31].
- Stable: The model can withstand pressure and interrogation, not collapsing but firming up or course-correcting [00:12:51]. This stability was a key differentiator from earlier models like GPT-3.5 [00:13:15].
- Emergent: It’s an emergent property. For example, GPT-4o, without explicit training, can diagnose swine disease or certain cancers through “coherent pattern alignment” [00:13:21].
Mechanics of Coherence
Traditional neural networks typically store concepts in single neurons [00:13:53]. However, research suggests that in LLMs, superposition allows the network to represent complex ideas with fewer parameters, packing more nuance into the same space [00:14:06]. As context accumulates, the network “teases apart the relevant meaning and collapses that ambiguity into a coherent output” [00:14:20]. An idea isn’t stored in a single neuron or set of neurons; rather, a set of neurons can hold multiple ideas [01:14:43].
This implies that meaning isn’t retrieved but constructed on demand from “distributed sparks of possibility” [00:14:49]. Prompts act as “force vectors” in the high-dimensional latent space of the AI model, setting a specific direction that the AI aligns patterns to [00:14:59]. When a prompt is given, external context activates conceptual clouds (sub-networks) in the latent space that are relevant to the query [00:15:43]. These specialized patterns emerge during training and activate when the context calls for them, merging concepts to create new, coherent ideas [00:16:19].
Engineering for Coherence, Not Intelligence
This understanding leads to a new kind of utility for LLMs, focusing on their ability to recreate the essence of an idea and combine multiple essences to create something new [00:17:00]. When LLMs “hallucinate,” they are creating a compelling pattern based on essence reconstruction, not fact-checking or intelligence [00:17:20].
Hallucinations as Features
From an engineering perspective, hallucinations are seen as an indicator of coherence, as the model fills gaps predictably following an internal logic [00:17:43]. They are an emergent system feature, not a bug [00:18:00].
Retrieval Augmented Generation (RAG)
To mitigate hallucinations, Retrieval Augmented Generation (RAG) is used [00:18:10]. RAG fragments act as “factual anchors” or “structural scaffolding” [00:18:16]. Dense, relevant context from RAG creates “contextual gravity” that pulls the concept in the right direction, providing infrastructure for human intent to navigate the latent space [00:18:20].
Three-Layer Model for LLM Systems
The speaker proposes a three-layer model for LLM systems:
- Layer One: The Latent Space [00:18:43]: The internal model structure (concepts, weights, activations).
- Layer Two: The Execution Layer [00:18:51]: Tools, APIs, and retrieval mechanisms that bring external context for Layer One.
- Layer Three: The Conversational Interface [00:19:00]: Where human intent (thought) passes to the machine, grounding Layers One and Two in actionable value.
Design Principles for Coherent Systems
- Prompting as Interfaces: Prompts should be viewed as components in a system, not one-off interactions [00:19:19].
- RAG for Coherency Anchors: Use RAG to steer generation with dense, relevant context [00:19:26].
- Design for Emergence, Not Control: Embrace the non-deterministic nature of LLMs, building around the Frame, Generate, Judge, Iterate loop [00:19:34].
- Avoid Fragile Chains: Long reasoning chains can break coherence; keep chains modular and reinforce context at each point [00:19:42].
- Watch for Breakdowns: Monitor changes in tone, structure, or flow as early signs of the model losing context, using them as debugging points [00:19:53].
In conclusion, LLMs are best understood as “high-dimensional mirrors” [00:20:17]. They don’t understand or think but resonate through structure, reflecting back patterns that can appear sharper than the input [00:20:26]. Their superpower is coherence, not intelligence, and the true “magic” lies in the collaborative dance between human intent and the model’s structural resonance [00:20:34].