Decision Intelligence and Pair Programming with AI

From: aidotengineer

Travis Fry Singinger, Technical Director of AI ETHLite, has explored the perceived “magic” of large language models (LLMs) and why prompting feels powerful despite LLMs lacking true intelligence or intent [00:00:00]. His journey, marked by experiments and analysis, led to a theory explaining the effectiveness of LLMs through “coherency” and the development of frameworks for more reliable human-AI collaboration [00:00:24].

The Uncanny Shift: GPT-3.5 to GPT-4

Initially, the release of GPT-3.5 in November 2022 brought disappointment due to its brittle understanding, prompt sensitivity, and surface-level fluency that collapsed at edge cases [00:00:35]. However, the subsequent release of GPT-4 around January 2023 presented an “uncanny moment” where the output felt like understanding, transcending mere text generation [00:01:21]. This shift was widely noticed, with Microsoft Research publishing a paper on “sparks of artificial general intelligence” [00:01:37]. Fry Singinger felt there was a space between “very dumb chatbots” and AGI that needed further exploration [00:02:33].

Early Experiments in AI-Assisted Programming

Driven by his background as an engineer and scientist, Fry Singinger began conducting experiments to understand this new phenomenon [00:02:43]. He started with live stream “chat assisted programming,” also known as “chat oriented programming” or “vibe coding” [00:03:02]. While initially requiring significant effort to produce usable code, these sessions evolved into prototypes for AI pair programming [00:03:34].

A key utility developed during this phase was Webcat, a Python Azure function that scraped web pages to provide content to ChatGPT-4, which at the time lacked internet access [00:04:00]. This utility was crucial for overcoming the model’s limitations and making it useful for real-world problems [00:04:10].

Collaborative Projects and Design Process Improvements with AI

The success of Webcat led to deeper collaborative experiments:

• AIBuddy.software Blog: Fry Singinger collaborated with AI to build his blog, AIBuddy.software, demonstrating a “collaborative essence” rather than treating AI as a source of all answers [00:04:33]. The AI assisted with platform selection, instructions, and template building [00:04:59]. Webcat was used to pull in article snippets for discussion and blog generation [00:05:25].
• “Mr. Fluff’s Reign of Tiny Terror” Concept Album: Leveraging his music background, he aimed to produce an AI-assisted feline metal concept album [00:06:00]. This project used ChatGPT for lyrics and music composition, alongside image editing, noting the new ability of ChatGPT’s image generation to refine outputs rather than acting like a “slot machine” [00:06:21]. The album, uploaded to YouTube without promotion, garnered over 3,000 views and significant positive feedback within a month, demonstrating the ability to create valuable content beyond individual capabilities [00:07:01].

These projects indicated that LLMs could help create a single, coherent concept across various domains [00:07:33].

Discovering the AI Decision Loop

To understand the underlying behaviors, Fry Singinger hypothesized that traditional decision intelligence and pair programming might hold the “secret sauce” [00:07:45]. He built an analysis tool using his “vibe coding” skills to process his ChatGPT history, looking for qualitative and quantitative metrics related to decision intelligence and pairing behaviors [00:08:19].

This research culminated in a 21-page case study of AI interview agent development outlining the technique and prompts used, available on his AI Buddy site [00:08:50].

The study revealed the AI Decision Loop, which he condensed into a four-step process, or “nudge and iterate” framework [00:10:37]:

1. Frame: Define the problem and context (prompt engineering) [00:09:35].
2. Generate: Produce one or many outputs [00:09:45].
3. Judge: Evaluate the quality and fit of the output. This includes validating against external requirements and asking further questions [00:09:57].
4. Iterate: Refine the prompt and nudge the model based on what was right or wrong, continuing the cycle to produce more reliable outputs [00:10:15].

“This isn’t a once-off prompt and accept type scenario.” [00:10:17]

Following this loop led to significantly better outcomes in his interactions with AI [00:11:02].

Coherency: The LLM Superpower

Despite the effective mechanics, the question remained: why do LLMs work so well if they aren’t intelligent? Fry Singinger proposes a “coherence theory” based on natural language processing and research into feature superposition and concept circuits [00:11:25].

Coherence is a system property, not a cognitive one [00:12:01]. It’s the infrastructure thought navigates [00:12:06]. It has four key properties:

• Relevant: Outputs feel topical, connected, and purposeful [00:12:24].
• Consistent: The model maintains a singular tone, terminology, and structure across multiple turns [00:12:31].
• Stability: The model can withstand pressure and interrogation, firming up or course-correcting rather than collapsing [00:12:51]. This was a significant improvement over earlier models [00:13:15].
• Emergent Property: Coherence allows for abilities not explicitly trained, like GPT-4o diagnosing swine disease through “coherent pattern alignment” [00:13:21].

Mechanics of Coherence: Force Vectors in Latent Space

Instead of concepts being stored in single neurons, LLMs use superposition, representing complex ideas with fewer parameters by packing more nuance into the same space [00:14:13]. Meaning is “constructed on demand from distributed sparks of possibility” [00:14:52].

A prompt acts as a “force vector” in the high-dimensional latent space of the AI model [00:14:59]. This vector sets a specific direction, causing the AI to align patterns [00:15:10]. When a prompt is given, relevant “conceptual clouds” or sub-networks in the latent space activate and merge, causing new ideas to emerge based on the combined context [00:15:43].

This leads to a new kind of utility: LLMs don’t just compress and retrieve knowledge; they recreate the essence of ideas and combine multiple essences to create something new [00:17:00]. This explains why hallucinations can still feel correct—they are creating a compelling pattern, not fact-checking [00:17:20].

Building for Coherence, Not Intelligence

Understanding LLMs as coherent systems rather than intelligent ones changes the engineering approach:

• Hallucinations as Indicators: Hallucinations are a system feature, an emergent behavior of coherence, indicating where the model fills gaps due to insufficient information [00:17:43]. They follow internal logic and coherency vectors [00:17:55].
• Retrieval Augmented Generation (RAG) as Factual Anchors: RAG fragments act as “factual anchors” providing “contextual gravity” to steer generation towards reality, forming a “structural scaffolding” for coherence [00:18:10].
• Three-Layer Model:
  1. Latent Space: Internal model structure (concepts, weights, activations) [00:18:43].
  2. Execution Layer: Tools, APIs, and retrieval mechanisms that bring external context to Layer 1 [00:18:51].
  3. Conversational Interface: Where human intent and thought are passed to the machine, grounding Layers 1 and 2 in human needs [00:19:00].
• Design Principles for Coherence:
  • Prompts as Interfaces: Treat prompts not as one-off commands but as components within a larger system [00:19:19].
  • RAG for Grounding: Dense, relevant context acts like gravity, pulling output towards reality [00:19:26].
  • Design for Emergence, Not Control: LLMs are not deterministic; build around the “Frame, Generate, Judge, Iterate” loop [00:19:34].
  • Avoid Fragile Chains: Long reasoning chains often break coherency. Keep chains modular and reinforce context at each point [00:19:42].
  • Watch for Breakdowns: Early signs like breakdowns in tone, structure, or flow indicate the model is losing context. These serve as debugging points to adjust chunk size, vector database, or integrate other tools [00:19:53].

In conclusion, LLMs are best understood as “high-dimensional mirrors” that resonate with human intent through structure, not thought [00:20:17]. Their superpower is coherence, and the “magic” lies in the collaborative dance between human and machine [00:20:34]. The focus should shift from chasing intelligence to designing for structured resonance [00:20:40].