Tubegraph

Regulatory and safety issues in AI model development

7 min read

From: redpointai

Introduction to AI Safety and Regulation Concerns

The development and deployment of AI models, particularly large language models (LLMs), have brought forth significant concerns and considerations for AI safety and regulation [00:00:19]. While the goal is to build smaller, cheaper, and more affordable models and applications [00:00:05], the industry faces challenges in AI model training and deployment related to costs, market risks, and the strategic implications of powerful AI [00:03:59].

Applying Lean Startup Principles in AI Development

Eric Reese, author of The Lean Startup, notes that many lessons from his book apply to AI, but the industry often deviates [00:00:50]. He observes large funding rounds and significant spending on models and compute before market interaction [00:00:57]. This approach is partly due to AI’s ability to create “magical” demos, leading developers to believe they don’t need extensive customer testing [00:01:36].

The "SAS stack" problem:

Many AI companies have directly copied the Software-as-a-Service (SaaS) stack model, assuming its principles apply unchanged to AI [00:02:23]. This often pushes the product-market fit question to deeper layers of the value chain, creating a disconnect between the model builder and the end-user [00:02:45]. If the AI stack operates differently from traditional software, this approach could lead to “a lot of carnage in applications” [00:03:16].

Eric Reese emphasizes the importance of understanding the “end end end customer” regardless of one’s position in the stack [00:03:25]. He draws parallels to physical manufacturing and deep infrastructure projects, where physical constraints and operating costs are significant [00:03:47].

The Role of Moats and Defensibility

There’s debate about whether AI companies should prioritize “moats” (defensibility) from the start [00:04:46]. While large platforms (like OpenAI) theoretically could replicate anything a smaller startup does, their focus is limited [00:05:31]. The historical lesson suggests that fast, adaptable startups can “pick up dimes in front of a steamroller” by addressing niche use cases [00:05:51]. However, one misstep could be fatal [00:06:03].

Strategic Humility:

The current uncertainty in AI, regarding future model capabilities and societal impact, makes confident long-term predictions difficult [00:07:58]. This necessitates building companies with mechanisms for rapid iteration and the ability to pivot, constantly testing assumptions [00:07:37]. This ability to adapt and get feedback has “never been more important” [00:08:33].

The For-Profit R&D Lab Model: Answer AI

Jeremy Howard and Eric Reese co-founded Answer AI, a for-profit R&D lab, aiming to build the “Bell Labs of AI” [00:01:01]. Their mission is to maximize the public benefit of AI by making it more accessible, particularly through language modalities [00:10:01]. This addresses Jeremy’s concern that AI could lead to power centralization and decreased opportunities [00:13:07].

This R&D lab model is distinct from traditional startups [00:14:35]. It embraces an integrated “research and development” (R&D) approach, unlike the modern hyperspecialization that separates research from practical application [00:15:35]. The core belief is that the best research occurs when the researcher is directly “coupled to the application” and customer needs [00:16:04].

Data Center Anecdote:

In one instance, industrial researchers focused on making data center technology energy-efficient, requiring “fundamental physics breakthroughs” [00:17:15]. However, taking an MVP to the data center builder revealed that their primary concern was the physical footprint (how many racks could be squeezed in), not operating cost [00:17:21]. This misaligned goal highlighted the importance of customer feedback driving scientific inquiry [00:17:55].

Critiques of Current AI Model Development

Jeremy Howard views much of the current investment as “overinvestment in training Foundation models from scratch” on expensive hardware [00:25:40]. There’s an “underinvestment on like the real world which is resource constrained” [00:25:49]. Bringing down the cost of AI, even if it’s “just the same thing but cheaper,” is critical for accessibility and wider use [00:26:42].

Cost as a "Difference in Kind":

Making inference costs cheaper doesn’t just improve margins; it makes previously impossible applications feasible [00:27:21]. Lowering costs could enable “continuous fine-tuning” of individual models on inexpensive virtual machines, allowing for hyper-personalization and memory in AI agents, unlike current “amnesiac models” [00:28:45]. This focus on “manufacturability” and deployability, rather than just “splashy demos,” is crucial for moving towards deployed products [00:29:55].

Regulatory and Policy Implications of AI

The California Bill SB147

Jeremy Howard expresses concern over California’s proposed bill SB147, which aims to place limitations and regulatory checks on training foundation models [00:38:22]. While created with good intentions, Jeremy believes it would be “uneffective” and “likely to cause the opposite” of its intended result, leading to a “less safe situation” [00:40:01].

The Dual-Use Technology Problem:

The core issue, according to Jeremy, is that an AI model is a “purely kind of dual use technology” like a pen, paper, or calculator [00:40:32]. If a model is released, it can be fine-tuned or prompted to do “whatever the hell you like” by the user [00:40:55]. It’s impossible to “ensure the safety” of the raw model itself [00:41:17].

Imposing such regulations in practice means that models in their raw form cannot be released [00:41:37]. Instead, only “products on top of them” can be offered (e.g., a service that writes things down for you, not a pen and paper) [00:41:39]. This creates a situation where underlying models become “extremely rival risk good” and “a jealously guarded secret,” accessible only to “big states and big companies” [00:43:10].

This restriction prevents widespread use of models for beneficial, defensive purposes (e.g., improving cybersecurity, vaccines) and instead fosters a “highly competitive space,” “massive centralization of power,” and reduced transparency [00:43:40]. This is a key concern in the regulation and policy implications of AI.

Safety and Frontier Models

Eric Reese states that his primary concern is not with foundation model labs themselves, as their mission is often to discover whether AGI is possible [00:44:54]. However, he highlights a “missing out on a lot of actual practical uses” because fundraising incentives push entrepreneurs toward “science fiction and speculative stuff” rather than utility [00:45:22].

Safer Alternatives:

If the only models available are large “Frontier models” (closest to AGI), deploying them in real-world systems poses significant safety risks [00:45:50]. Many valuable applications could be built with “smaller models that are properly fine-tuned and are basically safer by definition” [00:46:15]. If these intrinsically safer options are not provided, people will default to less safe alternatives [00:46:38].

Internal tensions within large foundation model labs, particularly between safety research and commercial objectives, can lead to “schizophrenic” organizations where commercial teams prioritize “number go up” over safety [00:47:06]. To address this, Eric suggests re-establishing the connection between research and customer needs, actively engaging with users to understand deployment challenges and ensure successful, safe application of technology [00:47:40].

Overhyped vs. Underhyped Areas

Desired Breakthroughs

Two major breakthroughs that would change the current landscape of challenges and advancements in AI model development are:

  1. Massive reduction in energy/resource requirements: Current models have huge energy demands, posing economic and physical obstacles [00:50:34].
  2. Breakthrough in planning and reasoning beyond subgraph matching: Current auto-regressive models (picking the next word) excel at pattern matching but struggle with complex, novel planning [00:51:16]. Approaches like “Jeeper-based models” or “diffusion models for text” could overcome these limitations [00:52:12].

Ultimately, the goal is to “unlock” the full potential of language as a powerful modality for computing [00:10:01]. This involves making AI broadly applicable and societally beneficial, particularly in areas like law and education, where language is central and societal impact can be significant [00:34:43].

Graph View

Backlinks