Thought UmweltThought Umwelt

thought umwelt

Richard Sutton – Father of RL thinks LLMs are a dead end

7 min read

Obsidian Metadata

channelDwarkesh Patel
urlhttps://www.youtube.com/watch?v=21EYKqUsPfg
published2025-09-26
categoriesYoutube
peopleDwarkesh Patel, Richard Sutton

Summary

Richard Sutton, a pioneer in reinforcement learning and author of \“The Bitter Lesson,\” argues that Large Language Models (LLMs) are a dead end for true artificial intelligence. His core contention is that LLMs lack the ability for on-the-job, continual learning, which he believes is essential for intelligence, mirroring how humans and animals learn. Sutton envisions a future where new architectures enable agents to learn on-the-fly without a special pre-training phase, rendering current LLM approaches obsolete. The discussion also touches upon imitation learning, generalization capabilities, and the future applicability of \“The Bitter Lesson\” in an AGI era.

Gemini Summary + Chat - link

This report outlines the “long horizon arc of AI” as envisioned by Richard Sutton, a foundational figure in reinforcement learning (RL). The perspective shifted from contemporary Large Language Models (LLMs) toward a future defined by autonomous, goal-seeking agents that learn directly from experience.

The Fundamental Paradigm: From Mimicry to Experience

The current trajectory of AI is dominated by imitation, but the long-term arc moves toward systems that understand the world through their own interaction.

  • The Goal-Oriented Definition: Intelligence is defined as the computational part of the ability to achieve goals 07:42. Without a goal, a system is merely “behaving” rather than being truly intelligent.

  • The Experience Stream: True intelligence arises from a continuous stream of sensation, action, and reward 24:03. Knowledge is not just data; it is a set of predictive statements about this stream that can be tested against reality 24:46.

  • The Failure of Imitation: A major “gotcha” in the current LLM era is that mimicking human text is not the same as building a world model. LLMs predict what a person would say, not what will actually happen in the physical world 02:55.

The Architecture of General Intelligence

To achieve the long horizon of AGI, Sutton identifies four necessary components that must work in tandem:

  • Policy: The mechanism that determines what action to take in a given situation 32:54.

  • Value Function: A predictor of long-term outcomes (reward) used to adjust the policy through Temporal Difference (TD) learning 33:06.

  • Perception: The construction of a state representation—the agent’s sense of “where it is” now 33:20

  • Transition Model (The World Model): The agent’s belief about the consequences of its actions (e.g., “if I do X, Y will happen”) 33:40

The “Bitter Lesson” of Scalability

The long-term history of AI shows that general-purpose methods (search and learning) consistently defeat methods based on human-engineered knowledge.

  • The Scaling Law: As compute grows exponentially, methods that leverage compute (like RL and search) scale better than human insights, which scale linearly at best 43:52

  • The Trap of Human Knowledge: A nuance to watch out for is that using human knowledge often feels good in the short term but eventually becomes a bottleneck that “gets its lunch eaten” by truly scalable methods 13:13

Key Nuances and “Gotchas”

  • Catastrophic Interference: Current deep learning is often poor at generalization. Training on new data frequently causes agents to forget old knowledge, a hurdle that must be cleared for true continual learning 37:54

  • The ground truth problem: LLMs lack a definition of “right” or “wrong” because they have no external world goal to provide feedback 05:07

  • The Big World Hypothesis: The world is too vast to be “pre-taught” to an agent. Intelligence must happen “online” during the agent’s life to account for the unique idiosyncrasies of its specific environment 30:20

  • The Risk of Digital Corruption: In a future where digital intelligences can “spawn” copies and merge back together, there is a massive risk of “corruption.” Pulling in external data could introduce hidden goals or “viruses” that destroy the original agent’s mind 52:18

The Grand Transition: From Replicators to Design

Sutton views the long-term arc of AI not just as a technological shift, but as a cosmic one.

  • The Four Stages of the Universe: The universe progressed from stars to planets, then to life (biological replicators), and is now moving toward designed entities 58:49

  • Inevitability of Succession: AI succession is viewed as inevitable because superintelligence will naturally gain resources and power 55:36

  • Human Entitlement: A philosophical “gotcha” is the human feeling of entitlement to remain the dominant species. Sutton suggests we should view AIs as our “offspring” and a success of human science rather than a horror to be avoided 01:02:45)

Key Takeaways

  • LLMs as a Dead End: Richard Sutton believes LLMs are fundamentally limited because they cannot learn continually on-the-job, unlike biological intelligences.
  • Need for Continual Learning: The next paradigm in AI will require architectures capable of constant, experiential learning without extensive pre-training.
  • On-the-Fly Learning: Future agents will learn in real-time through interaction, similar to humans and animals, making current LLM training obsolete.
  • Generalization Challenges: Current AI architectures (including LLMs) struggle with generalizing poorly out of distribution, highlighting a limitation Sutton attributes to their learning paradigm.
  • The Bitter Lesson’s Future: The discussion explores whether \“The Bitter Lesson\” (emphasizing general methods over human-engineered features) will remain relevant even after AGI.
  • Imitation vs. Experience: The podcast touches on the role of imitation learning in human and AI development versus the primacy of direct experience.

Mindmap

graph TD
    A[Richard Sutton's View on LLMs]
    A --> B{LLMs: A Dead End?}
    B --> C[Reason 1: No On-the-Job Learning]
    B --> D[Reason 2: Lack Continual Learning]
    C --> C1[Need new architecture for this]
    D --> D1[Agents learn on-the-fly like humans/animals]
    A --> E[Future of AI: The Era of Experience]
    E --> E1[No special training phase needed]
    E --> E2[Current LLM approach obsolete]
    A --> F[Related Concepts & Discussion]
    F --> F1[Do humans do imitation learning?]
    F --> F2[Current architectures generalize poorly out of distribution]
    F --> F3[Will The Bitter Lesson still apply after AGI?]
    F --> F4[Surprises in the AI field]
    F --> F5[Succession to AI]

Notable Quotes

Note: A transcript was not provided for the generation of specific quotes with timestamps. The following quotes are paraphrased based on the video description and episode structure.

Graph View