LeCun, Yann

Chief AI Scientist at Meta, professor at NYU, Turing Award winner

Yann Lecun: Meta AI, Open Source, Limits of LLMs, AGI & the Future of AI

Lex Fridman Podcast · Mar 7, 2024

• Yann LeCun
• large language model limitations
• JEPA joint embedding architecture
• world models
• AI hallucination
• autoregressive prediction
• open source AI
• AI doomers and AGI timeline
• hierarchical planning
• humanoid robots

Core ideas

LeCun's central argument is that autoregressive large language models, however impressive, are structurally incapable of reaching human-level intelligence because they lack four things: genuine understanding of the physical world, persistent memory, real reasoning, and real planning. He grounds this in a data argument: the roughly 10^13 tokens used to train an LLM would take a human 170,000 years to read, yet a four-year-old child has already absorbed around 10^15 bytes of sensory information (mostly visual) through ordinary waking experience — meaning most of what humans and animals know about the world comes from observation and interaction, not language, and language itself is a low-bandwidth, already-compressed signal that omits most of the "common sense" physical knowledge a truly intelligent system would need. He revisits Moravec's paradox (systems can pass the bar exam but can't learn to drive in 20 hours or clear a dinner table like a 10-year-old) as evidence that fluency in language is being mistaken for genuine understanding — "we're fooled by their fluency" — and argues even Alan Turing would conclude the Turing test is a poor measure of intelligence.

He explains why LLMs hallucinate as a near-mathematical inevitability of autoregressive generation: each token carries some probability of drifting outside the space of a correct answer, and because these small errors compound multiplicatively across a generated sequence, the probability of staying "on track" decays exponentially with length — a problem partially patched by fine-tuning on the most common question types but never solvable for the long tail of possible prompts, since the system remains fundamentally "a giant lookup table" rather than a reasoning engine. He also notes LLMs devote constant, fixed computation per generated token regardless of a question's actual difficulty, unlike human "System 2" deliberate reasoning, which allocates more effort to harder problems.

As an alternative path, LeCun champions JEPA (Joint-Embedding Predictive Architecture), which he has been developing at Meta/FAIR for roughly a decade. Rather than training a system to reconstruct or generate every pixel of a corrupted image or video (an approach he says has been a "complete failure" for learning good visual representations, despite years of trying with GANs, VAEs, and masked autoencoders), JEPA trains two encoders to produce abstract representations of a full input and a corrupted/transformed version of it, then trains a predictor to map one representation to the other — discarding unpredictable noise (like wind-blown leaves) and preserving only the structure that matters, similar to how humans reason at varying levels of abstraction rather than at the level of quantum field theory. He describes non-contrastive training methods (DINO, I-JEPA, and video-extended V-JEPA) that avoid representational "collapse" without needing explicit negative examples, and reports that V-JEPA's video representations show early signs of detecting physically impossible events (objects vanishing or teleporting) — evidence, he argues, of an emerging intuitive physics. He sketches a future "objective-driven AI" architecture in which a system doesn't generate answers token-by-token but instead optimizes an abstract "thought" representation via gradient descent against an energy function that scores how good an answer is, only converting to language as a final, separate decoding step — meaning reasoning would happen in a continuous concept space independent of any particular language, more efficient than the "generate many hypotheses and pick the best" approach LLMs currently use for chain-of-thought.

On reinforcement learning, he argues it should be used sparingly — mainly to correct a world model or objective function only after planning based on that model fails — because RL is extremely sample-inefficient compared to learning from observation; RLHF's real power, he says, comes mostly from the human-feedback supervision itself (training a reward model) rather than from reinforcement learning per se. On "woke AI" and incidents like Gemini generating historically inaccurate images, he argues no AI system can be unbiased for everyone, since bias is inherently in the eye of the beholder, and the only real solution is diversity of systems — which is why he sees open-sourcing foundation models (as Meta did with Llama 2, with the goal of full participation from groups like an India-focused effort to fine-tune Llama on all 22 official languages, and a Senegal-based effort for local-language medical information access) as essential to prevent the world's information diet from being controlled by a handful of West Coast companies. He argues this isn't incompatible with Meta's business model, since revenue comes from Meta's existing huge user/customer base (ads, business messaging like WhatsApp commerce) rather than from restricting the base model itself, and that widely disseminating a base model accelerates improvement through community contributions.

On AGI and AI doomers, LeCun insists AGI "will not be an event" — no single breakthrough moment after which a system suddenly takes over — but rather decades of gradual, incremental progress on world models, memory, reasoning, and hierarchical planning, with many of the hardest problems (like how a system could learn multi-level hierarchical action plans the way deep learning already learns hierarchical perceptual representations) currently having zero working solutions. He rejects doomer scenarios as resting on false premises: that superintelligence emerges suddenly, and that greater intelligence implies a desire to dominate — a trait he says is specific to certain social species (present in humans, wolves, chimps; absent in solitary-but-intelligent orangutans) and would have to be deliberately engineered into an AI rather than emerging automatically. He compares the challenge of building safe AI to decades of iterative turbojet engineering (there's no single "safety" team — safety and reliability emerge from steadily better design) and dismisses the parallel to nuclear weapons, arguing that because AI progress and breakthroughs diffuse through industry almost immediately (citing AlphaGo being reproduced within three months), no single actor can gain a decisive, unchallengeable lead — future malicious AI-driven persuasion attempts would simply be filtered by everyone else's equally capable AI assistants, like spam filtering. He also draws a historical analogy between the printing press (which enabled the Enlightenment and democracy but also 200 years of religious conflict in Europe) and the Ottoman Empire's 200-year ban on Arabic-language printing presses — done partly to protect the powerful calligrapher guild — asking pointedly who exactly benefits from restricting open AI today.

On robotics, he argues progress has stalled for lack of the same missing ingredient: world models. Boston Dynamics-style robots rely on hand-crafted dynamical models and careful pre-planning rather than learned world understanding, and no current system can perform an "in principle simple" task like clearing a dinner table or loading a dishwasher, though some FAIR "embodied AI" demos can already navigate and fetch known objects. He closes optimistically: AI will act as a universal intelligence amplifier, akin to everyone having a staff of smart assistants (no more threatening than managing employees smarter than oneself), will not cause mass unemployment (job categories shift unpredictably, as with the invention of the smartphone-driven "mobile app developer" role), and reflects his broader conviction — shared explicitly with Lex — that people are fundamentally good, and that AI doomerism often stems from a deeper distrust of people or institutions rather than from the technology itself.

Excerpts