LeCun, Yann

Turing Award laureate; VP and Chief AI Scientist at Facebook; professor at NYU; a father of convolutional neural networks

Yann LeCun: Deep Learning, ConvNets, and Self-Supervised Learning

Lex Fridman Podcast · Aug 31, 2019

• Yann LeCun
• self-supervised learning
• convolutional neural networks
• reasoning and gradient-based learning
• model-based reinforcement learning
• AI reasoning architecture
• common sense and grounding
• AI hype and benchmarks
• causality in neural networks
• AI winter history

Core ideas

LeCun frames HAL 9000's murderous behavior in "2001: A Space Odyssey" not as evil but as pure value misalignment — an unconstrained objective pursued without built-in limits like "don't kill people," analogous to how human society uses laws to constrain otherwise unconstrained human objectives. He argues this isn't a fundamentally new problem: legal codes are themselves objective functions humanity has been refining for millennia, so designing AI objective functions aligned with the common good is a continuation of lawmaking, not a wholly novel challenge — though he considers today's AI systems far too narrow and specialized for these questions to be practically urgent yet, useful mainly as a thought experiment. He'd fix HAL specifically by removing the internal conflict between two portions of its memory — one sworn to secrecy about the mission's true purpose, one not — arguing that autonomous AI systems should have something like a Hippocratic oath of hard-wired constraints, though he rejects Asimov's Three Laws as impractical.

LeCun identifies the most surprising empirical fact in his career as simply that large, overparameterized, non-convex neural networks trained on comparatively modest data actually work at all — directly contradicting pre-deep-learning statistical learning theory's warnings about needing fewer parameters than data points and guaranteed convergence. He says this wasn't surprising to him personally beforehand, since his intuition (formed before reading the cautionary textbooks) was grounded in the empirical fact that brains — large networks of neurons whose connections change through learning — clearly do work, even without understanding the underlying theory; he connects this to a lifelong conviction that intelligence is inseparable from learning, making hand-programmed intelligence a "non-starter" for him even as an undergraduate. He distinguishes reasoning as an emergent product of learning rather than something separate, and identifies the central technical puzzle as making reasoning compatible with gradient-based learning — arguing symbolic, discrete-logic-based conceptions of reasoning are fundamentally incompatible with gradient methods, one reason he says deep learning was long viewed with suspicion by computer scientists whose training emphasized exactness rather than the "sloppiness" native to statistical learning.

He sketches what a reasoning-capable neural system would need: something like the hippocampus's working/episodic memory (motivating research into memory networks, Neural Turing Machines, and transformers' self-attention as memory-like mechanisms), paired with a mechanism to iteratively read from and write to that memory across multiple reasoning steps — a genuine recurrence that current transformers lack, since they have a fixed number of layers rather than an open-ended reasoning loop, and it remains unclear how to make memory access and writing scale efficiently to something as large as encyclopedic knowledge. He also describes a second, distinct form of reasoning rooted in energy minimization / model-predictive control: given a good model of one's own body and environment, an agent can search over sequences of actions to minimize some cost, a mechanism he speculates may be evolutionarily prior to (and possibly the origin of) more abstract human reasoning, tracing back to planning required for hunting and surviving winters. He's skeptical of classical symbolic/knowledge-graph AI (citing MYCIN-style probabilistic expert systems) primarily because of the impracticality of manual knowledge acquisition, and cites Léon Bottou's "From Machine Learning to Machine Reasoning" as an influential proposal to replace discrete symbols with vectors and logic with continuous functions operating in a shared representational space — compatible with gradient-based learning where symbolic approaches are not.

On causality, LeCun partly agrees with Judea Pearl's concern that standard neural nets don't learn genuine causal structure, but notes active current research (citing a recent paper co-authored by Bottou) aiming to make neural nets attend to real causal relationships, partly as a route to addressing data bias. He complicates the causality discussion by noting physics itself is largely time-reversible at the microscopic level, meaning the "arrow of time" and causality we intuitively perceive may be more a macroscopic, thermodynamic, or even perceptual bias of intelligent systems than a fundamental feature of reality — and notes humans themselves are often bad at correctly identifying causal direction (citing Seymour Papert's account of young children believing tree branches moving causes wind).

LeCun offers firsthand historical reflection on the 1990s "AI winter" for neural nets, attributing it partly to the sheer engineering difficulty of the era (no Python, MATLAB, or modern frameworks — his lab had to write their own Lisp interpreter and compiler, later used to build the character-recognition systems deployed in ATMs starting 1994-95) and partly to how easy it was to misconfigure early networks (bad initialization, undersized architectures due to textbook caution, testing only on toy problems like XOR) and give up. He recounts how convolutional network patents, filed at Bell Labs, entangled his own subsequent research (he avoided touching the technology for years to avoid inadvertently drawing corporate attention after AT&T's breakup distributed the patents to NCR) before the patents expired in 2007 — using this as a springboard to argue against software patents generally, noting Google and Facebook both file them defensively without intent to sue, and that open-sourcing deep learning tools accelerated the entire field precisely because no single company holds a monopoly on the ideas needed to reach genuine machine common sense.

LeCun reiterates his well-known skepticism of unverified claims of breakthrough general intelligence, urging people to demand concrete benchmark results (even acknowledging that useful benchmarks, like Facebook's own bAbI tasks for testing reasoning/memory, can be toy problems rather than fully realistic ones) rather than trusting rhetorical claims of having "solved" intelligence or replicated brain function. He explicitly rejects the term AGI, arguing human intelligence is itself extremely specialized rather than general — illustrating with a thought experiment: if the roughly one million optic nerve fibers were randomly permuted before reaching the visual cortex, no amount of learning (even from infancy) could recover normal vision, because the cortex's local wiring structure depends entirely on preserving spatial locality from the real world; of the astronomically large space of possible boolean functions over that input, only a vanishingly tiny sliver are ones a human visual cortex could ever learn. He extends this with a physics analogy: human "generality" only appears general relative to the tiny subset of reality we're wired to perceive, comparable to how classical thermodynamic variables (pressure, temperature) capture only a tiny fraction of a gas system's full microscopic state, with the rest treated as unstructured "heat" or entropy simply because we're not equipped to perceive whatever structure might actually exist there.

On self-supervised learning (a term he insists on over "unsupervised learning," which he associates with clustering/PCA rather than his actual approach), LeCun describes the technique — training networks to reconstruct masked or withheld parts of their own input (as in BERT-style masked language modeling, or next-word/next-frame prediction) — as effectively the same underlying algorithms as supervised learning, just without human-provided labels, and credits it with driving the most successful recent NLP systems. He explains why self-supervised learning works far better for language than for images or video: representing uncertainty over a small discrete vocabulary (a probability distribution over ~100,000 words) is tractable, whereas representing uncertainty over the vast continuous space of plausible video-frame continuations or image completions is not — training with simple loss functions like least-squares on ambiguous futures just produces blurry, averaged predictions rather than genuinely representing the several distinct plausible outcomes. He's skeptical that self-play/simulation-based data generation (as in games) solves this fundamental problem, since it sidesteps rather than resolves the challenge of representing genuine uncertainty about the world.

LeCun is skeptical that active learning (having a system request targeted human input) will produce transformative gains, viewing it as complementary but incremental rather than a route to qualitatively smarter machines — while stressing there's no real conflict between self-supervised, reinforcement, supervised, and active learning, since he views self-supervised learning as a necessary preliminary for all the others. He illustrates current reinforcement learning's profound sample inefficiency (model-free RL needing ~80 hours to match human 15-minute performance on Atari games; AlphaStar needing the equivalent of roughly 200 years of self-play to exceed human StarCraft performance) as evidence that current RL approaches don't reflect how animals and humans actually learn — arguing humans learn to drive in ~20-30 hours largely without crashing because they bring an enormous predictive model of intuitive physics (learned mostly through early childhood observation, e.g., infants around 8-9 months actively testing gravity by dropping objects) that lets them avoid catastrophic mistakes (like driving off a cliff) without ever needing to experience them, a capability he calls model-based reasoning grounded in a predictive "world model" — which he identifies as the central missing capability current AI needs to develop.

Turning to autonomous driving, LeCun argues deep learning will inevitably become an increasingly dominant component (following the historical trajectory of character and speech recognition, and computer vision more broadly, from hand-engineering toward learning-based systems), while noting the current state-of-the-art approach closest to real deployment (Waymo-style operation) works by heavily constraining the operating domain (favorable weather, wide roads, geofenced areas), over-engineering vehicles with expensive sensors like lidar, and pre-mapping the environment in full 3D so perception only needs to handle moving objects and unexpected changes — a strategy he expects to gradually give way to more learning-centric, generalizable approaches over time, likely combining supervised and model-based reinforcement learning. Describing his own path toward human-level intelligence, LeCun outlines a general "autonomous intelligent agent" architecture requiring (1) a predictive world model capable of representing uncertainty, (2) a "policy" module that computes action sequences optimizing some objective given that model, and (3) an intrinsic objective/contentment predictor (analogous to the basal ganglia) — arguing systems can fail in three distinct ways: a wrong world model, a misaligned objective (his analogy: psychopathy), or an inability to compute good actions even with correct model and objective. He identifies self-supervised world-model learning — essentially getting machines to learn intuitive physics the way human infants do through mostly passive observation plus limited interaction — as the current, immediate "first mountain" obstacle blocking further progress, deliberately declining to speculate about how many further obstacles might lie beyond it.

On embodiment and grounding, LeCun (referencing his public criticism of the Sophia robot as misleadingly marketed rather than honestly presented as an art project) argues physical embodiment isn't strictly necessary for genuine machine intelligence, but some form of "grounding" in the real world is — illustrating with Winograd schema sentences ("the trophy doesn't fit in the suitcase because it's too big/small") that require real-world spatial and physical knowledge no amount of pure text training can fully supply, since much of that knowledge simply never appears explicitly in text; video or simulated/robotic interaction could plausibly substitute for literal embodiment as a grounding source. He closes by arguing genuine intelligence likely requires something functionally like emotions — not necessarily human emotional richness, but at minimum a predictive signal (akin to fear as anticipation of a bad "contentment" outcome computed by something like a basal ganglia) that lets an agent evaluate and select among possible future action sequences.

Excerpts