LeCun, Yann

Chief AI Scientist, Meta; professor, NYU; Turing Award laureate

Yann LeCun: Dark Matter of Intelligence and Self-Supervised Learning

Lex Fridman Podcast · Jan 22, 2022

• Yann LeCun
• self-supervised learning
• dark matter of intelligence
• joint embedding architectures (VICReg/Barlow Twins)
• contrastive vs. non-contrastive learning
• world models and model predictive control
• grounded intelligence vs. language-only AI
• consciousness as executive world-model configuration
• Facebook/Meta polarization defense
• peer review reform proposal

Core ideas

LeCun frames self-supervised learning as an attempt to reproduce the kind of learning humans and animals do effortlessly — building broad "background knowledge" of how the world works purely through observation — which current supervised learning (requiring massive human-labeled datasets) and reinforcement learning (requiring implausibly many trial-and-error attempts) cannot explain, illustrated by the contrast between a teenager learning to drive in ~20 hours versus a self-driving car needing enormous simulated practice. He describes the core mechanism as training a system to fill in gaps — predicting masked or future parts of an input (words in text, frames in video) from the rest — and argues this "cake" metaphor (where self-supervised learning is the cake, supervised learning the icing, reinforcement learning the cherry on top) reflects how much richer a training signal unsupervised prediction provides per sample compared to the single scalar reward of reinforcement learning or few-bit label of supervised learning. He notes this approach has been transformatively successful in NLP (masked-word pretraining underlies essentially all modern language models) but has not yet succeeded for video, largely because representing the many statistically plausible continuations of a video clip in a high-dimensional continuous space is a much harder problem than representing a probability distribution over a few thousand discrete word choices — and that current NLP approaches even understate this difficulty by treating predicted missing words as independent of each other, which is not actually true (e.g., "lion" and "gazelle" co-occurring is not independent of "cheetah" and "zebra" co-occurring).

Responding to the criticism that self-supervised prediction is "just statistics, not real intelligence," LeCun argues this framing is not particularly meaningful — intelligence may indeed be "statistics" at some level of description, but the relevant question is what kind, and whether learned world models capture genuine causal structure (which he says requires an agent's own actions to be part of its model's inputs, or observation of other agents acting). He identifies three core unsolved challenges for machine learning: learning to represent/predict the world (self-supervised learning), learning to reason via gradient-based (differentiable) methods compatible with deep learning rather than classical symbolic logic, and learning hierarchical, gradient-based representations of action plans — noting that classical optimal control's "model predictive control" (used since the 1960s for rocket trajectories) already demonstrates a form of differentiable planning/reasoning once you have a good predictive model, but that building such learned, uncertainty-handling world models for the messy real world (not just rockets or robot arms) remains unsolved.

LeCun estimates that the roughly 800 million neurons of a cat's brain likely encode most of what's needed for rich physical/causal common sense, with almost all of this knowledge acquired through self-supervised observation (with a small role for hardwired "intrinsic motivation" drives at the base of the brain, e.g., hunger, pain, and social drives, which differ meaningfully across species — humans and many primates are wired to seek social contact, while orangutans are largely solitary) — implying that language, social interaction, and explicit logical reasoning, often considered central to human intelligence, may be comparatively minor evolutionary additions (less than 1% of the human-chimp genome difference) layered on top of capacities already largely present in cats and dogs. He argues strongly for "grounded intelligence" — that machines cannot become genuinely intelligent purely from text, because vastly more information about how the physical world works (e.g., that a table pushed will carry an object sitting on it) is simply absent from any text corpus, making projects like Cyc that tried to hand-encode common sense as facts fundamentally limited.

On technical methods, LeCun describes his own history with Siamese networks and contrastive learning (developed in the early 1990s for a signature-verification project) and explains his current strong preference for non-contrastive "joint embedding" methods (Barlow Twins and its successor VICReg, which he co-developed with student Adrien Bardes and collaborator Jean Ponce) over both contrastive learning (which requires prohibitively many negative examples in high dimensions) and generative/pixel-level prediction approaches — arguing that learning abstract, mutually-predictable representations of two views of the same data (rather than predicting raw pixels) is likely necessary for eventually training systems on video the way masked-language modeling has worked for text. He discusses data augmentation as a "temporary necessary evil" pending better generic self-supervised methods (citing recent success with simple masking/denoising autoencoder approaches for images), and notes a real tradeoff where representations trained to be invariant to standard augmentations (crop, rotate, color shift) become good at classification but poor at object localization, since they've learned to discard exactly the positional information localization requires.

On broader philosophical questions, LeCun offers a speculative hypothesis that consciousness may result from having only a single, reconfigurable "world model engine" in the prefrontal cortex — since we can only attend to one task at a time, an "executive" module that configures this single engine to the situation at hand might be what we experience as consciousness, meaning consciousness could be a consequence of the brain's architectural limitation rather than its power. He pushes back against Ernest Becker's "denial of death" framework (raised by Fridman) as the source of uniquely human motivation, instead attributing humans' distinctive awareness of mortality to simply having a better long-term predictive/planning engine than other animals, and argues that a sufficiently sophisticated autonomous AI — driven by intrinsic motivations and a learned "critic" that predicts good/bad outcomes — would necessarily develop something functionally equivalent to emotions (fear, elation, attachment), dismissing the sci-fi trope of an emotion "chip" that can simply be switched off. He states unequivocally that he believes machines will eventually surpass humans in every domain of intelligence, while stressing this is much further off and more difficult than many prominent figures (naming Elon Musk and some at DeepMind) have publicly estimated.

LeCun defends Facebook/Meta against media narratives that the platform is a net negative for society, citing independent academic studies (including one from Stanford and one involving a Balkans region around a genocide-memorial anniversary) that found no causal link between social media use and increased political polarization — arguing U.S. polarization has risen continuously for roughly 40 years predating Facebook, that comparable countries with equal Facebook usage haven't seen the same polarization trend, and drawing an analogy to the printing press (whose role in enabling mass Bible-reading arguably contributed to the Reformation and subsequent religious wars) as a historical precedent for transformative technologies producing real but non-disqualifying negative side effects. He describes the internal history and structure of FAIR (Facebook AI Research, founded ~2013, now nested within the broader Meta AI organization) and characterizes both Mike Schroepfer and Mark Zuckerberg as far more personally engaged with the underlying science and more "human" than their public media portrayals suggest. He closes discussing peer review's structural flaws (junior reviewers optimizing for finding flaws rather than recognizing valuable new ideas, self-interested and inexperienced reviewer pools), his own long-standing proposal for a decentralized, reputation-based review system (non-exclusive reviewing entities publicly evaluating preprints, with reviewer reputation tied to predictive accuracy about a paper's later impact), and his own hobbies (building electronic wind instruments and model aircraft) — ending with encouragement for young people to prioritize foundational, long-shelf-life knowledge (quantum mechanics, statistical physics, classical mechanics, Fourier analysis) over narrowly applied skills, and to consider applying deep learning to fundamental scientific problems like clean hydrogen production, fusion plasma control, and materials discovery.

Excerpts