Silver, David

leads the reinforcement learning research group at DeepMind; lead researcher on AlphaGo and AlphaZero; co-led AlphaStar and MuZero

David Silver: AlphaGo, AlphaZero, and Deep Reinforcement Learning

Lex Fridman Podcast · Apr 3, 2020

• David Silver
• AlphaGo
• AlphaZero
• MuZero
• reinforcement learning
• self-play
• Monte Carlo tree search
• Lee Sedol
• DeepMind
• deep learning

Core ideas

Silver traces his path to AI from childhood programming (BBC Micro, then Prolog exposure via his father's AI studies) through a games-industry career building handcrafted, non-learning "AI" that could out-react but not out-think humans, to a PhD applying reinforcement learning to Go — a domain he chose because, unlike chess, checkers, backgammon, and Othello (all conquered by heuristic search by the late 1990s), Go resisted brute-force methods due to its astronomical search space (~10^170 positions) and, more importantly, the absence of any reliable way to statically evaluate a board position: strong Go players rely on an intuitive sense of territory and life-and-death that nobody knew how to encode into rules. He recounts that the strongest Go program in 2000 still lost to a nine-year-old given a 29-stone handicap, and argues that cracking Go's evaluation problem was seen as requiring something like human intuition — making it a genuine proxy challenge for general AI progress, not just a game milestone. He frames reinforcement learning (an agent that takes actions, observes an environment, and gets a reward signal it seeks to maximize over time) as, in his view, not merely one AI technique but potentially the correct formalization of the intelligence problem itself, while remaining agnostic about whether current RL solution methods (value functions, policies, and models — the three basic building blocks he describes for decomposing an RL solution) are the final ones needed.

Silver describes AlphaGo's evolution in stages: first a pure deep-learning system (no search at all, trained on human expert games) that surprised the team by already reaching strong amateur ("dan") level, which convinced him that professional and eventually world-champion-level play was achievable; then AlphaGo proper, combining that learned evaluation with search, which beat European champion Fan Hui and then world champion Lee Sedol 4-1 in 2016 (Silver had privately predicted this exact score based on AlphaGo's known ~20% rate of temporary "delusions" — knowledge gaps that could persist for dozens of moves). He recounts vivid moments from the Lee Sedol match: Game 1's audacious invasion of Lee's territory, Game 2's now-famous "Move 37" (a probability-defying shoulder-hit on the fifth line that Go commentators initially thought was an operator error, later absorbed into human Go theory as genuine computer-discovered creativity), Game 3's double-ko crisis that Lee conjured from a losing position, and Game 4's brilliant human comeback that plunged AlphaGo's win-probability estimate before it recovered — a sequence Silver says taught the team to trust a system that has exceeded its own designers' judgment. He notes Garry Kasparov, who initially saw his 1997 loss to Deep Blue as a personal failure before recognizing its historical significance, considers AlphaZero's chess breakthrough even more profound than Deep Blue's because it achieved superhuman play through learning rather than hand-coded evaluation — and that world chess champion Magnus Carlsen has publicly credited studying AlphaZero's games with improving his own play to a new career-high rating.

The conversation traces the conceptual arc from AlphaGo (trained on human data plus self-play) to AlphaGo Zero (learning purely from self-play, discarding human game data entirely, which Silver says was always the deeper scientific goal since removing built-in knowledge makes a system less brittle and more transferable to other domains) to AlphaZero (the identical algorithm, unmodified, applied out-of-the-box to chess and Shogi, achieving superhuman performance in each with zero domain-specific tweaking) to MuZero (which learns to plan successfully in domains, including Atari games, where it isn't even given the rules, building its own implicit model of environment dynamics from raw observations and reward alone). He explains the core intuition behind self-play's power as a mechanism for iterative self-correction: any system with errors ("delusions") can improve as long as it can play out consequences and notice when its predictions were wrong, and this bootstrapping process — from random play to expert level — has, in principle, no ceiling other than the size of the game itself. He frames AlphaGo Zero's creativity concretely: over 40 days of self-play training, the system independently rediscovered essentially all of the corner-opening patterns ("joseki") that human Go players had refined over thousands of years, then began discarding and improving on some of them with novel variations that top human players have since studied and adopted into modern play. He cites two real-world extensions of the AlphaZero approach published in Nature — retrosynthetic chemical pathway planning and a quantum-computation optimization problem — as early evidence that these ideas generalize well beyond games.

Asked about intrinsic motivation and reward specification, Silver argues that any coherent notion of intelligence requires a well-defined ultimate goal against which a system's behavior can be measured, even though a system (like a human) may set flexible sub-goals in service of that ultimate objective; without a defined purpose, he says, there's no clearly defined problem of intelligence to solve at all. Pressed for his own view on human meaning, he offers an explicitly speculative, multi-layered account (borrowing from Max Tegmark): the universe's second law of thermodynamics can be read as an entropy-maximizing "goal," subserved by evolution's discovery of efficient energy-dissipating reproduction, subserved in turn by evolution's development of brains as flexible, learning-capable goal-achieving mechanisms — with humans now creating further layers (like AI systems built to pursue goals, such as Go mastery, more effectively than their creators can pursue them directly) that Silver suggests may recursively continue beyond human-designed AI itself.

Excerpts