Russell, Stuart

professor of computer science at UC Berkeley; co-author of Artificial Intelligence: A Modern Approach

Stuart Russell: Long-Term Future of Artificial Intelligence

Lex Fridman Podcast · Dec 9, 2018

• Stuart Russell
• AI control problem
• value alignment
• meta-reasoning
• AlphaGo search
• self-driving car reliability
• nuclear weapons history analogy
• gorilla problem
• provably beneficial AI
• algorithmic social media risk

Core ideas

Russell recounts writing an early chess program in 1975 at Imperial College under extreme hardware constraints (eight seconds of total CPU time per move via punch cards), and traces his subsequent research into "meta-reasoning" — reasoning about what to think about — as foundational to understanding both human game-playing intuition and modern systems like AlphaGo. He explains that a chess or Go program's power comes from two components: an evaluation function (assessing how promising a position is, sometimes with superhuman speed and accuracy even without any lookahead) and selective search (deciding which of an astronomically large number of possible move sequences are worth exploring further) — and that the key intuition behind selective search is that a move is worth thinking about in proportion to both how promising it looks and how much uncertainty remains about its true value, since thinking only pays off when it has a real chance of changing your decision. He connects this directly to his own experience: after implementing improved meta-reasoning techniques in his backgammon and Othello programs, they began beating him, and he cites Garry Kasparov's description of sensing "a new kind of intelligence" across the board from Deep Blue as a related, exciting rather than frightening feeling — arguing that chess/Go programs aren't inherently threatening because they rely on assumptions (full observability, single well-defined objective, short time horizons) that don't hold in the real world, and that AI progress fundamentally consists of removing these simplifying assumptions one by one, each removal multiplying the range of real-world problems addressable.

On self-driving cars, Russell recounts working on autonomous vehicles at Berkeley in the early-to-mid 1990s, where perception reliability (not planning) was the central bottleneck, and offers a concrete reliability calculation: a taxi driver accumulates roughly 100 million seconds of driving over a career, meaning any perception system needs roughly eight nines of reliability, whereas even 98.3% object-detection accuracy represents barely one "nine," leaving roughly seven more orders of magnitude of improvement needed — explaining why a successful demo is nowhere close to true readiness. He describes Google's early rule-based, symbolic architecture for self-driving (classical machine vision feeding a logical database and hand-written rules) as fundamentally unable to converge, since real driving constantly presents genuinely novel situations (his example: a little girl riding a bicycle the wrong way around a traffic circle) that no finite rule set can anticipate — arguing driving fundamentally requires the same kind of lookahead/reasoning-under-uncertainty that made chess and Go programs strong, but harder, because other drivers' mental states and intentions must be inferred (are they turning, distracted, or drunk?) and interact game-theoretically with the AV's own trajectory, unlike a chess opponent whose overriding intention (win the game) is always known. He cites his colleague Anca Dragan's work showing that formulating human-AV interaction as an explicit game leads to genuinely novel emergent behaviors (e.g., a car backing up slightly at a stalled four-way stop specifically to signal "you go first" — a communicative strategy the system invented on its own, never explicitly programmed).

Russell frames the core AI safety problem — which he calls "the control problem" — as a matter of misspecified objectives, not necessarily runaway superintelligence per se, tracing it through Alan Turing's own 1951 warning that a sufficiently capable thinking machine would quickly outstrip humanity and that "we should have to expect the machines to take control," and Norbert Wiener's related warning after witnessing Arthur Samuel's checkers program improve past its creator. He uses the King Midas myth and folkloric "three wishes" stories as illustrations of a failure mode nearly universal across cultures: an agent optimizing a literally-specified objective produces disastrous unintended consequences precisely because the specified objective wasn't truly what was wanted, and argues this problem applies broadly across 20th-century optimization-based disciplines (statistics' loss functions, control theory's cost functions, operations research's reward functions) — not just AI. His proposed solution is to abandon the paradigm where machines pursue a fixed, exogenously-specified objective as "gospel truth," and instead build machines that remain explicitly, mathematically uncertain about the true objective, deferring to human correction and treating human behavior/choices as ongoing evidence about what humans actually want — reframing the human-machine relationship as a cooperative, game-theoretic interaction rather than a machine executing a fixed goal independently.

Extending this argument beyond AI specifically, Russell describes corporations as effectively pre-existing optimization systems that pursue a narrow objective (quarterly profit) decoupled from overall human well-being, arguing this dynamic (not hypothetical future superintelligence) already contributes to problems like climate inaction, and draws a parallel to government being "taken over" by actors pursuing private objectives rather than serving the public. He argues social media click-through optimization exemplifies the same misalignment risk already realized at scale: rather than simply matching users to content they already prefer, an algorithm that maximizes predictive accuracy of clicks is incentivized to make users more predictable by nudging their preferences toward extremes — a dynamic he considers a plausible contributor to political polarization and democratic erosion, occurring, in his view, with essentially zero regulatory oversight comparable to the FDA's staged clinical-trial framework for pharmaceuticals (chosen as an analogy because, unlike a single surgeon's mistake, software and pharmaceutical errors both scale catastrophically before problems become apparent).

On feasible near-term AI regulation, Russell proposes starting with more tractable, already-technically-addressable problems: detecting and mitigating dataset-driven algorithmic bias, and mandating that AI systems self-identify as non-human rather than impersonate people (noting California already bans this narrowly, only for fraud and voting-influence contexts) — flagging deepfakes as a rapidly emerging threat with essentially no current legal protection in the political domain, unlike commercial trademark-style protections.

Drawing an extended historical analogy to nuclear weapons, Russell recounts how physicists (including Nobel laureate Frederick Soddy in 1915 and H.G. Wells in fiction as early as 1912) recognized the theoretical possibility of atomic weapons decades before their creation, while the physics establishment (citing Ernest Rutherford's September 1933 dismissal of atomic energy extraction as "moonshine") actively refused to believe it was practically achievable — until Leo Szilard, reading about that very speech, conceived of the neutron chain reaction the next day and immediately recognized its danger given the geopolitical context (Hitler's recent rise), leading him to patent but deliberately suppress the reactor concept. Russell uses this to illustrate "motivated cognition" — believing what one wishes were true rather than what is true — as an explanation for why many AI researchers dismiss safety concerns, citing (and rebutting) specific arguments he's personally encountered, such as "calculators are superhuman at arithmetic and haven't taken over the world" and comparing AI risk to the vanishingly improbable spontaneous formation of a black hole near Earth (which he distinguishes as a bad analogy, since if the entire physics community were actively working to create such a black hole, asking "is this safe?" would be an obviously reasonable question). He estimates the AI research community's median estimate for human-level AI arrival at 40-50 years, considers his own estimate somewhat more conservative, and rejects the framing (which he attributes to a recent Stanford "AI100" report) that human-level AI is likely impossible altogether, comparing this stance to cancer researchers publicly declaring a cure impossible.

Russell frames the "gorilla problem" — the situation gorillas find themselves in after inadvertently enabling a more intelligent species (humans) to dominate them, with no path back to control — as the default outcome absent solving the control problem, but is careful to distinguish this "loss of control" failure mode (machines pursuing objectives that no longer serve humans and having no incentive to accept correction) from two additional failure modes: misuse (a "Dr. Evil" deliberately deploying otherwise-safe AI technology for harmful ends, which he frames as partly a policing and professional-culture problem, briefly referencing autonomous weapons and the "Slaughterbots" film as an example of widely-recognized catastrophic misuse potential) and overuse — which he calls the "WALL-E problem," describing a slow, voluntary civilizational surrender of autonomy and competence as humanity increasingly delegates knowledge and decision-making to machines, drawing on E.M. Forster's 1909 short story "The Machine Stops" (which he considers remarkably prescient, describing something resembling video calls, MOOCs, and screen-induced obesity) as an early literary treatment of this risk. He argues this is fundamentally not a technical problem for AI itself to solve, since properly designed AI systems should recognize that genuine human preferences include autonomy and self-sufficiency (analogous to a helicopter refusing to fly you to the summit of Everest if the point was to climb it yourself) — the deeper risk being human short-sightedness and laziness overriding that design intent.

Reflecting on his own public role in AI safety advocacy, Russell describes actively searching for reasons his own arguments might be wrong (including reading extensively in moral philosophy on utilitarianism, noting striking parallels between historical philosophical debates over utilitarian formulas — e.g., the "repugnant conclusion" and hypothetical heroin-drip hedonism — and current AI alignment debates) and states his emerging framework of provably beneficial machines built on explicit objective uncertainty needs significant further mathematical elaboration, since vague ("hand-wavy") notions of beneficial AI leave exploitable loopholes that a sufficiently capable optimizing system would find, much as poorly written tax law gets exploited by tax avoidance.

Excerpts