Levine, Sergey

professor at Berkeley; researcher in deep learning, reinforcement learning, robotics, and computer vision, known for end-to-end neural network policies combining perception and control

Sergey Levine: Robotics and Machine Learning

Lex Fridman Podcast · Jul 14, 2020

• Sergey Levine
• reinforcement learning
• robotics
• deep reinforcement learning
• off-policy learning
• robotic manipulation
• Moravec's paradox
• common sense
• reward functions
• robot learning

Core ideas

Levine opens by distinguishing robot hardware capability from robot intelligence, using a 2004 Stanford PR1 video (a robot tidying a room and fetching beer) that turns out to be teleoperated by a human as a punch line: if you give a robot body a human brain, the hardware gap is nearly closeable with money and engineering, but the "intelligence gap" remains vast — and grows explosively as the environment becomes less tightly controlled (a factory robot can be superhuman on a fixed task, but any variation, as in an open kitchen, exposes the gap). On nature versus nurture, he reframes the question around sufficiency: rather than asking what evolution hard-wired, he asks what minimal capability suffices, noting humans handle genuinely novel situations (like operating an unfamiliar joystick-controlled robot arm under pressure) not through evolved specific skills but through general adaptability — and speculates that the human "iceberg" of common-sense knowledge is itself built up over a lifetime of experience, which is encouraging for machine learning if learning systems can learn to distill unstructured lifelong experience rather than relying on rigid, narrowly supervised input-output training.

A central open question Levine raises is whether it matters where training experience comes from — whether simply amassing huge, passively-collected datasets (the "download the internet" approach of large language models) suffices, or whether machines need to actively interact with and intervene in the world, observing the consequences of their own actions, to build genuine common sense; he suggests active experience may better support answering "what if" counterfactual questions (central to what we call common sense) and proposes that this active, ongoing "perpetual improvement loop" — occasionally testing what your current model claims is a good idea, and correcting on failure — is what most needs to be captured for future methods to surpass today's approaches. He argues robotics is valuable not primarily as an end in itself but as a lens for understanding intelligence generally: robotics forces integration of perception, prediction, and control rather than allowing modular, independently-solved subproblems, and it throws Moravec's paradox (hard-for-humans tasks like calculus are easy for machines; easy-for-humans tasks like manipulation or drinking from a cup are hard for machines) into especially sharp relief, which he sees as a signal pointing at something conceptually missing rather than a mere engineering gap. He attributes manipulation's special difficulty partly to the poor fit between motor control and the "tightly supervised" input-output paradigm that has worked well in domains like computer vision (labeled images) — humans and animals are never told exactly which muscle commands to issue, so systems modeled on that kind of supervision are structurally mismatched to the problem.

Discussing his own 2014 work on end-to-end reinforcement learning for a peg-in-hole manipulation task, Levine highlights the finding that jointly training perception and control lets errors trade off optimally between subsystems (e.g., tolerating vertical perceptual imprecision because it self-corrects via contact, while penalizing horizontal imprecision more) — outperforming a modular pipeline where each component must be independently accurate — and connects this to naturally-occurring perceptual heuristics like the "gaze heuristic" used by baseball players, pilots, and even frogs to intercept moving objects without solving explicit trajectory equations. He discusses robotic grasping as a once-canonical "hard because endlessly novel" problem that shifted from being treated as an inverse-physics/geometry problem to one solved much more effectively via learning from simulation and real-world trial and error, while noting the field may now be nearing diminishing returns on grasping specifically as the guiding benchmark, given the difficulty of generalizing across arbitrarily varied objects and materials. On common sense, he argues systems like image captioners fail at "obvious" real-world facts (mistaking a person in a fur coat for a teddy bear) not because they lack a component called common sense, but because they inhabit a narrower universe (pixels and English sentences) that simply doesn't require acquiring it — implying that forcing AI systems to actually operate within, and maximize utility inside, the full messiness of the real world (which robotics does) may be what's needed to induce common-sense-like understanding.

Levine explains reinforcement learning as today's broad umbrella term for learning-based control (rather than its narrower historical "literal reward and punishment" meaning), frames it as a generalization of supervised learning, and distinguishes value-based methods (predicting utility of an action) from model-based methods (predicting resulting world state) as different ways of answering the same underlying "what if I take this action" counterfactual question. A recurring practical theme is off-policy (or offline/batch) reinforcement learning — learning from data generated by some other policy rather than the agent's own trial and error — which he sees as the key unsolved bottleneck for deploying RL at scale (since most real-world settings, like autonomous vehicles, can't tolerate unconstrained live exploration): the core challenge is knowing when a model's predictions about untried actions are trustworthy, addressed via density-estimation methods and regularization that discourage relying on out-of-distribution predictions, closely paralleling problems in causal inference. He calls the mathematical fact that near-optimal control can be achieved without a complete model of a system's governing equations "beautiful," and explains deep reinforcement learning's importance as solving RL's especially acute version of the feature-engineering problem that plagued pre-deep-learning supervised learning (it's far harder to hand-design useful features for chess or robotic control than for images).

Pushed on why real-world robot learning still lags dramatically behind simulated game-playing successes like AlphaZero, Levine argues the bottleneck isn't network capacity but "scaffolding" problems invisible in simulation: a robot learning to wash dishes through raw trial and error will simply break all the dishes, and defining a reward function (how do you even score "poured the water correctly"?) can itself require a whole separate perception system — exposing real gaps in the RL problem formulation itself rather than just engineering shortfalls. He sees reusing prior experience (multitask learning, meta-learning) as the more realistic solution to real-world sample inefficiency than expecting isolated mastery from scratch, is skeptical that simulation alone can ever substitute for learning from real data (any human-built, non-improving component — a simulator or a hand-designed controller — eventually becomes the bottleneck), and is cautiously optimistic that self-play-style continual improvement could eventually generalize to robotics and human interaction, provided the "mediating mechanism" (real-world physics and social dynamics, rather than a board game's fixed rules) can itself be learned from experience. On reward design, he treats it as a genuinely open and underexplored research question — including unsupervised/intrinsic-motivation approaches like minimizing Bayesian surprise (drawing on Karl Friston's computational neuroscience work), which his group has shown can produce useful emergent skills — and, on AI safety, says he currently worries less about superintelligent misalignment than about systems that are insufficiently well-optimized for safety-critical domains, and more broadly about nefarious human use of AI than about AI systems themselves becoming existential threats.

Excerpts