Abbeel, Pieter

professor at UC Berkeley; director of the Berkeley Robotics Learning Lab

Pieter Abbeel: Deep Reinforcement Learning

Lex Fridman Podcast · Dec 16, 2018

• Pieter Abbeel
• deep reinforcement learning
• imitation learning
• third-person demonstration learning
• self-play
• hierarchical reinforcement learning
• transfer learning and generalization
• simulation-to-reality transfer
• robot safety testing
• human-robot affection

Core ideas

Abbeel frames a robot beating Roger Federer at tennis as constrained more by hardware than software — citing Boston Dynamics' progress (including parkour-style stair climbing) as impressive but still far from human-level physical mobility, estimating roughly 10-15 years for the hardware alone. He's more optimistic about the narrower manipulation problem: a stationary robot arm learning to swing a racket against a ball machine via reinforcement learning (with simulation pretraining) seems feasible to him, though achieving human-level shot precision (hitting lines, applying spin) remains an open and interesting question.

On robots and psychology, Abbeel describes finding it nearly impossible not to anthropomorphize robots even while fully aware of their mechanisms — citing his lab's own PR2 robot ("BRETT," Berkeley Robot for the Elimination of Tedious Tasks, casually referred to as "he") and a fully scripted SoftBank Pepper robot at a Fidelity event that felt like an active social participant despite zero underlying learning. He speculates this social/emotional dimension could itself become a reinforcement learning objective: since RL optimizes whatever objective is specified, there's no in-principle reason a robot couldn't be trained to optimize for being "fun to be around" and thereby acquire increasingly interactive, pet-like or person-like qualities — with the practical objective-specification challenge solvable not through explicit numerical reward scoring (hard for humans to produce reliably) but through relative preference comparisons, citing OpenAI's Paul Christiano's work training a simulated one-legged "Hopper" robot to perform backflips purely from a human comparing pairs of video clips ("this is better than that") without ever explicitly stating the goal.

Asked why reinforcement learning works at all given sparse, delayed rewards, Abbeel explains the sample-inefficiency problem directly: with a reward received only after e.g. 100 actions, a single trial can't reveal which specific actions were good or bad, requiring many trials before policy gradient methods can statistically identify which actions are consistently associated with higher versus lower reward and adjust the policy accordingly. He offers a structural intuition for why deep RL nonetheless works surprisingly well: ReLU-based neural networks function as piecewise-linear feedback controllers, and linear feedback control alone is already known to solve many complex real-world problems remarkably well (citing his own past experience stabilizing helicopter hover flight, a highly complex dynamical system, with a comparatively simple linear controller) — deep networks effectively "tile" the state space into many linear controllers, sharing structure across the tiling in a way manually-designed piecewise systems don't, gradually improving on pure linear control's limitations.

He identifies the biggest gap between current RL successes and real-world applicability as extreme variation in temporal scale and credit assignment horizon: a decision like "pursue a PhD" is astronomically abstract relative to the underlying stream of muscle-fiber-level actions that actually constitute a life, and no current RL algorithm can perform credit assignment across such vastly different timescales — a capability he says would require genuine hierarchical reasoning, which 1980s-90s symbolic planning systems approximated but never grounded in real perception. He describes two research directions his lab pursued toward hierarchy: "bolting" deep learning onto traditional planning/dynamical-systems representations via end-to-end training (citing Aviv Tamar's causal InfoGAN-style work), and an information-theoretic approach treating high-level action choice as selecting a latent variable predictive of future outcomes — which repeatedly failed to produce results, leading his lab to instead directly meta-learn for the downstream benefit hierarchy is supposed to provide (faster learning) rather than hierarchy itself, exemplified by the "RL²" (Learning to Reinforcement Learn) paper led by Rocky Duan, where hierarchical-looking behaviors (consistent hallway traversal, avoiding revisiting explored areas) emerged from meta-learning without being explicitly designed in, though not yet at real-world scale.

On transfer learning and generalization, Abbeel is optimistic, citing AlexNet's 2012 ImageNet breakthrough as significant less for raw performance than for demonstrating genuinely reusable learned representations (fine-tunable for new tasks) — a pattern he sees extending via scale in large language models (OpenAI and Google's work) and DeepMind's UNREAL maze-navigation results. Referencing a distinction (possibly from a prior Yoshua Bengio conversation) between "learning to master" and "learning to generalize," he illustrates the harder form of generalization with an example: current deep learning could predict planetary motion well under normal conditions but would fail to predict the effect of a new massive object entering the solar system, unlike physics built from genuinely simple, compressed first principles — a kind of generalization current deep learning methods haven't yet achieved, though he considers it uncertain whether pursuing ever-simpler learned representations (as physicists pursue simpler unifying equations) is the right path forward. Referencing a conversation with Vladimir Vapnik about a hoped-for unifying "theory of learning," Abbeel finds this a worthwhile pursuit specifically because of evidence for brain modularity (citing cases where blind individuals' visual cortex gets repurposed for other functions) — suggesting a similarly modular, scalable computational architecture, rather than one single master theory, may be the more realistic target.

Asked whether he leans toward mathematical theory or empirical trial-and-error, Abbeel says he'd prefer progress via mathematics (since formal results can "leapfrog" long stretches of costly experimentation) but acknowledges that in practice, RL progress has mostly come through gradual, one-experiment-at-a-time empirical insight rather than derivable theoretical shortcuts. On self-play versus imitation learning, he highlights self-play's core advantage — reliable learning signal is available on both sides of every interaction, since one player's win is inherently contrasted against the other's loss — and expresses hope that more RL problems beyond games (with natural built-in opponents) could someday be reformulated as self-play (even speculating about a robot learning to build a hut this way), while noting most real-world tasks currently lack an obvious self-play formulation, making dense reward shaping or demonstration necessary instead. He describes teleoperation-based imitation learning (robots directly experiencing high signal-to-noise demonstrations, letting a robot learn skills like relocating a bottle in about ten minutes) and highlights third-person imitation learning — led by his student Chelsea Finn — as a major 2018-era breakthrough: a robot watches a human perform a task (without ever controlling the human's body) and learns to "translate" the observed demonstration into its own embodiment via meta-learning, analogous to machine translation between demonstration "languages."

On autonomous driving specifically, Abbeel argues the third-person/first-person distinction matters much less than in manipulation, since vehicle dynamics (getting the car from point A to a point a few meters ahead) are already well understood, unlike the complex contact/interaction forces in robotic manipulation; he considers the more important open question to be imitation learning's lack of explicit goal/objective representation (addressable via inverse-reinforcement-learning-style imitation), arguing purely reactive behavioral cloning is unlikely to generalize as well as an approach reasoning explicitly about objectives. On simulation, he proposes reframing the standard "make one simulator precise enough to transfer to reality" goal into training across an ensemble of imperfect, individually non-representative simulators, treating the real world as simply another (unseen) sample from that same distribution of simulators — a domain-randomization-style strategy for robustness rather than pursuing ever-higher simulator fidelity.

On safety, Abbeel distinguishes near-term, practical physical-safety concerns (a strong robot or car unintentionally causing harm through error) from longer-term AI-alignment concerns, and argues the field currently lacks a real equivalent of a human driving test — noting that the actual bar humans clear for a driver's license (a short supervised drive around the block) is a strikingly minimal but apparently representative proxy for broader driving competence, whereas no comparably validated, rigorous testing framework yet exists for verifying that an updated self-driving car software version is strictly better than its predecessor without having silently introduced new failure modes, especially relative to the extremely high real-world reliability bar (millions of miles between serious incidents) actually expected of such systems.

Closing on a more philosophical note (crediting Andrew Ng's example for showing him "the value of kindness"), Abbeel considers whether kind versus exploitative policies are equally easy for agents (human or artificial) to discover, framing much human prosocial behavior as encoded by a long evolutionary "outer" optimization process (innate drives like hunger, thirst, and pain motivate learning) layered with a tendency toward in-group cooperation and out-group suspicion that requires explicit teaching to overcome — while citing Steven Pinker's Better Angels of Our Nature thesis (declining violence over historical time) as reason for optimism that this arc bends toward broader cooperation. Asked whether an RL-trained robot could genuinely love and be loved by a human, Abbeel reframes the question around the sophistication threshold actually required for strong affection — noting dogs, despite reasoning far below human sophistication, already reliably produce genuine strong human affection and happiness through simple greeting/companionship behaviors — and concludes he sees no principled reason an AI system couldn't achieve a comparable (if not equivalent-to-human) level of mutual affection, while explicitly declining to weigh in on whether that would be a good outcome.

Excerpts