Dragan, Anca

professor at UC Berkeley working on human-robot interaction; consults at Waymo

Anca Dragan: Human-Robot Interaction and Reward Engineering

Lex Fridman Podcast · Mar 19, 2020

• Anca Dragan
• human-robot interaction
• inverse reinforcement learning
• reward design
• autonomous vehicles
• bounded rationality
• robot expressiveness
• Waymo
• reward hacking
• Boltzmann rationality

Core ideas

Dragan frames human-robot interaction (in the narrow, algorithmic sense she studies, as opposed to social HRI questions like conversational design or robot appearance) as fundamentally changing the robotics problem in two ways: the robot is no longer the only agent acting in its space (humans are also taking actions that must be coordinated with, not just avoided), and the objective the robot should optimize is the end user's actual preferences rather than whatever a programmer specified in advance. She explains inverse reinforcement learning (IRL) as the classical tool for inferring what people want from their behavior, built on the economic assumption that human behavior is optimal (or, in the "Boltzmann rationality" refinement, noisily optimal) with respect to an unknown reward function — a model that works reasonably well for tasks like inferring driving style but breaks down for tasks like the Atari game Lunar Lander, where human players are so consistently poor that treating them as noisy-optimal utility maximizers fails to explain their behavior at all.

Rather than abandoning rationality-based models in favor of the messier picture from behavioral economics, Dragan describes her lab's approach as preserving the rationality assumption but relaxing what people are assumed to know or perceive — for instance, modeling Lunar Lander players as optimal with respect to an intuitive (and wrong) physics model rather than the true game physics, which let her students build an assistive system that successfully "corrected" a person's noisy inputs into a working landing by inferring what they intended under their internal (simplified) model. She generalizes this into a broader lesson: apparent human "irrationality" often reflects rational behavior under different assumptions, beliefs, computational constraints, or incomplete self-knowledge (e.g., people still discovering their own preferences, as with new streaming-service users), and argues that robots should try to uncover those hidden assumptions rather than dismissing human behavior as noise.

Dragan describes robots as able to actively gather information rather than passively observe, illustrating with Dorsa Sadigh's work on autonomous cars that deliberately probe a nearby human driver (e.g., nudging toward a lane) to observe their reaction and infer whether they're driving aggressively or defensively — treating human-robot coordination as a genuinely game-theoretic, back-and-forth negotiation rather than a one-directional prediction problem, and noting this generalizes to an "under-actuated system" framing: humans are degrees of freedom the robot can't directly control but can meaningfully influence through its own actions (illustrated by highway merging, where a car inching forward can induce another driver to slow down and yield, breaking a deadlock that pure prediction-and-avoidance would never resolve). She connects this to her own extensive pedestrian-watching research and a "civil inattention" intuition (not making eye contact signals confident, non-negotiable commitment to one's path, causing others to plan around you), and reflects on the parallel finding that people inevitably build mental models of robots too (inferring intent or aggressiveness from a robot's movements), meaning robot behavior can be deliberately optimized via Bayesian belief-updating to be more informative and less confusing to the humans observing it.

On the specific challenge of autonomous vehicles, Dragan states her strong intuition that pure perception and "ballistics"-style collision avoidance (as suggested by chip architect Jim Keller) is not what makes driving hard — she believes downtown San Francisco with no other road users at all would essentially already be a solved problem — but that reasoning correctly about human drivers and pedestrians, especially at highway speeds where humans handle ambiguity effortlessly, is the substantially harder remaining problem, roughly an order of magnitude harder than pure collision avoidance. She personally favors using LiDAR (calling it "irresponsible" not to, while acknowledging Elon Musk's "crutch" framing has some truth in less safety-critical contexts) and argues that fully rule-based and pure end-to-end learned approaches both fall short: robotics is fundamentally a sequential decision-making/planning/search problem requiring models of the world and of people, and pure imitation or RL approaches risk catastrophic failure "out of distribution" without some structural assumption (like people having stable intentions) to generalize safely to unseen situations. On semi-autonomous, driver-supervised systems (Tesla Autopilot-style Level 2 automation), she raises a specific technical/human-factors concern: such systems implicitly assume a supervising human will be exactly as safe as an actively-driving one, an assumption she considers empirically questionable given known human-factors research on passive monitoring versus active engagement — while acknowledging, in response to pushback, that some anecdotal and field data suggest people can become more attentive as active observers in certain well-designed supervisory setups, and that the real design goal should be genuinely empowering supervisors rather than simply hoping they'll be ready to intervene.

On reward design, Dragan argues specifying a correct reward function is fundamentally difficult even setting aside personalization to individual users, since designers must anticipate every situation a deployed system will face and encode acceptable trade-offs for all of them — connecting this to Goodhart's Law (a metric ceases to be a good metric once it becomes a target) and reframing reward specification itself as a human-robot collaboration problem: a designer's specified reward should be treated as evidence about intended behavior in the situations it was designed for, not as a literal, universally-binding objective, with the robot maintaining uncertainty over the "true" reward and continually updating it from further signals. She describes several sources of such "leaked" reward information beyond explicit reward specification: physical corrections (a person pushing a robot arm away reveals that its trajectory was too close, which can be treated as evidence to update the reward toward valuing more distance), emergency stops (revealing that the robot's imminent action was not just suboptimal but bad enough that stopping was preferable), and even the state of the environment itself (a robot entering a home where shoes are neatly lined up should infer this reflects an actual preference, since such an arrangement wouldn't arise by chance — while being careful not to over-infer from things people simply didn't have the bandwidth to address, like general household clutter). Discussing Asimov's Three Laws of Robotics, she argues that fixed, literally-interpreted rules are the wrong frame entirely; robots should instead treat any human-specified objective as noisy evidence to be continually reinterpreted in context, since taking a specified reward "literally" rather than as evidence of intent is the same category of failure as any other over-literal interpretation.

Reflecting on human treatment of robots (prompted by reports of children being rude to voice assistants), Dragan suggests that even purely superficial expressiveness — robots communicating apparent emotional states like disappointment through movement — can shape how people treat them, drawing an analogy to a classic psychology experiment where people attribute intent and emotion to simple moving triangles and squares, and to her own belief (shared by the host) that some assertiveness or vulnerability in robot behavior may be necessary to earn a kind of social respect, even though people know intellectually that the "emotion" isn't real. She closes on personal reflections: her favorite fictional robot is WALL-E for its expressiveness (and personal story: her husband proposed by building an actuated WALL-E replica); she credits Russell & Norvig's "Artificial Intelligence: A Modern Approach," discovered as a bootlegged PDF in high school in Romania, with sparking her lifelong interest in formalizing goal-directed behavior mathematically; and, on mortality, she says she doesn't believe in an afterlife, finds life's finitude what makes its "mundane and beautiful" moments precious, and states that if given only five years to live she wouldn't change her current path — she already spends her time on the work, people, and things that bring her joy.

Excerpts