Kaelbling, Leslie

roboticist and professor at MIT; founder and former editor-in-chief of the Journal of Machine Learning Research

Leslie Kaelbling: Reinforcement Learning, Planning, and Robotics

Lex Fridman Podcast · Mar 12, 2019

• Leslie Kaelbling
• reinforcement learning
• POMDPs and belief space
• hierarchical planning
• robotics abstraction
• model-based vs. model-free RL
• perception representation
• Journal of Machine Learning Research
• peer review reform
• AI value alignment

Core ideas

Kaelbling traces her path into AI to reading Gödel, Escher, Bach in high school (drawn to how simple primitives combine into complex behavior) and into robotics specifically through her first job at SRI's AI lab working on a successor to the Shaky robot, despite having no formal robotics background at the time. She praises Shaky's original tech report as remarkably prescient — it already contained A* search, symbolic planning, learned macro-operators, configuration-space planning, and vision-based self-localization — and credits her advisor Stan Rosenschein's "situated automata" philosophy (that logical reasoning tools are for the engineer analyzing a system, not necessarily something the system itself must manipulate internally) as formative, while defending her own group's habit of "reinventing wheels" (including reinvented reinforcement learning, initially calling rewards "pleasures") as a valuable way to appreciate why established solutions work once you've found bad ones yourself.

She frames the history of AI as an oscillating, sociologically-driven cycle — from early cybernetics/homeostasis-inspired robots, to expert systems attempting surface-level symbolic encodings of expertise — arguing 1980s-90s expert systems failed not primarily due to cost but because humans generally lack real introspective access to their own decision processes (people can offer plausible post-hoc stories about how they set up a microphone or pick a tie, but these explanations aren't reliably accurate accounts of the actual underlying process). She's careful to separate this critique from symbolic reasoning generally, arguing computers should absolutely reason at multiple representational levels depending on the problem, with abstraction — spatial, temporal, and goal-based — being essential for making otherwise computationally intractable real-world reasoning (like planning "in the afternoon" rather than at the level of joint angles at a specific timestamp) feasible; she explicitly rejects methodological dogma pitting symbolic reasoning against neural networks, arguing computer science itself should determine the right tool per problem, though she suspects future machine learning systems will eventually learn useful abstractions automatically rather than relying solely on hand-designed ones.

Explaining MDPs and POMDPs, Kaelbling describes an MDP as a modeling stance — assuming full knowledge of a system's current state such that history adds no further predictive value, with probabilistic (not deterministic) transitions — while noting MDPs are a poor fit for most real problems, which are genuinely partially observable: an agent must maintain a probabilistic belief about hidden world state based on noisy, local observations and its action/observation history. She candidly acknowledges that optimal POMDP planning can be undecidable, but rejects treating this as a reason to avoid the framework altogether ("the problem you have to solve is the problem you have to solve") — arguing this intractability is precisely what defines the field of AI, requiring layered approximations in both modeling and solution algorithms rather than an excuse to abandon principled uncertainty modeling. She describes a related "methodological crisis" in the field: unlike earlier eras when computer science theory offered formal approximation guarantees (regret bounds, provable convergence), current empirical deep learning work largely lacks equivalent formal solution concepts, meaning many systems work well without anyone having a principled account of why — engineering, in her view, currently running well ahead of the underlying science, analogous to bridges once built by trial and error before structural theory caught up.

She distinguishes "belief space" from "state space" reasoning: rather than controlling the actual world state, an agent operating under uncertainty is really controlling its own belief (a probability distribution over possible world states), which can justify actions taken purely to gather information (asking a question, looking over one's shoulder while driving) rather than to directly change the world — a distinction relevant to any problem requiring deliberate information-gathering, and to weighing whether one's current certainty (e.g., about where a doorway is) is sufficient to proceed with an action versus needing more information first.

On hierarchical planning, Kaelbling uses the example of navigating an unfamiliar airport: humans don't plan the specific sequence of movements through security and gates in advance, both because it would be wasteful effort and because the necessary information (gate location, crowds) isn't yet available — but successful high-level planning ("fly to New York, get from gate to office") still requires having learned enough about the abstract feasibility and difficulty of intermediate sub-goals (how hard is walking through an unfamiliar airport, roughly) to make a credible "leap of faith" that the details will resolve themselves — and she identifies acquiring these abstract feasibility/duration models, and generalizing them aggressively to novel situations, as a live open research question. On goal regression / backward-chaining planning, she notes classical AI planning research (going back to Herbert Simon's means-ends analysis) hasn't empirically shown backward search from the goal outperforming forward search within its typical problem classes and methods, despite her own persistent intuition that it should.

Asked whether human life itself could be formalized as a planning problem, Kaelbling firmly rejects the framing — arguing intelligence necessarily requires "pieces and parts" using different styles of reasoning, representation, and learning (just as brains have varied substructure), rather than one universal algorithmic solution; she frames the model-based-vs-model-free reinforcement learning debate similarly, as fundamentally a time-space computational tradeoff (a directly stored policy trades memory for fast action-selection; a compact world model plus a planner trades slower computation for compactness) rather than a question with one universally correct answer, with different subproblems (algebraic manipulation vs. riding a unicycle under real-time constraints) likely favoring different representational choices.

On perception, Kaelbling argues the central open question is representational — not "can we classify images" (which has seen real progress), but what perception should actually output to interface productively with the rest of an integrated intelligent system, a question she says remains largely unanswered. She's skeptical of the pure end-to-end "one giant neural network" position primarily on sample-complexity grounds (since reducing hypothesis space via built-in structural bias, as convolution does for spatial reasoning, is the only known way to reduce data requirements), and argues the field needs to discover more structural priors "like convolution" for other kinds of reasoning — citing forward search, abstraction, relational/graph-based representations, and (controversially, she notes) the basic assumption that discrete objects exist as candidate examples of useful built-in bias that could scaffold more efficient learning.

On machine consciousness and self-awareness, Kaelbling states she doesn't spend much time worrying about consciousness, noting most philosophers who care about it will concede a behaviorally human-indistinguishable "zombie" is conceptually possible, and that she'd be satisfied with that outcome for now; she does consider a limited form of self-awareness — a system component that monitors whether other components are functioning correctly — practically important, without needing to resolve deeper questions about subjective experience. She resists benchmark-driven definitions of success (like the Turing test) for guiding robotics research, preferring to spend effort making robots work better rather than debating philosophical tests, though she acknowledges the value competitions (like the DARPA driving challenges) provide in motivating researchers and occasionally producing genuinely good ideas amid a lot of "hacking."

Recounting the founding of the Journal of Machine Learning Research, Kaelbling describes it as arising from repeated, unresolved complaints to Kluwer (publisher of the existing Machine Learning journal) about excessive institutional subscription costs — leading roughly 75% of the editorial board to resign and found a new, fully open-access journal modeled partly on the existing Journal of AI Research, which she notes was cheap and straightforward to run (a few hundred dollars a year) specifically because computer scientists already possess the technical skills (typesetting, web infrastructure) that make self-publishing infrastructure trivial in ways it wouldn't be for other academic fields. Reflecting on peer review more broadly, she describes an unrealized alternative model she considered at JMLR's founding — a curated group of trusted, articulate reviewers publicly reviewing papers (without a formal submission/acceptance journal structure) — as potentially more valuable than either the traditional gatekeeping journal model or a pure arXiv-plus-social-upvoting free-for-all, since social buzz doesn't reliably indicate genuine expert judgment. She also expresses concern that current publish-or-perish incentives have shortened researchers' horizons relative to earlier eras when a PhD thesis could involve years of sustained work on one hard problem before any papers were published, worrying the field is losing space for the kind of longer-horizon research some problems genuinely require.

On AI risk, Kaelbling recounts a conversation with military ethicists on autonomous weapons who assumed robots are programmed via explicit step-by-step scripting (like Lego Mindstorms), leading them to find it strange that a robot might behave unexpectedly — she considers correcting this misconception (that most modern AI systems instead specify a hypothesis class and objective function, then run optimization without knowing in advance what specific solution will emerge) a personal "educational mission." She frames value alignment as the practically important near-term concern flowing directly from this dynamic — since practitioners are effectively becoming "engineers of objective functions" rather than just algorithms, and since (as any optimization practitioner knows) you must be careful what you wish for, because an optimizer can return a technically-optimal solution that reveals the objective itself was specified wrong. She explicitly declines to weigh in on the economic/sociological effects of automation on jobs, citing it as outside her expertise, and expects continued boom-bust cycles in AI's popularity (driven partly by inevitable overselling relative to delivered results) while maintaining that each cycle raises the field's overall "high-water mark" of real capability, rather than expecting smooth, monotonic progress toward human-level intelligence.

Excerpts