Prasad, Rohit

Vice President and Head Scientist of Amazon Alexa; one of its original creators

Rohit Prasad: Amazon Alexa and Conversational AI

Lex Fridman Podcast · Dec 14, 2019

• Rohit Prasad
• Amazon Alexa
• Alexa Prize
• conversational AI
• far-field speech recognition
• natural language understanding
• voice assistant privacy
• Turing test
• self-learning AI
• dialogue systems

Core ideas

Prasad frames the comparison between AI assistants and human intelligence as a category error if it's limited to "human-likeness" — assistants like Alexa have genuinely superhuman properties (existing simultaneously in multiple locations, near-infinite and unambiguous memory) that deserve equal consideration alongside more human-like conversational ability, and different interactions call for different blends: a "turn the lights on" command should stay purely transactional, while open-ended conversation should lean human-like. He argues that dialogue — not simply parsing intent or entities from natural language, but full open-ended conversation with an unknown, evolving goal state — is the hardest and most legitimate test of machine intelligence, harder than game-playing domains like Go or chess precisely because the "win condition" and even the current state are often undefined and shift as a conversation unfolds (e.g., planning a weekend that changes destination or event mid-conversation).

He describes the Alexa Prize, a multi-year academic Grand Challenge where university teams build "socialbots" that must sustain a coherent, engaging 20-minute conversation with real Alexa customers, judged first through live customer ratings and then by human expert judges in controlled finals (requiring at least two of three judges to agree a conversation "stalled" to fail it). Three years in, Prasad estimates the field remains five to ten years from reliably clearing the 20-minute bar, but notes striking incidental progress — bots developing rudimentary humor and personality-aware behavior (recognizing domain switches, masking understanding gaps) — while cautioning that much of what looks intelligent is still closer to sophisticated lookup and context-matching than genuine reasoning (illustrated by a bot relating an out-of-context "fun fact" about Tom Brady rather than reasoning about the actual game just played). He frames the competition's real institutional purpose as counteracting an AI talent and resource drain from academia into industry, giving university researchers access to real customers, compute, and live feedback loops that a typical NSF-funded study with a static annotated corpus can't replicate — with success measured via live customer ratings (1-to-5, "how likely are you to interact with this bot again") rather than any offline benchmark.

On the history and engineering of Alexa itself, Prasad recounts joining in 2013 to find eight of ten teammates skeptical that far-field speech recognition (recognizing speech from 20-40 feet away, amid household noise) was even solvable — a problem for which no training data existed, requiring the team to bootstrap novel far-field datasets before any customer base existed, then combine that data with then-nascent deep learning and distributed GPU training to cut error rates by roughly 5x within six months. He describes the notoriously hard "wake word" detection problem (reliably distinguishing "Alexa" directed at the device from "I love my Alexa" or overheard conversation) as effectively a separate, still-unsolved research problem, and recounts an early multi-domain natural language understanding challenge — resolving ambiguous entities like "the Stones" (Rolling Stones vs. Stone Temple Pilots) via a statistical intent classifier and entity resolver rather than hand-authored grammar rules, since a purely rule-based approach couldn't scale to the range of real customer requests. He credits Amazon's internal "working backwards" process — writing a hypothetical press release and FAQ before building the product, akin to writing the paper before doing the research — with keeping the team focused despite feature creep, and traces Alexa's evolution from ~13 core "skills" (music, weather, timers) at launch to over 90,000 third-party skills today, alongside newer initiatives like Alexa Conversations (letting developers train multi-turn dialogue flows from sample interactions via RNNs rather than hand-authoring dialogue trees) and cross-skill goal inference (e.g., inferring that a movie-ticket purchase might imply wanting a nearby dinner reservation or a ride).

On privacy, Prasad grounds Alexa's design in two explicit principles, transparency and control: visible/audible cues when audio streams to the cloud, a physical mute button, the ability to review and delete individual utterances (including by voice command), and an opt-out from human review of voice data for training. He directly refutes the popular anecdote that Alexa "listens" to ambient conversation and serves related ads, attributing such experiences to coincidental seasonal advertising patterns rather than covert listening, while acknowledging that public trust requires continuous, proactive education rather than a one-time disclosure. On what's next, he identifies "more conversational" (goal inference beyond one-shot transactions), "more self-learning" (Alexa already auto-corrects millions of misrecognitions from implicit customer feedback signals like repeated retries, without human supervision), and "more natural" (skill-agnostic requests, natural-sounding neural text-to-speech, multi-item commands) as the three current fronts of development, and identifies reasoning — genuinely inferring a customer's latent goal from limited, ambiguous signals across a long tail of highly diverse requests — as the hardest unsolved problem, one he doesn't expect to be fully solved even on a 40-year horizon, since it depends on unresolved deeper questions about how the human brain represents and processes knowledge. He is explicit that deep learning alone is not sufficient for that reasoning layer, even as he affirms it remains sufficient for many of Alexa's current prediction-based, non-reasoning tasks, and points to transfer learning, active learning, and learning-with-less-data as the more promising near-term research directions.

Excerpts