Vinyals, Oriol

senior research scientist at DeepMind; formerly Google Brain and Berkeley; lead researcher on AlphaStar

Oriol Vinyals: DeepMind AlphaStar, StarCraft, and Language

Lex Fridman Podcast · Apr 29, 2019

• Oriol Vinyals
• AlphaStar
• DeepMind
• StarCraft II
• reinforcement learning
• imitation learning
• self-play population training
• sequence-to-sequence learning
• Turing test feasibility
• meta-learning and generalization

Core ideas

Vinyals traces his path from a childhood love of computers and semi-professional StarCraft play in Europe in the late 1990s (he deliberately played all three races, including in tournaments, both to build broad skill and to understand what strategies were most annoying from the opposing side, though Zerg was his strongest) to becoming lead researcher on AlphaStar. He describes StarCraft's core structural challenge relative to chess or Go: players start with no pieces and must decide how to allocate resources between economy and military buildup, the game is real-time rather than turn-based (updating 22 times per second) rather than turn-based, and — crucially — it is partially observable, meaning players must actively choose whether to scout for information at the cost of revealing their own position. He reflects on early competitive online gaming (LAN parties, then Battle.net) as a formative, unexpectedly sociological experience, connecting him to people across geographic and political lines he wouldn't otherwise have encountered, and notes StarCraft's outsized cultural importance in Korea specifically, where top players achieved celebrity status as early as 1999-2000.

He recounts AlphaStar's origin at a 2014 joint Google Brain/DeepMind summit, where he presented an earlier Berkeley StarCraft-bot project and was told by DeepMind researchers it was premature — until Blizzard's own outreach and the post-AlphaGo momentum made the project viable after Vinyals moved to DeepMind in 2016. The project's guiding principle from the start was to avoid hand-coded, rule-based heuristics (of the kind used in his earlier Berkeley work) in favor of genuinely learned behavior. He identifies action-space exploration — not partial observability or long-horizon planning — as the single hardest a priori problem: since nearly any random early-game action (like idly moving worker units off automatic mining) actively harms an agent, and useful actions (like building an expansion, which requires selecting a worker, choosing a location, waiting for construction, then reassigning workers) require long, specific, unlikely-to-be-randomly-discovered action sequences, undirected reinforcement learning alone would essentially never stumble into productive strategies. The critical unlock, in his account, was Blizzard's open-sourced dataset of human replay games, which let the team bootstrap via supervised imitation learning rather than pure exploration from scratch.

He describes AlphaStar's architecture as centered entirely on a single neural network policy: given the right observation encoding and action space, the trained weights alone constitute the full agent, with no auxiliary hand-coded logic. Observations combine a spatial/image-like view of the map with an explicit list of visible units as structured objects (each with properties like health, position, and unit type) — a representation he connects to the Transformer architecture (borrowing an idea from a Facebook computer vision paper that treats an image as a scrambled set of pixels tagged with x,y coordinates) and, more broadly, to his own background in sequence modeling: predicting the next action given the full history of prior observations and actions is structurally identical to the "translate a prefix into what comes next" problem in machine translation, and AlphaStar's imitation-learning phase trains on human replays using essentially the same LSTM/Transformer machinery used for language modeling and translation.

Vinyals explains a technique for conditioning the imitation-learned policy on human skill level (MMR — matchmaking rating), where the network is told during training what skill tier each replay came from, letting the same trained policy be asked afterward to "play like" a specified rank; a policy asked to imitate a 6000-MMR player still only reaches roughly platinum level, illustrating that pure imitation captures superficial behavioral statistics without true strategic understanding of winning, which is why self-play against wins/losses is still necessary to refine beyond imitation. He details specific engineering choices around fairness relative to human players: unlike earlier scripted StarCraft bots that could issue tens of thousands of actions per minute (versus a strong human's 300-800), AlphaStar's imitation-trained policy naturally acts at human-like rates and precision because it statistically mimics human replay data, and the team additionally imposed explicit action-rate caps calibrated against professional players' observed distributions — while noting this "human-likeness" constraint is itself an unsolved design problem, since some resulting agents still developed unnaturally high, borderline-inhuman actions-per-minute combined with unusual precision. He also discusses a genuine simulation gap: cloaked (invisible) units produce a subtle visual "shimmer" that trained humans learn to notice but that AlphaStar, seeing structured game state directly, perceives as a hard binary signal — a case where the AI's perception is arguably too sharp relative to the intended human-comparable challenge.

On strategy, Vinyals describes StarCraft's opening game as functioning like a continuous, real-time rock-paper-scissors dynamic — players build an implicit belief state about the opponent's likely tech path and scout to sharpen that belief when uncertainty gets too high — plus "cheese" strategies (all-in, high-risk aggressive plays exploiting the opponent's uncertainty, such as hiding tech buildings to enable an undetectable invisible-unit rush) that can win decisively or lose the game outright if they fail. He explains that AlphaStar addressed strategic diversity not through pure self-play (which tends to converge on a narrow subset of viable strategies) but via the "AlphaStar League," a deliberately curated population of agents encouraged to develop distinct "personalities" (cheese specialists, greedy economy-focused agents, etc.) so the final training population is exposed to and can counter the full range of strategies found "in the wild" — a design he frames as building an internal analogue of Battle.net's own emergent strategic diversity.

Reflecting on the actual 5-0 victory over professional player TLO and the closer, higher-stakes series against Mana (a dedicated Protoss specialist, brought in on short notice after the team's initial calibration bet against a 5000-MMR player), Vinyals is candid that AlphaStar's Grandmaster-level play is not unbeatable or flawless — the agents sometimes failed to scout critical information and had exploitable weaknesses, and the players themselves had systematically miscalibrated priors about what to expect from a novel opponent. He describes the moment of the first confirmed win as intensely emotional and professionally significant, calling it likely "the highlight of my career" for its combination of scientific validation and genuine surprise (he had privately expected the team might lose 5-0). He frames the achievement's importance partly in terms of public science communication — games serving as an accessible, viscerally understandable way for a broad audience to witness AI progress, much as spaceflight did for an earlier generation.

On generalization, Vinyals identifies it as deep learning's core unresolved challenge, illustrated by adversarial examples: neural networks remain a stack of matrix multiplications with exploitable numerical fragility that would never fool a human, and while scaling data and model size (the empirically dominant strategy across sequence-to-sequence tasks, translation, and language modeling) continues to deliver real gains, he doesn't believe scale alone resolves generalization in a fundamental sense. He expresses interest in combining neural networks with more discrete, program-like structures — using a sorting-algorithm example to argue that a genuinely correct program provably generalizes to all valid inputs, whereas a neural network trained to approximate sorting can fail unpredictably outside its training distribution — and voices openness to reviving expert-systems- and rule-based-AI ideas from the 1980s where doing so meaningfully simplifies an important problem, describing this as compatible with rather than opposed to deep learning.

Discussing the Turing test, Vinyals frames it (via reference to Hamming's essay on feasible versus infeasible grand challenges in physics) as currently premature as a serious research target — not because a breakthrough couldn't in principle happen, but because he sees no credible execution plan analogous to the one that made AlphaStar tractable (imitation learning plus self-play population diversity), and is skeptical that a purely statistical approach to language modeling (even citing GPT-2's then-recent impressiveness) is sufficient to solve genuinely open-domain, indefinitely sustained conversation. He describes his own research methodology as oscillating between two modes — tackling a specific hard, important problem and letting new techniques emerge from what it demands (citing how machine translation drove sequence-to-sequence learning, combinatorial problems drove pointer networks, and StarCraft drove population-based self-play), versus exploring an open research direction and later finding a suitable test domain — while stressing that collaboration, mentorship (citing Demis Hassabis's early instinct that StarCraft wasn't yet ready in 2014, and being proven wrong by colleagues' ideas he initially doubted), and disciplined judgment about research timing/feasibility matter as much as raw technical ideas.

On AGI, Vinyals argues the term itself is often poorly defined, and proposes a more concrete proxy: real meta-learning, where a trained system's learning algorithm (not its specific learned weights) generalizes across genuinely different domains — contrasting this with AlphaStar's current limitation that a network trained for Protoss-vs-Protoss must be discarded and retrained from scratch for a different matchup, unlike a human player who can transfer meta-skill across races much faster. He calls for the field to define new benchmarks capable of driving this kind of progress (as ImageNet once did for vision), referencing Léon Bottou's argument that the strict, closed train/test paradigm should itself be reconsidered in favor of "meta-training" and "meta-test" sets. On existential AI risk, he states he is not personally worried in the near term, welcomes the emergence of dedicated AI safety research as valuable, and says he personally finds it more compelling to focus on the technology's positive potential (broadly democratizing access to capabilities like assistants) while remaining open to engaging more with AI safety literature going forward. Looking ahead for AlphaStar and DeepMind, he identifies validating the approach across StarCraft's other races (to confirm the architecture isn't Protoss-specific), and deepening the agent's opponent-modeling/theory-of-mind capacity (belief-state reasoning and deliberate deception, which he sees as underdeveloped relative to AlphaStar's strong micromanagement and macro-economic play) as the clearest next research directions, alongside broader interest in combining graph neural networks with structured, extractable knowledge representations (e.g., automatically derived from Wikipedia or StarCraft's own tech-tree dependencies).

Excerpts