Vinyals, Oriol

Research Director and deep learning lead at DeepMind; his second appearance on the podcast

Oriol Vinyals: Deep Learning and Artificial General Intelligence

Lex Fridman Podcast · Jul 26, 2022

• Oriol Vinyals
• DeepMind
• Gato generalist agent
• Flamingo and modularity
• tokenization across modalities
• transformer attention mechanism
• meta-learning and few-shot prompting
• emergent abilities in language models
• AI sentience and LaMDA
• Ilya Sutskever and the bitter lesson
• AGI timeline

Core ideas

Vinyals opens by questioning whether a fully AI-driven interview (replacing either interviewer or interviewee) would actually be desirable, noting that current systems only observe data passively during training and don't keep learning during a conversation — retaining context only within a limited "working memory" window (he cites roughly 2,000 words at the time) before forgetting, a sharp contrast to genuine lifelong learning. He and Lex explore, half-playfully, what would make an AI conversational partner feel authentically compelling rather than merely fluent — floating the idea that qualities like consistent identity, "mortality" (irrecoverable reputational stakes, e.g. being permanently "canceled"), and strongly-held opinions might be learnable from human data the same way factual correctness is, since there's abundant data on what makes humans engaging.

He walks through Gato, DeepMind's "generalist agent" (named via wordplay on "general agent" and the Spanish word for cat, hence his cryptic "meow ­🐱" tweet), explaining it as a single transformer trained to predict the next token in a unified sequence spanning text, images, and actions — all mapped into disjoint integer ranges (words, then images, then discretized actions) that only become connected through learned embeddings during training, not through any built-in cross-modal structure. Gato was trained on a mix of internet-scale text/image data plus recorded trajectories from many separately-trained DeepMind reinforcement learning agents (Atari, 3D navigation, robotics), producing one relatively small (~1 billion parameter) network capable of acting across many of those environments, though not yet as proficient as task-specialized agents — which is why he called it "the beginning, not the end": he expects synergy between modalities/tasks to emerge with further scale and better-structured data (e.g. providing textual context before an Atari game, similar to how instructions help humans).

Vinyals discusses modularity as a promising alternative to always training from scratch, citing Flamingo — built by freezing the 70-billion-parameter Chinchilla language model and grafting on new vision-processing sub-networks and additional training data — as proof a model can gain new capabilities (like reading subtle humor in an image, referencing a case where Andrej Karpathy had previously said no vision system could parse) without discarding prior learned weights, contrasting this with Gato's approach of training everything jointly from scratch on the same data. He frames the central open question for the field as whether to keep training a shared core from scratch, embrace modularity (freezing and recombining pretrained components, potentially at the scale of hundreds of networks), or find some hybrid — analogizing this to software engineering's move from reimplementing libraries to reusing them.

On meta-learning, he traces its redefinition from a narrow 2019-era notion (learning new object categories from a few examples, tied to vision benchmarks) to the "language models are few-shot learners" (GPT-3) revelation that language itself can serve as the interface for teaching new tasks via prompting, later extended to vision+language in Flamingo (e.g. learning to do arithmetic from a few labeled digit-image examples). He speculates that within five to ten years this could evolve into a fully interactive teaching process — a single set of pretrained weights that a user teaches a new task (like a new video game) through ongoing dialogue and feedback, rather than one-shot prompting, and draws a technical analogy between prompting and the classic nearest-neighbor algorithm (both retrieve from stored context rather than requiring retraining).

Discussing what makes projects like AlphaFold succeed, Vinyals emphasizes that the "genius" is disproportionately distributed across mundane but critical factors people underrate: engineering quality of data pipelines and compute infrastructure, the sustained human effort of building good benchmarks (crediting the field's shared metrics, e.g. protein structure prediction benchmarks that predated AlphaFold, and repurposed ones like Atari that fed into Gato), and team dynamics and individual researcher persistence over multi-year efforts — alongside the historical "accident" that GPUs built for gaming happened to be available when deep learning needed them. On the Transformer architecture specifically, he explains its core "inductive bias" as content-based attention — querying the past not by recency but by relevance — as the key advance over LSTMs, while noting its major current limitation is context length, an open problem he discusses (with Lex) in terms of possible hierarchical or increasingly "learnable" attention mechanisms.

On emergent abilities in large language models, Vinyals distinguishes benchmarks requiring only "one-hop" reasoning (which tend to show smooth, predictable scaling curves, as in ImageNet) from harder, multi-step benchmarks where performance seems to jump abruptly past a scale threshold — an empirically observed but not yet theoretically explained phenomenon, though he notes techniques like the Chinchilla scaling-law study allow useful extrapolation from smaller-scale experiments for some (though not all) benchmarks.

Asked directly, as a private individual rather than a DeepMind spokesperson, about the Google engineer who claimed the LaMDA model was sentient, Vinyals says he does not believe current models are close to sentience, grounding this partly in his sense — shaped by adjacent work on AlphaFold and biology — that biological complexity operates at orders of magnitude beyond what current neural networks achieve, while being careful to say he wouldn't tell others how to feel about their own experience of these systems. He predicts, however, that a "civil rights movement for robots" is plausible as AI systems become more integrated into ordinary people's lives (people naming them, forming attachments, perceiving apparent suffering), and suggests engineering practice may eventually need explicit norms or rules around designing (or refraining from designing) displays of apparent sentience into products like customer-support bots. He personally doubts consciousness in the human sense is a necessary ingredient for building highly capable, human-level AI, though he thinks biological and evolutionary insights (broadly, not at the level of precise neuronal mechanisms) can still usefully inform algorithm design.

He praises his longtime colleague and friend Ilya Sutskever's early, sustained conviction that scale plus supervised data would solve most problems (recalling Sutskever rehearsing an even more provocative early version of a now-famous talk), broadly agrees with Rich Sutton's "bitter lesson" that general, computation-scalable methods tend to win out over hand-crafted ones, but is more skeptical specifically about the "search" component of that lesson, citing AlphaCode (built on ideas shared with AlphaStar) as one clear case where scale plus massive search paid off, while noting search's applicability elsewhere remains unclear. He states he's confident an AI system will reach human-level general intelligence within his lifetime, primarily through imitation learning at scale, but is far less certain about "going beyond" that (which would require well-defined reward functions in the way Go's win condition enabled superhuman play), and voices measured worry (more than excitement) about a possible "singularity"-level transformation, primarily around resource and energy limits, while remaining most excited about AI's potential to broaden access to knowledge and capability for people currently without it. He closes by saying that in a future multi-planetary or interstellar civilization, he would want humans, not just AI systems, meaningfully "part of the equation" — hoping for something like parity rather than humans being displaced.

Excerpts