Poggio, Tomaso

professor at MIT; director of the Center for Brains, Minds and Machines

Tomaso Poggio: Brains, Minds, and Machines

Lex Fridman Podcast · Jan 19, 2019

• Tomaso Poggio
• Center for Brains Minds and Machines
• compositionality theorem
• deep learning vs. biological learning
• curse of dimensionality
• face recognition neuroscience
• cortex modularity
• neuroscience of ethics
• consciousness and Turing test
• David Marr levels of understanding

Core ideas

Poggio traces his fascination with intelligence back to a teenage infatuation with Einstein and relativity, but describes a deliberate pivot in motivation: rather than trying to match Einstein by solving one deep problem, he reasoned it would be more valuable to find "the key to an intelligence ten times better or faster than Einstein" — a meta-approach to solving many hard problems at once — which redirected his research focus toward understanding human intelligence itself as, in his view, the greatest open problem in science, partly because it's a question about the very tool (the brain) used to do science in the first place. He draws a lesson from Einstein's biography (the weakest of five PhD students at ETH Zurich, and the only one who didn't land an academic position, instead going to the patent office) as evidence that unconventional, non-conformist thinking — not credentialed conformity — tends to produce breakthroughs, comparing it to stock market advice to "never buy what everybody else is buying."

Asked whether strong AI can be built without first understanding the biological brain, Poggio frames this as a genuine open bet rather than a settled question, drawing an analogy to flight (where deep bird biology wasn't strictly necessary, though basic principles like heavier-than-air lift mattered) while noting that recent AI breakthroughs — reinforcement learning (tracing to Pavlov and later Minsky), deep learning generally, and the specific layered convolutional architecture (tracing to Hubel and Wiesel's 1960s visual cortex research) — have historically drawn significant inspiration from neuroscience, leading him to bet that neuroscience will continue to meaningfully inspire (though not necessarily solely drive) future breakthroughs. He describes his own view of artificial neural networks evolving over time: once considering them oversimplified caricatures of biological neurons, he's come to appreciate that despite this simplification, they're structurally much closer to actual neural networks (units communicating with other units) than prior symbolic-logic-based computational paradigms (Lisp, Prolog) ever were.

On the central gap between artificial and biological learning, Poggio identifies deep learning's dependence on massive labeled datasets (ImageNet's roughly one million human-labeled images) as fundamentally mismatched with how children learn — a baby sees millions of images but receives comparatively few explicit labels, framing today's dominant paradigm as "n going to infinity" (labeled examples) versus biology's apparent "n going to one." He connects this to nature-vs-nurture, hypothesizing that evolution — constrained by humans and fruit flies having surprisingly similar total gene counts, given Drosophila's largely hardcoded, minimally-learning nervous system — likely couldn't afford to hardcode detailed neural circuitry, and instead encodes weak, general-purpose "priors" alongside broad learning machinery. He illustrates this with face-recognition neurons: rather than being hardwired from birth, evidence (citing Margaret Livingstone's experiments raising baby monkeys deprived of visible human faces, using masked technicians) suggests these areas are highly plastic and get rapidly imprinted by whatever visual stimulus is most reliably associated with feeding in early life — since some of the monkeys' "face" neurons instead became tuned to the blue gloves used to feed them milk.

On brain organization, Poggio notes mid-20th-century "equipotentiality" theories (that any brain region could substitute for any other, based on Lashley's rat lesion studies) have been overturned by more precise lesion, wartime-injury, and fMRI evidence showing highly specific functional modules — while noting the cortex specifically presents a genuine puzzle: despite radically different functions across regions (vision, audition, motor control, language), its underlying neuron type and connectivity architecture appears largely uniform, leaving open whether this reflects one shared underlying computational mechanism applied to different inputs (an idea he finds instructive for artificial network design) or something more differentiated. He's candid that basic neuroscience questions — even something as fundamental as why humans sleep — remain unanswered, and argues understanding intelligence genuinely requires integrating multiple levels of abstraction (molecular, physiological, algorithmic, psychological) simultaneously, contrasting this with computers, where engineers deliberately separated hardware (transistors) from software (PowerPoint) such that understanding one level offers little insight into the other — a decoupling he argues doesn't cleanly exist in biological brains, making the problem of understanding intelligence harder than understanding an engineered system.

Poggio explains his own concept of compositionality as the mathematical property distinguishing when deep (multi-layer) networks meaningfully outperform shallow ones: specifically when the function being learned is itself a hierarchical composition of simpler local functions (e.g., recognizing images by first processing small pixel groups, then combining those computations, analogous to reading letters into syllables into words into sentences). He recounts a friendly disagreement with physicist Max Tegmark over why so many real-world learning problems have this compositional structure — Tegmark attributes it to physics itself being fundamentally local (atoms interacting with nearby atoms, planets with nearby planets), while Poggio raises the alternative (or complementary) possibility that brains evolved local, hierarchical wiring specifically because growing long-range neural connections is biologically costly, meaning organisms may simply be selected to be good at solving compositional problems (and to perceive the world as compositional) rather than compositionality being a pure physical fact independent of the observer — a distinction he frames as almost philosophical but empirically grounded in this specific mathematical property. He and colleagues proved that networks with a hierarchical, locally-connected (convolutional) architecture applied to genuinely compositional functions can escape the "curse of dimensionality" that otherwise makes shallow universal-approximation theorems (which he traces to concepts as old as 1980s Weierstrass-style polynomial approximation results) practically useless — the number of units needed to approximate a function normally scales exponentially with the number of input dimensions (a 200×200 pixel image having ~40,000 dimensions), but deep, compositional architectures avoid this blowup entirely.

On why stochastic gradient descent works, Poggio highlights the counterintuitive fact that modern neural networks massively over-parameterize relative to classical statistical wisdom (which favored roughly 2-10x more data points than parameters, the opposite of typical deep learning ratios), and describes using Bézout's theorem (an algebraic result estimating the number of solutions to a system of polynomial equations) to argue that heavily over-parameterized networks likely have vastly more global minima than there are atoms in the universe — explaining why gradient descent can find a good solution relatively easily, though he stresses that explaining why the particular solutions found also generalize well is a separate, still partially open question. He considers the universal approximation theorem for shallow networks unsurprising (having known equivalent 1980s-era results since entering the field) precisely because its practical uselessness (the curse of dimensionality) was already well understood; the genuinely interesting recent result, in his view, is proving that hierarchical/compositional structure specifically defeats that curse.

Poggio expresses skepticism, relative to prevailing enthusiasm, about GANs as a path toward reducing deep learning's reliance on massive labeled datasets — acknowledging their genuine value for computer graphics-style generative applications (citing his own older work training networks in the reverse direction, from pose parameters to realistic images, achieving comparable quality through simpler mechanisms) but cautioning against expecting GANs to meaningfully solve the "n going to one" labeled-data problem, framing this as a "no free lunch" expectation mismatch. On reducing the field's dependence on massive labeled data more broadly, he speculates (echoing his views on weak evolutionary priors) that early visual development may bootstrap from simple, evolutionarily conserved motion-detection machinery — since moving objects naturally segment themselves from a background, giving an organism free, label-free "clean" examples of individual objects during a critical early window, before backgrounds and static context get incorporated later.

Poggio explicitly distinguishes basic pattern/object recognition (an area he considers largely solved and now in a "golden age" of practical application, alongside speech recognition) from genuine scene, language, and action understanding — the latter, in his assessment, remaining far off despite popular claims otherwise. On existential AI risk, he says worrying early is generally good practice and supports safety research, but pushes back specifically on comparisons (attributed to Elon Musk, Nick Bostrom, and others) equating AI risk with nuclear weapons risk as misleading in terms of relative priority, given nuclear weapons' comparatively immediate and demonstrated danger; he estimates general human-level AI is roughly a century away (in contrast to Rodney Brooks's more skeptical ~200-year estimate he cites), while declining to hold this prediction with much confidence given the historical unreliability of technology forecasting.

On whether a future AGI system would be humanly understandable, Poggio revisits and extends a decades-old framework he co-developed with David Marr (adding "learning" as a fourth level of understanding alongside hardware, algorithm, and implementation) — arguing it's possible to have a deep, genuine understanding of how a learning system is constructed and trained without being able to predict or fully explain every specific thing it subsequently learns, analogous to understanding child development generally without being able to explain a particular child's specific taste in music. He considers ethics a plausibly learnable phenomenon rather than a purely philosophical mystery, citing fMRI evidence (work by colleague Rebecca Saxe and others) identifying specific brain regions involved in ethical judgment, and even experiments where transcranial magnetic stimulation of these regions measurably alters people's ethical decisions — suggesting a "neuroscience of ethics" (distinct from the ethics of neuroscience practice) could eventually inform how to engineer genuinely ethical machines.

On consciousness, Poggio notes the field remains divided even on defining terms (with theorists like Giulio Tononi proposing consciousness as a graded property present to some degree in all systems), and states that while most AI researchers would say consciousness isn't strictly necessary for a system to be "intelligent" in an engineering sense, he personally believes consciousness would be required to pass an extended, sufficiently rich Turing test — a position his own former student Christof Koch disagrees with. He's open but noncommittal on whether mortality/finitude is necessary for consciousness (citing Steve Jobs's Stanford commencement argument that finite life stimulates achievement) but doesn't think it's strictly required. Discussing an MIT "Quest for Intelligence" moonshot project on visual intelligence he leads, Poggio cites a striking Google X virtual-reality experiment in which subjects experiencing the world through a nearby robot's cameras/microphones reported an unshakable sense of embodied presence in the robot's location — suggesting scene understanding may be deeply intertwined with self-location and self-awareness, edging close to (without fully requiring) the hard problem of consciousness.

Reflecting on mentoring researchers who went on to prominent careers (including Demis Hassabis, Amnon Shashua, and Christof Koch), Poggio identifies curiosity, genuinely enjoying the process of discovery (especially collaboratively, with other curious and fun people), and disciplined ambition as the key ingredients — describing a cultural norm he values in good research groups where new ideas are met first with enthusiasm and only later, after a genuine hearing, with critical scrutiny, contrasting the more blunt, less socially cushioned critical style he experienced in Germany with a more diplomatically hedged American style. Closing with a reflection on the Flowers for Algernon story (about a mouse and a man who temporarily achieve genius intelligence before tragically losing it while remaining aware of the decline), Poggio says he doesn't believe intelligence and happiness are causally linked in either direction, expressing hope that happiness is simply "invariant" to intelligence level.

Excerpts