Vapnik, Vladimir

co-inventor of support vector machines, support vector clustering, and VC theory; professor at Columbia University

Vladimir Vapnik: Statistical Learning

Lex Fridman Podcast · Nov 16, 2018

• Vladimir Vapnik
• statistical learning theory
• VC dimension
• support vector machines
• invariance and predicates
• admissible function sets
• deep learning critique
• AlphaGo critique
• intelligence vs. statistics
• instrumentalism vs. realism

Core ideas

Vapnik opens by distinguishing two philosophical stances toward scientific laws — instrumentalism (a theory is simply a tool for prediction) and realism (a theory describes what is objectively, permanently true, effectively "God's law") — and situates his own machine learning work as instrumentalist: the practical goal is a rule for prediction, not necessarily an account of the true underlying conditional probability governing outcomes, though he notes the latter would constitute genuine "understanding" rather than mere predictive capability. Reflecting on Eugene Wigner's essay on the "unreasonable effectiveness of mathematics," Vapnik argues that careful mathematical derivation — not imaginative leaps or fantasy — is what actually reveals deep, previously-hidden truths about reality, illustrating with the least squares method: a few steps of careful algebraic derivation reveal that classical least squares implicitly discards meaningful information about the geometric configuration of observation points, a discovery he considers proof that raw human intuition is comparatively "primitive" and often can't see what rigorous derivation exposes. He extends this into a broader claim that even apparent moments of scientific genius (citing Einstein) are less about imagination than about correctly choosing axioms and rigorously following where they lead, with those axioms themselves representing accumulated collective wisdom refined across generations of scientists rather than any single flash of individual insight — and states he deliberately excludes imagination from his own work, since ideas like "features" or deep architectures that originate from imaginative interpretation, rather than being derived directly from the mathematics of the learning problem, tend not to actually solve the problem.

On learning theory, Vapnik proposes that two distinct mechanisms of convergence exist: the classical "strong" convergence mechanism (relying purely on statistical estimation from labeled examples, requiring the uniform law of large numbers and concepts like VC dimension) and a "weak" convergence mechanism that incorporates externally supplied predicates or invariants — general knowledge about a problem domain, of the kind a good teacher supplies — which can dramatically reduce the number of training examples needed. He illustrates this with the English proverb "if it looks like a duck, swims like a duck, and quacks like a duck, it's probably a duck": each predicate ("looks like," "swims like," "quacks like") functions as a filter that eliminates large swaths of the space of candidate classification functions, and crucially, a good teacher knows which predicates are actually informative (swimming is relevant to duck identification) versus technically valid but useless (jumping is not, since it carries no discriminating information) — a judgment call Vapnik identifies as the genuinely unsolved "intelligence" component of learning, distinct from the well-understood statistical machinery.

Vapnik frames the actual goal of learning theory as constructing an "admissible set of functions" — a restricted hypothesis space with small VC dimension (capacity/diversity) that nonetheless contains a genuinely good solution function — arguing that virtually all classical statistical learning theory (as taught in textbooks) simply assumes this admissible set is already given, sidestepping what he considers the truly hard problem: how to actually construct such a well-chosen restricted space from a much larger, less constrained set of candidate functions. His answer is that invariants (properties/predicates whose average value should match between training data and the learned model) are precisely the mechanism for constructing admissible function sets, with the specific choice of which invariants to use (general-purpose ones like "looks like," versus problem-specific ones tailored to a particular domain) constituting the "intelligence part" of the process — a part he states plainly that neither he nor the field currently understands, comparing it to a great piano teacher's instruction to "play like a butterfly," which demonstrably and reliably improves a student's playing despite nobody being able to formally explain what information is actually being transmitted.

On deep learning, Vapnik is sharply critical, characterizing it (along with concepts like "features") as unmoored fantasy/interpretation rather than mathematically grounded reasoning — arguing that mathematics itself "does not know" neurons or deep architectures, only functions (e.g., piecewise-linear functions), and that appeals to brain-inspired justification are especially weak since neuroscience doesn't actually understand how the brain works either. He draws an extended historical analogy to Churchill's account of WWI peace negotiations (contrasting professional, restrained aristocratic dealmaking after earlier wars with the greedy, amateur mismanagement of the Treaty of Versailles) to characterize a similar dynamic in machine learning: mathematically rigorous researchers versus computer scientists who, in his view, generate loosely justified interpretive narratives (like deep learning) rather than working from first principles. He cites the Representer theorem as showing that the mathematically optimal solution to standard learning problems lies in a shallow, not deep, network, and directly challenges AlphaGo's celebrated success at Go — arguing that beating humans at Go, given the amount of training data (and, in his telling, non-optimal deep learning methods) required, actually demonstrates that Go was "not a very difficult problem" from a formal standpoint, rather than validating deep learning's underlying method, and poses his own challenge: solving comparable problems (like MNIST digit recognition) using roughly 100x less training data by properly incorporating informative invariants instead of brute-force scale.

Discussing intelligence more broadly, Vapnik expresses reserved skepticism about whether intelligence is purely internal to an individual mind, citing the historical pattern of multiple independent researchers proving the same difficult theorem (or discovering non-Euclidean geometry, as with Lobachevsky, Gauss, and Bolyai) within roughly the same decade as suggestive evidence that mathematical discovery might connect to something like a distributed, shared "world intelligence" rather than being purely private cognition — an idea he presents as genuine open speculation, not a settled claim. On worst-case complexity analysis (P vs. NP, VC-dimension-style bounds), he defends the value of studying mathematical extremes (best-case and worst-case bounds) over messier "average case" or intermediate scenarios, arguing that intermediate cases resist general mathematical description and that the edges — while harder to derive exact bounds for — are where genuinely useful, generalizable principles tend to be discovered.

Vapnik states his recently completed work on invariance represents, in his own assessment, a genuinely complete theoretical framework for learning — one that formally identifies only two possible convergence mechanisms (statistical/strong and invariant-based/weak) and thereby cleanly separates the well-understood statistical machinery of learning from the unsolved "intelligence" problem of how predicates/invariants themselves are generated by a teacher or mind. He frames this separation as itself valuable progress: even without solving the intelligence problem, correctly identifying its boundary and formal role clarifies what future work must actually target, and he offers a concrete open research challenge for the field — using a small number of well-chosen invariants (e.g., digit-specific geometric properties like relative horizontal symmetry for distinguishing "3" from "2") to match deep learning's MNIST digit-recognition accuracy using roughly a hundredfold reduction in training examples, framing this as a genuine test of whether machine learning has actually incorporated real intelligence versus simply exploiting statistical scale.

Closing on a more personal and philosophical note, Vapnik connects his mathematical worldview to an appreciation of "ground truths" he says he perceives in both music (citing Bach's clear, classical structural simplicity as directly analogous to elegant geometric axioms) and, more selectively, in poetry and philosophy, describing moments of mathematical discovery as genuinely joyful but requiring repeated, honest self-scrutiny to distinguish a fleeting personal impression from something that actually connects to durable, general truth — citing his own early confidence (shared at the time only by mathematician Richard Dudley) that statistical learning theory and support vector machines would prove lasting contributions, a conviction later vindicated by their widespread adoption, and framing his newer invariance-based theory of learning with the same degree of confidence that it represents a genuinely complete account of the mechanisms of learning.

Excerpts