Vapnik, Vladimir

co-inventor of support vector machines, support vector clustering, and VC theory; professor at Columbia University; second appearance on the podcast

Vladimir Vapnik: Predicates, Invariants, and the Essence of Intelligence

Lex Fridman Podcast · Feb 14, 2020

• Vladimir Vapnik
• statistical learning theory
• predicates
• VC dimension
• weak convergence
• Plato's theory of forms
• Vladimir Propp
• admissible functions
• privileged information
• MNIST challenge

Core ideas

Vapnik opens by drawing a sharp distinction between engineering intelligence (imitating human behavior/function regardless of mechanism — the spirit of Turing's project) and the science of understanding what intelligence actually is, which he considers a fundamentally different and, to him, more important pursuit. He grounds his own theory of intelligence explicitly in Plato's theory of forms: there exists a small "world of ideas" (pure abstract concepts) and a much larger "world of things" (concrete reality), connected by a projection, and he argues intelligence consists precisely in the ability to construct that projection — going from limited observations of reality back toward the small set of abstract ideas that generated them. He connects this directly to Vladimir Propp's 1928 "Morphology of the Folk Tale," which identified just 31 recurring structural "units" (predicates) that describe the narrative structure of Russian folk tales and, notably, generalize far beyond folklore to films, TV, and everyday storytelling — offering this as evidence that the "world of ideas" governing even complex human phenomena may be surprisingly small.

Vapnik formalizes a "predicate" as simply a function (e.g., mapping pixel space to a real number) whose value, combined via inner product, measures some property of the data — his running example being degree of symmetry (and its many variants: vertical, diagonal, anti-symmetry) in handwritten digit images. His central technical claim is that a good predicate dramatically shrinks the "admissible set of functions" a learning algorithm must search within (formally reducing the VC dimension of that hypothesis space), meaning that if you have access to a small number of very good predicates, you need drastically less training data to learn — this is the mathematical content of his claim that "very few good predicates" constitute the essence of intelligent learning, in contrast to systems (like standard deep neural networks, which he says amount to a very large but still limited set of piecewise-linear functions) that lack this kind of explicit, principled constraint and thus require large amounts of data to compensate. He credits Yann LeCun's convolutional architecture as an example of a genuinely good predicate (translation invariance) discovered 25 years ago, but argues that instead of building elaborate "deep network" architectures around it, the same invariant could be more transparently expressed as a constraint on an admissible set of functions.

A key mathematical distinction Vapnik develops is between strong convergence (pointwise/functional convergence, where two functions' squared difference integrates to something small) and weak convergence (where only certain integral/aggregate properties, like inner products against admissible predicate functions, need to converge, not the function pointwise). He argues the field of machine learning has for 50 years focused almost exclusively on strong convergence (as in standard empirical risk minimization theory) while overlooking weak convergence — which he says is actually how humans intuitively reason (e.g., "looks like a duck, walks like a duck" — matching aggregate properties rather than exact replication) — and that recognizing this omission is what let him connect his own statistical learning theory back to Plato's framework: predicates act as the "admissible" constraints that narrow a Hilbert space of candidate functions down using only weak-convergence properties measured from relatively small training samples (invoking the classical Law of Large Numbers, not the stronger uniform Law of Large Numbers that his own earlier VC theory work relied on). He frames the challenge of beating MNIST digit-recognition performance using roughly 100x fewer examples (60 examples per digit rather than 60,000) as a deliberately simple, Einstein-style ("as simple as possible, but not simpler") test case for discovering universal image-understanding predicates, and states his belief that solving this specific challenge would generalize essentially immediately to broader 2D natural image understanding — while being candid that this remains hopeful conjecture rather than proven fact, and that he does not believe symbolic/logic-based AI approaches (from 1980s expert systems) can substitute for genuinely discovering the right predicates, since doing so requires deep, often inarticulable knowledge of "life" and reality akin to what a skilled human observer (a music critic, a folklorist like Propp) brings.

Vapnik expresses skepticism that reasoning mechanisms involving explicit recurrence or sequential multi-step "thinking" are necessary for the problems he's focused on, saying he has never encountered a case in his own formal problem-solving that required it, while readily admitting that broader human reasoning — especially the process by which people generate good questions in the first place — remains something he doesn't understand and considers extremely difficult. He recounts a striking personal anecdote as evidence for the existence and power of cross-domain "privileged information": collaborating with a professor of Russian poetry who described a set of ~50 handwritten digit images purely using vocabulary and imagery drawn from Russian poetry, which he and colleagues then used as auxiliary "privileged information" input (a technique he calls "learning using privileged information") that measurably improved digit-recognition performance — offering this as direct evidence that abstract, art-derived descriptive concepts genuinely connect to the same underlying "world of ideas" that governs simple visual pattern recognition. He expresses a similar hope (unfulfilled due to his collaborator's death) of systematically mining how music critics describe Bach versus Chopin to extract the vocabulary of universal predicates, believing this kind of humanistic, descriptive analysis could directly inform the search for machine-learning invariants. On mortality, he says he doesn't fear death much but regrets not having time to pursue this cross-domain "predicates in art" research program, and closes reflecting humbly on how much smarter he finds certain literary scholars and public intellectuals at intuitively grasping the deep "blocks" (predicates) of human life and behavior, in contrast to his own narrower formal expertise.

Excerpts