Howard, Jeremy

founder of fast.ai; distinguished research scientist at the University of San Francisco; former president of Kaggle

Jeremy Howard: fast.ai Deep Learning Courses and Research

Lex Fridman Podcast · Aug 27, 2019

• Jeremy Howard
• fast.ai
• transfer learning
• DAWNBench
• PyTorch and TensorFlow
• learning rate and super-convergence
• deep learning education
• active learning
• medical AI
• AI-driven labor displacement

Core ideas

Howard traces his programming journey through an unusually eclectic set of languages — from Microsoft Access/VBA (which he still considers an unmatched environment for tying data, UI, and logic together) and Delphi (a fast, compiled language as easy to use as Visual Basic) to the array-oriented APL/J family, which he singles out as uniquely elegant: built on the idea of math notation as a "tool for thought," J lets a programmer express dense computation without loops via extreme broadcasting, keeping entire programs comprehensible in a single screen. He connects this lineage to K, an extremely expensive but blazingly fast array language used by elite hedge funds, and frames his ultimate hope for the field around Swift, whose "infinite hackability" (per Chris Lattner's design goal) he wants to replace Python — which he considers fundamentally "unhackable" because its slowness forces researchers down into raw CUDA C the moment they need something Python's ecosystem doesn't already provide (his examples: nobody researches sparse convolutional networks or efficient RNN internals because rewriting CUDA kernels is prohibitively specialized work). He describes new compiler infrastructure (Halide, MLIR, tensor comprehensions) as a promising path toward writing GPU kernels in high-level, composable, J/APL-like syntax that a compiler optimizes automatically, potentially breaking NVIDIA's effective monopoly (helped further by Google's deliberate choice to make TPUs largely unprogrammable to protect their IP).

Howard founded fast.ai after running Enlitic, the first company applying deep learning to medicine, where he learned firsthand how vast the developing world's shortage of specialists is (citing just five pediatric radiologists for the entire African continent outside South Africa) and how deep learning could triage the ~95% of cases needing no expert attention, letting scarce specialists focus only on genuine risks — while stressing that full automation isn't the goal, since more productive experts don't change the economics of removing them, and humans remain better at many things machines aren't. He identifies the primary barrier to medical AI adoption as not technology or even privacy regulation per se (he notes HIPAA is actually about data portability) but hospital lawyers' risk-averse interpretation of ambiguous rules, which rewards inaction since data-sharing benefits are diffuse and abstract while privacy violations are concrete and attributable. Frustrated that he alone couldn't identify and pursue every high-value domain for deep learning, Howard founded fast.ai on the philosophy of empowering existing domain experts (who already have the data and problem knowledge but lack ML skills) rather than trying to become a domain expert himself in every field — a reflection, he notes, of his own consulting background at McKinsey giving him direct respect for where value actually gets created in organizations.

Howard is sharply critical of academic deep learning research incentives, arguing that peer-review culture pushes researchers toward safe, incremental work recognizable to reviewers rather than practically transformative but under-studied areas like transfer learning and active learning — illustrating with his own (reluctantly published, largely because a colleague wrote it up) ULMFiT paper, which introduced effective transfer learning to NLP and smashed a major benchmark within days of prototyping, despite him having no prior NLP background; he sees the subsequent explosion of transfer learning in NLP (GPT, BERT) as proof the idea just needed someone to demonstrate it. He also recounts how Leslie Smith's "super-convergence" discovery — that certain networks could train 10x faster using unexpectedly high learning rates via a gradual warmup schedule — went unpublished for years because peer review in deep learning (unlike physics) refuses unexplained empirical results, a norm Howard considers a real structural flaw compared to how other experimental sciences operate.

On DAWNBench, Howard recounts his team's roughly ad hoc entry (found with only ten days left) into a Stanford competition to train CIFAR-10 and ImageNet models as fast and cheaply as possible, where his team's disciplined, resource-constrained (single-GPU-first, eventually scaling only when needed) approach beat entrants from Google and Intel — key tricks included progressively training on smaller images before fine-tuning on full resolution (since low-resolution features transfer, much as showing a child blurry pictures of a dog before a sharp one still conveys "dog") and applying Smith's super-convergence learning-rate schedules. He generalizes this into a broader research philosophy: large-scale infrastructure and massive datasets (ImageNet's 1.3 million images) often aren't necessary and can actively suppress creativity by making experimentation slow and expensive — motivating his release of smaller curated subsets (Imagenette, Image Woof) that train comparably in minutes on a single GPU while remaining predictive of full-ImageNet results, and his broader criticism of vendors like Google and IBM whose business incentives push toward "more data and more compute" as the only path to good results, discouraging newcomers who wrongly conclude deep learning is inaccessible outside a few large companies.

He extends this "less is more" philosophy across several projects: DeOldify (a student's single-GPU black-and-white movie colorization tool achieving best-in-class results, developed using a discovered simpler loss function that removes the need for adversarial GAN training entirely) and an unrealized but "eminently doable" opportunity he identifies in audio — analogous to how computational photography now routinely combines multiple cheap camera sensors (or a single sensor with slight intentional movement) to reconstruct high-quality images (e.g., low-light photography), an equivalent multi-microphone audio reconstruction and denoising approach remains, in his view, surprisingly under-explored. On optimization more broadly, Howard argues the field is moving toward eliminating manually-tuned hyperparameters like learning rate altogether as researchers better understand the interacting dynamics of weight decay, learning rate schedules, and optimizer internals (citing his lab's own work on "discriminative learning rates" for transfer learning), predicting this will make deep learning progressively more accessible to domain experts without requiring a dedicated ML specialist to babysit training.

On tooling history, Howard traces his lab's own migration from Theano/Keras and TensorFlow/Keras (both requiring upfront static computational graphs that were hard to debug or use interactively) to PyTorch (allowing normal, interactive Python-native model development) and finally to their own fastai library — a multi-layered API letting a beginner train a state-of-the-art model in three lines of code while still allowing progressive, deep customization down to low-level control. He's highly critical of TensorFlow 2.0's "eager" mode as superficial syntax sugar layered atop a fundamentally disorganized, technical-debt-laden codebase (citing ~10x slower per-step performance than PyTorch for from-scratch model development), attributing this to TensorFlow's large, hastily-organized original engineering team, and expresses cautious optimism that "Swift for TensorFlow" (built on MLIR and LLVM, with the ability to directly call existing Python code as an interim bridge) could eventually be a genuinely better foundation, once Swift's ecosystem develops the numeric-computing culture and libraries it currently lacks.

Reflecting on teaching thousands of students through fast.ai's free courses, Howard states his strongest conviction is that "anyone can do it" — he's repeatedly seen non-programmers and domain experts from wildly different backgrounds achieve state-of-the-art results — and that the single decisive differentiator between success and failure is tenacity, not raw aptitude, statistics background, or credentials (in fact, he notes people with strong classical statistics training sometimes struggle more, since their intuitions about minimizing parameters run directly counter to deep learning practice). His core pedagogical advice is to immediately fine-tune real models on data the learner actually cares about (rather than merely running pre-trained models or working through purely theoretical exercises), then study a model's actual inputs, outputs, and misclassifications as a tool for becoming a better domain expert and for catching problems like data leakage. He advises aspiring practitioners to become deep learning experts specifically within their own domain of passion and existing expertise (medicine, ecology, language bias, ornithology) rather than chasing incremental improvements on already heavily-studied benchmarks, arguing that genuinely interesting research requires understanding a real problem well enough to know whether your results are actually good — comparing generic benchmark-driven research to studying self-driving cars without ever having driven a car.

On startups, Howard credits his own two self-funded companies' success to extreme cost discipline (profitable within six months in both cases, by charging users directly rather than seeking scale first) and argues this path was less psychologically stressful than venture-backed growth, where investors' "grow, grow, grow" pressure can push founders away from what they actually believe the business needs; he's comfortable with either a modest "lifestyle" exit or high-growth VC path as valid outcomes, provided the choice matches the founder's actual goals — while cautioning against the common mistake of commercializing a deep learning PhD topic that was chosen for its research interest rather than because it solves a real, validated problem. Personally, Howard uses spaced repetition (via Anki) exclusively for Mandarin vocabulary — a domain where he can't simply "look it up on Google" in the moment — while handling all other technical learning through deep engagement and daily practice (a decades-long habit of spending at least half of every day learning or practicing something new) rather than flashcard memorization, and stresses that constructing memorable, personally resonant mnemonic stories is essential to space-repetition’s effectiveness. Closing on the future, Howard declines to predict human-level AI timelines as unproductive speculation given the total absence of relevant data, and instead identifies AI-driven labor force displacement and the resulting hollowing-out of the middle class as a genuine, present-day societal concern he takes far more seriously — alongside a call for individual data scientists to recognize the high-leverage, ethically consequential nature of their work rather than viewing themselves as narrowly technical engineers exempt from broader responsibility.

Excerpts