Lambert, Nathan
▶ Watch on YouTubeCore ideas
The conversation is a technical survey of the state of AI heading into 2026, anchored by the "DeepSeek moment" (January 2025) as the inflection point that turned the open-weight Chinese ecosystem (DeepSeek, then Kimi, MiniMax, Z.ai/GLM, Qwen) into serious competitors to Western labs. Both guests reject a winner-take-all framing: Raschka argues ideas diffuse too quickly between labs (researchers rotate jobs, techniques get copied within months) for any durable technical moat to exist, so the real differentiator is capital and compute access, not secret sauce. Lambert frames the current US frontier as split by culture and specialization rather than raw capability: Anthropic's disciplined, code-focused organization is currently "winning" developer mindshare with Claude Code and Opus 4.5, Google has structural infrastructure advantages (TPUs avoid NVIDIA's margin, and Google can separate research from product), and OpenAI remains the most likely source of a genuinely new research paradigm despite operational chaos, because of its track record (o1, Deep Research, Sora). Chinese open models are eating Western open-source dominance (implicitly including Meta's LLaMA, pointedly never named by either guest until prompted) because of friendlier licenses and because releasing openly is a deliberate strategy for Chinese labs to build international influence and offset their inability to sell API access to security-conscious Western enterprises.
A large stretch of the conversation is a technical primer on how models are actually built: architecture has barely changed since GPT-2 (still an autoregressive decoder-only transformer) — what's changed is a set of well-understood tweaks (Mixture-of-Experts routing to add capacity without proportional compute cost, multi-head latent attention, group-query attention, sliding-window attention) plus systems-level throughput gains (FP8/FP4 training) and, above all, changes to how models are trained: pre-training (raw next-token prediction, now with heavy synthetic/rephrased data and OCR-extracted PDF corpora), mid-training (a newer, more curated intermediate phase for long-context and quality data), and post-training — where Reinforcement Learning with Verifiable Rewards (RLVR, a term Lambert's own Tulu 3 team coined before DeepSeek popularized the technique) is identified as 2025's single biggest driver of visible capability change: letting models try math/code problems repeatedly against a checkable reward unlocked long, self-correcting "reasoning traces," tool use, and the inference-time scaling behind reasoning models like o1 and R1's much-discussed "aha moment." Both are careful to caveat this: contamination in benchmarks like Qwen's suspiciously high MATH-500 scores means some claimed post-training gains may be partly memorization rather than genuine skill acquisition. They lay out competing scaling axes — pre-training compute, RL/post-training compute, and inference-time compute — arguing all three are "still alive" but pre-training has become primarily a capital-allocation question (is a marginally better base model worth the now-enormous training cost versus spending on inference scaling instead) rather than a question of whether scaling laws hold, which Lambert insists they still do, having held for "13 orders of magnitude."
On work and craft, both describe an anxious but ultimately optimistic view of AI's effect on human expertise: they distinguish a genuine "struggle is where learning happens" pedagogy (Raschka builds his books around forcing readers to implement things from scratch, matching against reference model weights as a "verifiable reward" for understanding) from an over-reliance on AI that risks producing practitioners who can supervise code they never learned to write themselves — while noting data that senior, not junior, developers actually ship more AI-generated code, because expertise makes them better at directing and verifying it. On careers, Lambert gives blunt practical advice favoring industry-lab jobs (well-compensated, high-impact, if less publication-credited) over academia for most people, while reserving genuine paradigm-shifting research for people willing to sacrifice income and ignore language models entirely. Politically, Lambert discusses his "Adam Project" (originally "American DeepSeek Project") — advocacy, backed partly by NSF grants to his own Allen Institute for AI, to get the US building state-of-the-art open-weight models, arguing this is about controlling where AI research and talent development happens, not a national-security "Manhattan Project" framing that both guests explicitly reject as premature and probably impossible to enforce given how cheap (roughly $1-100M) training a strong open model has become. They close on AGI skepticism (the "one model to rule everything" dream is "kind of dying" in favor of specialized, tool-integrated agents), a prediction that computer/tool use — not raw intelligence — is the actual bottleneck to economically transformative AI, and a shared belief that human authenticity, physical experience, and community will become more prized, not less, as AI-generated "slop" saturates digital life.
Excerpts
On why post-training/RLVR represented the year's central technical shift, and what it unlocked:
“The reasoning process when the models are generating a lot of tokens... it tries a tool, it looks at what it gets back, it tries another API, it sees what it gets back and if it solves the problem. The models, when you're training them, very quickly learn to do this... [it] gives this kind of general foundation where the model can use CLI commands very nicely in your repo, handle Git for you, move things around, organize things, or search to find more information—which, if we were sitting in these chairs a year ago, is something that we didn't really think of the models doing.”
On the suspicion that some post-training benchmark gains are contamination rather than genuine reasoning, debated live between the two guests:
“Raschka: "I was training the Qwen 3 base model with RLVR on MATH-500. The base model had an accuracy of about 15%. Just 50 steps... the model went from 15% to 50% accuracy. And you can't tell me it's learning anything fundamentally about math." Lambert: "There's been two papers this year... that talks about data contamination in Qwen... they train on problems that are almost identical to MATH... it's been this big debate in the research community: how much of these reinforcement learning papers... can you believe them?”
On why the "one model to rule everything" version of AGI is losing ground to a more specialized, agentic vision, and the deeper bottleneck to real economic transformation:
“Lambert: "I think that dream is actually kind of dying... it depends a lot on integrations and other things... I think it goes back to what is the 'one model to rule everything' ethos, which is just like a thing in the cloud that handles your entire digital life." Fridman frames the harder open question as tool use itself: "It boils down to the scientific question of, 'How hard is tool use to solve?'... We saw multiple demos in 2025 of, like, Claude can use your computer, or OpenAI had operator, and they all suck.”