Patel, Dylan & Lambert, Nathan
▶ Watch on YouTubeCore ideas
Recorded in the immediate aftermath of DeepSeek's January 2025 release, the conversation opens by disentangling DeepSeek-V3 (a mixture-of-experts base model post-trained into a standard instruction-following chat model) from DeepSeek-R1 (a separate reasoning model trained from the same base via large-scale reinforcement learning). Lambert walks through the pre-training/post-training distinction and the newer "reasoning" stage — RL against verifiable rewards (exact-match math answers, passing unit tests) rather than human preference data — and stresses that DeepSeek's MIT-licensed, fully-detailed technical reports make it unusually open by frontier-lab standards, even though the training data and code themselves aren't released. Patel explains the two central architectural innovations behind DeepSeek's efficiency: an extreme mixture-of-experts design (256 experts with only 8 active, a much higher sparsity ratio than prior MoE models like Mixtral) that lets a 671B-parameter model activate only ~37B parameters per token, and multi-head latent attention (MLA), a novel compressed-attention mechanism that cuts KV-cache memory usage by roughly 80-90%. Achieving this required DeepSeek to hand-schedule GPU streaming-multiprocessor communication in PTX (near-assembly-level CUDA) to work around the reduced interconnect bandwidth of their export-restricted H800 GPUs — necessity-driven low-level engineering that a well-resourced lab wouldn't normally bother with.
On compute, Patel argues DeepSeek's parent company, the quant hedge fund High-Flyer, has likely amassed closer to 50,000 GPUs total (versus the ~2,000 H800s DeepSeek publicly cited for V3's pre-training run alone), with the hedge fund subsidizing AI research as essentially a loss leader and recruiting tool — not, in his and Lambert's assessment, a Chinese-government-subsidized effort, since DeepSeek's CEO Liang Wenfeng operates independently of state-aligned labs like Huawei's. They dissect why DeepSeek's inference pricing (R1 roughly 27x cheaper than OpenAI's o1) reflects a combination of OpenAI's fat margins (~75% gross margin on inference), a genuine architectural efficiency advantage, and DeepSeek's total inability to actually serve the model at scale (it has far fewer inference GPUs than OpenAI/Microsoft, and quickly ran out of capacity, disabling new sign-ups). They also frame DeepSeek's rapid, unpolished shipping cadence (releasing on December 26th and during Lunar New Year) as a genuine cultural advantage over safety-conscious US labs like Anthropic, which reportedly hold back more capable reasoning models specifically because unfiltered chain-of-thought output is harder to make safe — a dynamic they compare to the Soviet space program's higher risk tolerance beating the US to orbit.
A long section addresses export controls and geopolitics: Patel argues restricting GPU exports to China only "wins" long-term for the US if transformative AI arrives within a 5-10 year window, since China's industrial capacity, engineering talent, and power-generation ability all exceed America's raw potential. He details the specific H100/H800/H20 chip variants and how U.S. restrictions have shifted from a two-factor (interconnect + FLOPs) framework to a FLOPs-only cap, ironically making the legally-exportable H20 chip better suited to reasoning-model inference (which is memory- and interconnect-bound) than to pre-training. Both express concern that export controls could push China toward military action on Taiwan, drawing on the historical pattern that global hegemonic transitions rarely happen peacefully, while also noting the two countries are simultaneously walling off from each other (Chinese firms can't access US tech, US firms are locked out of China) in a dynamic reminiscent of Cold War bipolarity. They walk through why TSMC's Taiwan dominance is nearly irreplaceable in the near term — a culture of extreme specialization and work ethic, R&D concentrated in only three world locations (Hsinchu, Hillsboro Oregon, and Pyongtaek South Korea), and the staggering capital cost of fabs — while conceding the U.S. could reshore manufacturing given a decade and roughly a trillion dollars of investment and immigration-driven talent import.
On AGI timelines, Lambert and Patel both say they're "feeling the AGI" (rapid, surprising progress) without committing to Dario Amodei's 2026 "powerful AI" prediction; Lex offers 2030 as his own guess for when autonomous AI agents/robotics become a genuine military and economic security threat, and the group agrees that AGI's economic impact will be gated less by raw capability than by the sheer dollar cost of inference at scale (an AGI-level query can cost $5-20 versus cents for a normal chat response). They discuss Andrej Karpathy's tweet framing reasoning-model "aha moments" as analogous to AlphaZero's shift from imitation learning to pure self-play/reinforcement learning, and speculate that the next "move 37" moment for language models will likely emerge from verifiable domains like computer-use agents or robotics rather than open-ended scientific discovery, since infinite self-play against a verifiable reward (did the code compile, did the account get created, did money get made) is what unlocked Go and could similarly unlock generalized agentic behavior.
The conversation covers censorship and alignment across the pipeline — pre-training data filtering (hard to fully scrub facts like Tiananmen Square from a model without also scrubbing them from the internet), post-training/RLHF shaping (illustrated by Llama 2's over-cautious "too much safety" era and Google Gemini's "Black Nazi" image-generation failure, attributed to a bungled prompt-rewriting system rather than the base model), and the more speculative risk of intentionally embedded ideological "backdoors" in widely-adopted open-weight models, especially as models move from answering questions to autonomously executing tasks (citing Sam Altman's line that "superhuman persuasion will happen before superhuman intelligence"). They discuss AI megaclusters in detail: Elon Musk's Memphis xAI cluster (200,000 GPUs, built in record time using water cooling, on-site natural gas generation, and Tesla Megapacks to handle power transients), OpenAI/Oracle's planned 2.2-gigawatt Stargate site in Abilene, Texas, Meta's and Google's comparable buildouts, and Google's little-known but larger multi-site TPU "super regions" — with Patel arguing Google under-monetizes its TPU/infrastructure advantage because of internal organizational silos between Search, DeepMind, Cloud, and the TPU team.
On the DeepSeek/OpenAI-training-data controversy, both push back on treating it as clearly unethical, noting that OpenAI itself trained on scraped internet text without consent, that model distillation (training on a stronger model's outputs) is standard industry practice, and that the real issue is a legally murky terms-of-service violation rather than anything illegal. Looking at business models, they argue chat-based API access is becoming commoditized and cheap fast (Patel cites a roughly 1200x cost-per-intelligence drop from GPT-3 to comparable-quality models in about three years), meaning OpenAI and Anthropic's long-term survival depends on capturing value in agentic/task-based domains rather than chat, whereas Google, Meta, and xAI can profit from AI indirectly through existing products (recommendation systems, ads, robotics). They close discussing open-source AI's future — Lambert's own Tülu model release, the deliberately permissive MIT license behind DeepSeek-R1 versus Meta's more restrictive Llama license, and Lambert's stated motivation that broad societal access to understanding AI, not just raw capability, is what open-sourcing should be optimizing for — before assessing the (still largely unfunded) Stargate announcement's actual financial backing, and closing on cautiously optimistic reflections about humanity's long-term trajectory, with Patel flagging techno-authoritarian concentration of power (via brain-computer interfaces and AI-augmented individuals) as more worrying to him than a runaway AI takeover scenario.
Excerpts
On DeepSeek's necessity-driven low-level GPU engineering under export restrictions:
“DeepSeek... had to figure out how to get efficiencies. And one of those things is that instead of just calling the Nvidia library NCCL, they scheduled their own communications... they scheduled specific SMs... And they were specifically scheduling, hey, which ones are running the model? Which ones are doing allreduce? Which one are doing allgather? And they would flip back and forth between them. And this requires extremely low level programming.”
On why export controls are a bet against China's underlying industrial advantage:
“If you believe we're in this sort of stage of economic growth and change that we've been in for the last 20 years, the export controls are absolutely guaranteeing that China will win long-term... it's reasonable... once you get to, hey, these timelines are below that time period, then the only way to create a sizable advantage or disadvantage for America versus China is if you constrain [on] compute.”
On Andrej Karpathy's framing of reinforcement learning as the source of emergent reasoning behavior, quoted in full during the episode:
“Almost every single shocking result of deep learning and the source of all magic is always [reinforcement learning, not imitation]... The model could never learn this with the imitation because the cognition of the model and the cognition of the human labeler is different... They have to be discovered during reinforcement learning as empirically and statistically useful towards the final outcome.”