Brockman, Greg

co-founder and CTO of OpenAI

Greg Brockman: OpenAI and AGI

Lex Fridman Podcast · Apr 3, 2019

• Greg Brockman
• OpenAI
• AGI safety
• OpenAI LP capped-profit structure
• GPT-2 responsible disclosure
• AI policy and regulation
• self-play reinforcement learning
• OpenAI Five Dota 2
• Turing test and reasoning
• AI consciousness speculation

Core ideas

Brockman traces his own draw toward the digital world (over chemistry or mathematics, both early passions) to iteration speed and leverage: a program, like a mathematical proof, becomes part of "humanity's library" once written, but code scales to affect the whole planet in a way a proof or physical invention cannot. He extends this into a broader systems view — considering humans as information-processing systems, and going further to frame companies and even civilization itself as emergent, superhuman optimization processes with something like "a will of their own," citing Asimov's fictional concept of psychohistory as a useful (if speculative) way to think about large-scale collective behavior. On invention and historical inevitability, Brockman argues that transformative ideas tend to arrive via momentum shared across many independent researchers (citing simultaneous telephone invention and Einstein's relativity as likely-inevitable-eventually discoveries), meaning the real degree of freedom available to an innovator isn't inventing something wholly unique but rather setting the initial conditions under which a technology is born — citing the internet's open, academic founding culture and Wikipedia's ad-free structure as examples of formative decisions with outsized long-term consequences, and framing OpenAI's own strategy as an attempt to set similarly durable initial conditions for AGI.

Asked what he'd ask a powerful AGI system, Brockman says the first question should be how to ensure AGI's development goes well for humanity — treating the system's own answer as one valuable input among many (including internal colleagues) rather than a directive to defer to. He argues the positive case for AGI is underdiscussed relative to existential risk because it's inherently easier to imagine and rhetorically support destruction than construction (destroying requires getting one thing wrong; creating requires getting many things right), and because genuinely picturing a transformative technology's future impact is hard by nature (comparing the difficulty to explaining Uber, GPS, and smartphones to someone in 1950). He describes OpenAI's structure as split into three functions — capabilities (technical progress), safety (technical alignment mechanisms), and policy (governance questions like "aligned with whose values") — and argues that the common assumption that value alignment is intractable is empirically weakened by existing systems already learning things (like cat-vs-dog image recognition) that resist explicit specification; OpenAI's technical safety work aims to similarly learn human preferences from data, analogous to how a human child's values are shaped by data (experience and feedback) rather than explicit rule-writing.

Brockman gives an internal history of OpenAI's formation in 2015, framing the founding question as whether it was too late to build a new AI lab with top talent given the field's shift from academia to well-resourced industrial labs — a bet the founders judged "not obviously impossible." He traces renewed confidence in general intelligence's feasibility to a reassessment of deep learning's post-2012 track record: contrary to a common narrative crediting backpropagation's invention for the 1980s AI resurgence, Brockman argues (citing period sources) that increased available compute, not new algorithms, actually drove the 1980s uptick, and that the same simple neuron/backpropagation ideas from the 1950s-1980s only started working dramatically well once compute and data caught up around 2012 — leading him to identify three properties of deep learning (generality across a small toolset, competitiveness against decades of specialized approaches, and scalability with more compute/data) as jointly suggesting, for the first time, a credible paradigm toward general intelligence, even while acknowledging clear missing pieces (like reasoning).

On OpenAI's transition to the capped-profit "OpenAI LP" structure, Brockman explains the change was driven by the recognition that building AGI would require raising many billions of dollars — far beyond nonprofit fundraising capacity — while trying to preserve the founding mission (spelled out in a Charter developed over roughly a year of internal deliberation) that AGI's benefits, expected to vastly exceed those of any existing company, be legally owned by and returned to the world via the nonprofit, with investor and employee returns explicitly capped. He frames the Charter's built-in tension explicitly: OpenAI aims to build AGI itself where possible, but is equally committed to supporting another organization's safety-focused effort if it gets there first, treating competitive dynamics as subordinate to the actual mission (AGI going well for humanity, not any single organization "winning"). Asked why not simply pursue this mission as an ordinary for-profit company, Brockman argues nonprofits often achieve less real-world impact than for-profits precisely because for-profit structures are self-sustaining and can compound their own momentum, but that unconstrained profit incentives can also cause serious harm (citing deforestation-driven logging companies) absent the right guardrails — arguing OpenAI LP represents a deliberate middle path, with a legal fiduciary duty to the Charter (rather than to shareholders) as the mechanism meant to preserve that balance even under financial pressure, reinforced by a workplace culture where employees are explicitly encouraged and empowered to challenge leadership decisions that seem to drift from the Charter's intent.

On competitive dynamics in AGI development specifically, Brockman identifies a first "run-up" risk — a race dynamic (drawing a parallel to self-driving car development) that creates pressure to cut corners on safety to avoid losing — and states OpenAI's explicit commitment to stop competing and instead assist another team if that team is closer to safely achieving the mission, while noting this commitment can't be unilateral and depends on reciprocal commitments from other serious AGI developers. On government's role, Brockman describes OpenAI's current message to policymakers (via congressional testimony) as favoring measurement over premature regulation, arguing existing regulatory bodies (like NHTSA for self-driving cars) are appropriate for current narrow-AI applications, while genuinely new AGI-specific governance frameworks will require the field to first develop enough shared understanding to know what rules would even make sense, cautioning against prematurely stifling a nascent field.

On the decision not to release the full GPT-2 model, Brockman describes the underlying capability trajectory (scaling the same architecture from an unremarkable "academic toy" predecessor to suddenly-coherent prose) as forcing an early, genuinely uncertain judgment call, framing the internal disagreement itself (strong voices on both "obviously release it" and "obviously don't" sides) as evidence that caution was the correct default. He frames the decision explicitly as an attempt to establish a "responsible disclosure" norm for AI research analogous to the security research community's decades-long evolution of coordinated vulnerability disclosure — arguing the field currently lacks any equivalent norm and that GPT-2 was a deliberate test case for building one before a genuinely dangerous model arrives. He discusses concrete GPT-2 misuse concerns (fake news, abusive/biased generated content reflecting its internet training data, impersonation via personal message-history mimicry) alongside genuine creative-writing use cases people requested.

On authenticity and the future information ecosystem, Brockman argues that distinguishing human from AI-generated content is likely "a losing battle" in the same way CAPTCHAs have grown increasingly fragile as OCR and vision systems improved, predicting that trustworthy identity verification will need to shift toward authenticating the source of content (via identity/reputation systems) rather than analyzing the content itself, while acknowledging the privacy and anonymity tradeoffs this creates. He argues the deeper ethical line worth defending isn't whether content originates from a human or an AI, but whether deception is involved — invoking the film Her as a template for meaningful AI-human relationships that need not be "fake" simply because one participant is artificial, provided the human isn't being deceived about what they're interacting with.

On reasoning and the Turing test, Brockman argues genuinely passing an unrestricted Turing test (e.g., being able to teach calculus and solve novel calculus problems) requires reasoning capabilities beyond what current language modeling demonstrates, and is skeptical that simply scaling GPT-2 further ("GPT-20") would produce full reasoning on its own — arguing the "type signature" of current models (a fixed, non-adaptive amount of computation per output) differs structurally from how humans deploy variable amounts of "thinking" effort to refine answers, and suggesting future systems may need mechanisms resembling internal chains of intermediate thought rather than a single forward pass. He responds to Richard Sutton's "bitter lesson" essay (that scalable, compute-hungry general methods tend to beat hand-engineered, domain-specific ones) by agreeing with its spirit while rejecting a "compute is all that matters" reading — arguing OpenAI seeks ideas that are specifically scalable (get better with more compute and data) rather than either pure algorithmic cleverness or pure scale alone, and that meaningful research contributions remain possible without massive compute resources (citing GANs and VAEs as ideas discoverable at small scale, alongside OpenAI's own June 2018 GPT precursor, while cautioning that some emergent behaviors — like the surprisingly long-horizon strategic planning that emerged in OpenAI Five's Dota 2 agents — could not have been predicted from small-scale experiments even by the original algorithm's inventor).

Describing OpenAI Five's Dota 2 project, Brockman frames Dota's continuous, long-horizon, partially-observable structure as a meaningful step beyond discrete board games like chess and Go, describing pure self-play (agents improving purely by competing against evolving copies of themselves) as the core method, scaled from 1v1 to full 5v5 team play, and drawing an analogy between the resulting bots and "insect-like intelligence" — highly capable within their trained environment despite having no capacity for abstract reasoning, and displaying strong generalization to human opponents' qualitatively different playstyles despite having trained exclusively against other bots. He recounts the team's public loss at The International (playing competitively, close, ultimately losing both games) followed by dramatic short-term improvement (an 80% win rate against the tournament-era bot within two weeks), framing the project's real goal as advancing reinforcement learning generally rather than specifically defeating humans, with an upcoming final showcase match against world champions positioned as a capstone rather than the project's ultimate measure of success. He also cites the Dactyl robotic hand project — trained via simulation using techniques derived from the Dota system, and successfully transferring to physical hardware — as evidence that simulation-based training can generalize much further into real-world physical tasks than skeptics often assume.

Describing OpenAI's typical project lifecycle (roughly two years from a single researcher's small-scale signal of promise to a fully scaled, well-resourced team effort, as happened with both language modeling and Dota), Brockman announces a new "reasoning team" (co-led with Ilya Sutskever) aimed at getting neural networks to genuinely reason, citing mathematical logic, programming, and code security analysis as related domains sharing a common need for out-of-distribution generalization — while cautioning this is a long-term research bet unlikely to show major results within the immediate year. Closing on more speculative territory, Brockman declines to take a strong position on the simulation hypothesis (comparing it to unfalsifiable questions like "is there a flying teapot between Mars and Jupiter") but is more willing to engage with whether embodiment or physical grounding is necessary for intelligence, noting GPT-2's ability to generate plausible reasoning about fire, smoke, and physical concepts despite having no body or physical experience as evidence that the answer is "probably not." He extends this into speculation about whether sufufficiently capable reinforcement learning agents (even Dota bots) might possess some primitive, non-zero degree of consciousness, floating the idea that consciousness itself may be a "convenient computational shortcut" evolution converged on for beings that need to avoid pain and pursue survival — while explicitly flagging this as speculative and difficult to verify either way.

Excerpts