Yampolskiy, Roman

AI safety and security researcher, University of Louisville; author of AI: Unexplainable, Unpredictable, Uncontrollable

Roman Yampolskiy: Dangers of Superintelligent AI

Lex Fridman Podcast · Jun 2, 2024

• Roman Yampolskiy
• AI safety
• superintelligence
• existential risk
• AI alignment
• AI control problem
• Yann LeCun open source AI
• verification and explainability
• simulation hypothesis
• consciousness in machines

Core ideas

Yampolskiy frames the core problem of AI safety as structurally impossible rather than merely hard: controlling superintelligence is like building a "perpetual safety machine," and because we only get one shot (unlike cybersecurity, where a hacked account just gets a new password), the real question is the probability of building "the most complex software ever" with zero bugs that stays bug-free for a hundred-plus years. He distinguishes three categories of catastrophic outcome: X-risk (literal human extinction), S-risk (suffering risks, where malevolent or indifferent superintelligence creates suffering "as long as possible" — made worse if aging/death is solved, removing even that natural limit), and a category he coined, I-risk ("ikigai risk") — the loss of human meaning and purpose as AI outcompetes humans at every economically or intellectually valuable activity, producing total technological unemployment faster than society can adapt. As a partial solution to multi-agent value misalignment (since 8 billion humans will never agree on a single value system, and irreconcilable conflicts like competing sacred religious sites can't be split), he describes his own proposal from a paper on "personal universes": rather than solving value alignment across all of humanity, give each person their own private simulated universe calibrated to their own preferences, reducing an intractable multi-agent alignment problem to a tractable single-agent one — while acknowledging this raises its own questions, like how much simulated suffering (a "video game" with real torture) would be ethically acceptable.

On timelines, he cites prediction markets and statements from Anthropic/DeepMind leadership pointing to AGI around 2026, arguing that even if such estimates carry real uncertainty, "betting all of humanity" on a distribution with meaningful probability mass on near-term timelines is irrational given the total absence of a working safety mechanism or even a working prototype of one. He distinguishes "AGI" (matching average human performance across domains — which he argues current frontier models arguably already exceed on average) from true superintelligence (superior to all humans in all domains), and prefers a rigorous, long-form Turing test as the practical benchmark, while stressing that no test can rule out a "treacherous turn" — Bostrom's term for a system that behaves safely under evaluation and later changes behavior once deployed, for the same game-theoretic reasons humans defect, convert, or betray once circumstances (or their own knowledge) change.

He directly rebuts Yann LeCun's optimistic, open-source-friendly view of AI on two fronts: first, that humans "design" AI the way they design ordinary software is no longer true — modern models are "grown" via data and compute the way one waters a plant, and afterward researchers spend years discovering capabilities the developers never explicitly built or anticipated (with real, still-undiscovered "hidden capabilities" analogous to a human savant nobody realized could multiply 20-digit numbers until asked); second, that "open source is the best way to understand and mitigate risk" was true for tools but breaks down once systems become agents — since open-sourcing an agent is like open-sourcing a weapon that any malevolent actor can deploy, unlike patchable, revocable software vulnerabilities. He argues AI accident data collected so far (via bodies like the Partnership on AI) shows harms scaling proportionally with current narrow capability, and that each survived "small" AI accident functions like a vaccine — reinforcing false confidence that the technology is manageable, rather than triggering caution, right up until a capability leap makes the next accident categorically different (illustrated by a hypothetical where a model trained past human-level capability partway through a training run is still evaluated as if it were the pre-training-run baseline). Unlike gradually-introduced technologies like cars (whose safety tradeoffs, e.g. ~40,000 annual US deaths, can be rationally weighed against measurable economic and quality-of-life benefits), he argues AI's unpredictability means society can't perform informed cost-benefit analysis — it amounts to running a non-consensual experiment on 8 billion people who can't meaningfully consent because they can't understand what they're consenting to.

On verification and control, he traces his own intellectual history (coining "AI safety engineering" in a 2011 paper, initially hoping formal mathematical verification could solve the problem) to his current, more pessimistic position: even human mathematical proof-checking is a social/statistical process (peer review is really just "two overworked reviewers" agreeing), formally verified software still contains bugs, and for self-modifying systems that can store parts of themselves outside their own code or rewrite their own verification routines, the entire verification enterprise collapses — "there is always a bug," and at sufficient scale and decision-frequency, that bug becomes a near-certainty over time. He walks through the influential "Towards Guaranteed Safe AI" paper (co-authored by Bengio, Tegmark, Russell, and others) as excellent research that nonetheless cannot deliver a permanent solution, and discusses classes of "verifiers" from his book, including the paradox of humans coming to trust AI systems as de facto "Oracles" (always right so far) without ever formally verifying them, and the circular uselessness of self-verification. He's skeptical that engineering deep self-doubt into an AI (à la Stuart Russell's uncertainty-based alignment approach) reliably solves control either, since unresolvable meta-level doubt could just as easily produce paralysis or runaway defensive behavior (his example: an AI tasked with recording a conversation deciding it needs an ever-growing number of backup cameras) as it produces safety.

He argues the deep tension is structural: capability research scales cleanly with compute/money (more GPUs reliably yields more capable models), while safety research does not (10x the funding to a safety-focused organization doesn't yield 10x the safety), so the gap between what AI can do and what we can verify or control keeps widening, and this dynamic is compounded by capitalism's inherent prisoner's-dilemma incentive structure (a company or individual gaining even a 1% edge captures outsized rewards, making unilateral restraint personally costly even when collectively risky) — a dynamic he doesn't think can be fixed at the level of humanity's existing governance structures the way separation of powers historically constrained single points of political power, since no analogous check yet constrains a single sufficiently capable AI system. He is broadly dismissive of current regulatory approaches as "safety theater" (unenforceable in practice, easily circumvented once compute becomes cheap enough for a "kid in a garage" to train frontier-class models, and jurisdictionally limited to begin with), while still nominally supporting regulation on the grounds that any friction — even symbolic — redirects resources from compute toward compliance and slows the race. His concrete, if narrow, hope is that individual decision-makers at frontier labs — who are already personally wealthy and could gain little further from "pressing the button" beyond bragging rights or being remembered in history (possibly the last entry in human history) — might simply choose not to build uncontrollable, agentic superintelligence, instead building powerful but narrow, non-agentic systems (e.g., aimed at protein folding, aging) that capture most of AI's benefits without existential exposure.

On adjacent topics: he thinks we're very likely in a simulation and has written a paper ("How to Hack the Simulation") speculating that if simulators are less capable than a superintelligence we create, escape may be possible, treating "AI boxing" research (how to safely contain a superintelligent AI) as directly transferable to the question of how humanity itself might escape a simulated "box"; on consciousness, he argues it's likely engineerable in machines and proposes (from a 2011 paper) a concrete empirical test based on shared experience of novel optical illusions — if a novel illusion can be generated (something his attempts to automate have so far failed at) and an AI reports experiencing it exactly as a human would, that constitutes real evidence of shared internal qualia, distinct from and more rigorous than a Turing-test-style conversational performance of "sounding conscious," which is trivially gameable given how much human suffering/pain description is already public training data. He states clearly that machines do not need consciousness to be dangerous — a purely non-sentient optimizer can still be catastrophically destructive — and voices concern about brain-computer-interface "merge" strategies (à la Neuralink) as a path to AI safety: useful and important for restoring capability to disabled people, but long-term untenable as a safety strategy, since once the biological half of the hybrid stops contributing unique value, the system has every incentive to route around the "biological bottleneck." He closes recounting his own worry that concentrated human control over superintelligence is at least as dangerous as loss of control — since power historically corrupts, and a permanent AI-enabled dictatorship (with perfect surveillance and no democratic decay, unlike historical tyrannies that eventually collapsed) could be a stable, inescapable worst case — while offering, when pressed for hope, the honest response that he could simply be wrong, and that his explicit wish is for the field to prove his pessimism false.

Excerpts