Song, Dawn

professor of computer science at UC Berkeley; researcher spanning computer security, adversarial machine learning, program synthesis, and privacy; founder of Oasis Labs

Dawn Song: Adversarial Machine Learning and Computer Security

Lex Fridman Podcast · May 12, 2020

• Dawn Song
• adversarial machine learning
• computer security
• adversarial examples
• differential privacy
• data ownership
• program synthesis
• physical adversarial attacks
• autonomous vehicle security
• blockchain

Core ideas

Song frames security as a field where you can never prove a system has "no vulnerabilities" (unlike most areas of computer science where you prove a positive statement true), only that specific, well-defined properties hold — citing formally verified microkernels, compilers, file systems, and crypto libraries as real achievements, while cautioning that formal verification of one property (e.g., memory safety) leaves systems open to entirely different attack classes. She identifies a long-term trend of attacks moving "up the stack" from systems and networks toward humans themselves — the "weakest link" — citing social engineering, phishing (even against sophisticated targets like Google, which was successfully tricked into wiring money), and disinformation/fake news as attacks that exploit human psychology rather than software flaws, and describes her own research using NLP-based chatbot proxies that observe conversations for red flags (like a "relative" claiming to be stranded abroad asking for money) and issue automated "challenge-response" tests to verify claimed identities.

The core of the conversation covers adversarial machine learning across two attack surfaces: inference-time attacks, where small, often human-imperceptible perturbations to inputs (the canonical adversarial example) cause misclassification, including targeted attacks that force a specific attacker-chosen wrong answer; and training-time "backdoor" attacks, where poisoning a small number of training examples causes a model to behave normally except on attacker-chosen trigger inputs (demonstrated in facial recognition, where wearing specific glasses — even a pattern imperceptible to humans overlaid on training images — can cause anyone wearing them to be misclassified as a specific target identity, such as a national leader). Song's team extended adversarial examples into the physical world with adversarial stop signs (now exhibited at London's Science Museum) that reliably fool image classifiers used in autonomous driving contexts across varying viewing distances and angles — a much harder engineering problem than digital attacks, since perturbations must survive printing, physical placement constraints (confined to the sign itself), and camera capture, yet the process of designing them is, she says, "mostly science" rather than trial and error, using a well-defined optimization objective incorporating these physical constraints.

She argues these vulnerabilities reveal that deep learning systems, despite strong benchmark performance, are not learning sufficiently rich, robust representations — analogous to how human vision integrates far more contextual, structural information than a simple label — and presents her own defense research on "spatial consistency" for semantic segmentation (where a model's segmentation of overlapping randomly-chosen image patches should agree in their intersection, a natural constraint that's easy for legitimate models to satisfy but very hard for an attacker to spoof consistently across arbitrary, unknown patch selections) and its extension to "temporal consistency" for audio/video, as a promising general defense principle contrasted with narrower ad hoc "patches" like adversarial training. She reports that real-world systems are demonstrably vulnerable too — her group showed that Google Translate could be attacked via a model-stealing approach (training an imitation model purely from API queries, then crafting adversarial examples against the imitation model that transferred successfully to the real Google Translate, in one case flipping a translated temperature from "6 Fahrenheit" to "21 Celsius" via a single-character change) — pushing back directly on claims (including one attributed to Elon Musk) that adversarial attacks are mainly lab curiosities; she distinguishes the separate question of technical feasibility (which she says is "100 percent" established) from whether malicious actors will actually deploy such attacks in practice, and recommends multi-sensor fusion (combining vision, radar, lidar, ultrasonic) as a key mitigation, since attacking multiple independent modalities consistently is far harder than fooling one.

On privacy, Song distinguishes security's "integrity" property (is the system's output correct) from "confidentiality" (can attackers extract sensitive information the system was trained on), noting neural networks' high capacity means they can memorize and leak training data — demonstrated in her collaboration with Google researchers extracting real Social Security and credit card numbers from a language model trained on the Enron email dataset, using only query access without knowledge of model internals. She presents differentially private training (adding calibrated noise during the training/gradient-update process) as an effective, empirically validated defense against this specific extraction attack while preserving model utility. Song then develops an extended argument about data ownership: drawing an analogy to how establishing and enforcing physical property rights historically drove economic growth (and their absence is argued to hinder growth in some developing economies), she argues personal data — now arguably more definitional of a person than physical traits — currently lacks clear ownership, with internet companies effectively controlling it by default; she argues establishing user ownership as a starting point would let individuals choose how their data is used (continuing to trade it for free services they value, or opting to pay for privacy-preserving alternatives) rather than eliminating today's ad-supported ecosystem, and calls for more nuanced, technically-grounded dialogue (rather than adversarial "big tech vs. users" framing) balancing utility and privacy — work she pursues both academically and through her startup, Oasis Labs, which combines zero-knowledge proofs, secure computation, and blockchain-based immutable logs of data ownership/usage policies into what she calls a "responsible data economy" platform.

On blockchain/distributed ledgers, Song explains the core security properties at stake — integrity (consensus mechanisms like proof-of-work preventing malicious rewriting of transaction history, with security theoretically quantifiable in terms of what fraction of nodes/resources an attacker must control) and confidentiality (public ledgers are transaction-transparent by default, so privacy requires additional cryptographic techniques like zero-knowledge proofs and confidential smart contracts, which Oasis Labs builds on). On program synthesis — teaching computers to write code, which she calls "the perfect playground" for building generally intelligent machines and her primary post-security research motivation — she reports genuine but early progress (e.g., translating natural language directly into SQL queries, and neural program synthesis that has learned recursive constructs with provably perfect generalization on certain tasks), and frames three key open challenges: increasing the complexity of synthesizable programs, improving generalization to unseen inputs, and — most importantly, in her view — enabling systems to transfer knowledge across tasks so that, like a trained human, a system need not be retrained from scratch for every new problem.

Excerpts