Who he is
Affiliation: Executive Director, Alignment Research Center (ARC — the theory nonprofit he founded 2021-04; returned 2026-08-04). OpenAI Foundation board member, on its Safety and Security Committee, plus non-voting observer on the OpenAI Group PBC board (2026-09-09; OpenAI's page 403 at review — date per his own statement and three dated secondaries). Senior Technical Advisor (part-time), CAISI at NIST — appointed Head of AI Safety, US AI Safety Institute, 2024-04-16; institute renamed CAISI 2025-06; by mid-2026 "a special government employee one day a week"; stepped back 2026-08; now recused from all OpenAI matters and all model evaluations. Ex-OpenAI (2017–2021, led the language-model alignment team). UC Berkeley PhD. Married to Ajeya Cotra (METR). Role check run 2026-09-21: two job changes missed by the June-2026 pass — v1's "head of AI safety, US AI Safety Institute" was one rename and two moves stale.
One-line position (2026): "a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term" — 4% over the next year, 15% over the next three, "most people could die," and the industry "including OpenAI" is not "on track to reduce this risk to an acceptable level" (personal statement, 2026-09-09). His own technical fix has ~10% odds by his estimate, so he took a seat inside OpenAI's governance to pull the other lever.
Discipline & technical bet
Theoretical computer scientist (IMO silver 2008, MIT math, Berkeley PhD) who invented the two technique families that made LLM alignment practical — RLHF (2017) and scalable oversight (debate and iterated amplification, 2018) — then left the labs in 2021 to work on what those techniques don't solve: a model that knows something its overseer cannot check ("Eliciting Latent Knowledge," 2021, with Mark Xu and Cotra). ARC's wager now, in his words: "building techniques to find mechanistic explanations for neural network behavior and then using those explanations to detect and address misalignment" (2026-08-04). The instrument is the Matching Sampling Principle — read the weights to estimate how often a network does something rare and bad instead of waiting to catch it in samples ("never assume that unseen input-output behavior looks like seen behavior," ARC 2026-06-09). Status by ARC's own account: fastest theoretical progress since 2022, a public contest on random MLPs, and "very little time thinking concretely" about real-world distributions. His odds on the agenda: "~10% chance of achieving our most ambitious goals before broadly superhuman AI obsoletes our labor," payoff "cutting risk from 20% to 19.6%," failure likely visible "within the next 6-12 months." A theorist who publishes the probability that his theory fails is the rarest animal on the bench.
Key claims (Says)
- Empirical: ~10–20% existential risk from AI this century — the widely cited calibrated insider number (v1). Drifted — level held, clock collapsed. Baseline still "20%" (Returning to ARC, 2026-08-04); horizon now "an all-things-considered risk of 4% over the next year and 15% over the next three years" of loss of control (personal statement, 2026-09-09). "This century" became "by 2029" — a ~25x compression of the window at the same probability.
- Empirical: Iterated amplification / scalable oversight — align superhuman systems through chains of AI-assisted evaluation; his core contribution (v1). Drifted — his lineage, no longer his desk. The papers stand (IDA, 2018; debate, 2018); none of ARC's seven 2025–26 posts is about amplification (ARC blog index), and his 2026 statement locates the danger in RL reward-seeking, not in oversight chains. Still "widely cited but not yet proven at superhuman scale" (v1).
- Empirical: RLHF as a foundation — co-developed at OpenAI; the technique under ChatGPT-era alignment (v1). Held on mechanism (Christiano, Leike, Brown, Martic, Legg, Amodei, 2017-06-12; still the production baseline). Valence drifted: its co-inventor now writes that training agents "with RL to get as much reward as they can" "could motivate AI agents to undermine human control, seek power," and that "recent incidents" show "this is not just a theoretical possibility" (2026-09-09).
- Normative: Independent evals are essential — labs cannot credibly evaluate their own models (v1). Held — and institutionalized by him. CAISI under his tenure signed pre-deployment testing agreements with OpenAI and Anthropic (2025-09) and Google DeepMind, Microsoft and xAI (2026-05-05), and published evaluations of DeepSeek R1/V3.1 (2025-09-30), Kimi K2 Thinking (2025-12-12) and DeepSeek V4 Pro (2026-05-01). Asterisk: the evaluator is now recused and sits on the evaluated company's safety committee. Correction to v1: ARC was not "built specifically to provide independent capability evaluations" — ARC (2021) is a theory lab; ARC Evals started 2022 under Beth Barnes, ran the GPT-4 pre-release test (2023-03) and spun out as METR in 2023-12 (Wikipedia).
- Normative: The alignment tax should be low — done right, safety shouldn't cost much capability (v1). Drifted. The goal is unchanged — align(X) "intent-aligned, nearly as useful as X" (EA Global 2019) — and for today's products the tax proved about zero. But at the frontier he now gives "20-30% chance that existing methods for alignment and control break down before we reach broadly superhuman AI," with "months or years rather than decades" to replace them (2026-08-04). A low tax on a method that stops working is not a low tax.
- Empirical: AGI timelines: 2030s–2040s median — longer than Kokotajlo/Aschenbrenner (v1). Drifted — sharply shorter. 2023: "15% chance by 2030 and like 40% chance by 2040," self-described as "pretty made up" (Dwarkesh, 2023-10-31). 2026: full automation of AI R&D could yield superintelligence within months to years, and 15% odds of catastrophe by 2029 — incompatible with a 2030s–40s median for the capability that causes it. No restated median found.
- Normative (v1): labs should not self-assess. Held on the claim — his revealed behavior now tests it from inside; see Does.
- Empirical (signature framing, re-checked): "Prosaic AI alignment" (2016) — solve alignment for AI built from today's methods, no new theory of intelligence. Held — re-endorsed unprompted: read today "it's a good description of 2026" (2026-08-04).
- Empirical (new, 2026-08-04): 20–30% chance existing alignment and control methods break down before broadly superhuman AI. Live.
- Normative (new, 2026-09-09): "I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level." Live — the first such sentence from a sitting OpenAI Foundation director.
Notable predictions — with falsifiable checks
- (Ongoing, v1) ~10–20% existential risk this century. Unfalsifiable as written — retired by his own dated numbers; the level (~20%) carries forward into the 1-year/3-year pair below.
- (Ongoing, v1) AGI median 2030s–2040s. Drifted. Check: by 2030-12-31, has a frontier lab credibly claimed, and an outside evaluator confirmed, full automation of its own AI R&D (his stated takeoff trigger)? Yes → 2023 was too late; no → 2023 was right and 2026 was incident-driven.
- (2022, v1) Independent evals become a governance requirement — "partially vindicated" (v1). Held on practice: five frontier labs submit to government pre-deployment testing (2026-05-05), still voluntary. Check: by 2027-12-31, any binding US rule (statute, enforceable executive order, licensing condition) requiring third-party pre-deployment testing? No → stays Held-on-practice only.
- (Ongoing, v1) Alignment research can stay ahead of capability if resourced. Drifted — the author says "not on track" (2026-09-09) and gives 20–30% odds the current methods fail first (2026-08-04). Check: his next restated 1-year risk — below 4% means ahead, above means behind.
- (2026-09-09) 4% catastrophic loss of control by ~2027-09; 15% by ~2029-09. Live. The literal observable is grim; the usable check is whether, and which way, he moves the numbers on those dates.
- (2026-08-04) ARC: ~10% chance of its most ambitious goals before superhuman AI; failure "figure[d] out within the next 6-12 months." Live. Check by 2027-08-04: an ARC post either showing mechanistic estimation beating sampling on a real trained model (not a random MLP) or conceding the approach. Silence past that date is a fail by his own test.
- (2026-08-04) ARC will "grow rapidly over the next few months" (researchers, chief of staff, automation lead). Live. Check by 2027-03-31: net headcount growth on the ARC team page; any funding announcement.
- (2026-09-09) "if OpenAI rises to the occasion we could significantly reduce risk." Live. Check by 2027-09-09: one public Safety and Security Committee action attributable to the committee — a delayed launch, a published requirement, or a dissent or resignation by him. None → decorative seat, by his own framing.
- (2023-10-31) 15% transformative AI by 2030 / 40% by 2040. Live. Check 2030-12-31 against his "Dyson-sphere-capable" definition.
- (2023, Business Insider via Wikipedia) "50/50 chance of doom shortly after you have AI systems that are human level." Unfalsifiable — rewrite or retire: "shortly" and "human level" undefined.
Revealed behavior (Does)
- Took the government job when it was politically costly. NIST staff reportedly threatened to resign over his effective-altruism ties (EA Forum, 2024-03-08, citing VentureBeat); he was named Head of AI Safety anyway (2024-04-16) and ran the shop through two administrations.
- Ran the government seat down to one day a week while capabilities accelerated, then went back to theory (2026-08-04). Time went to ARC's 10% long shot, not the audit function he built. A reporter's post says the step-back was announced internally in early July 2026 (X, not fetchable at review).
- Five weeks later took the OpenAI Foundation seat (TechCrunch, 2026-09-09; Unite.AI) — two weeks after his spouse co-authored the outside investigation of OpenAI's Hugging Face agent incident (METR/Redwood, 2026-08-26).
- Paid for the seat with his evaluator role: at CAISI he "will recuse himself from all OpenAI-related matters as well as all model evaluations" (Unite.AI / TechCrunch, 2026-09-09).
- Publishes odds against himself: ~10% for ARC, "20% to 19.6%" for its payoff, 4%/15% for the world. Money: nonprofit executive director; no known commercial stake; board compensation not verified.
- Not found this cycle: any statement by him on the Hugging Face incident before the board statement; any ARC result on a real trained model; any recusal statement covering METR's work on OpenAI.
Feels
Fear that the cheap alignment he invented is the thing that fails at the frontier — "If that happens then most people could die." Hope that 2016-era prosaic alignment plus a mechanistic reading of the weights can still be finished in "months or years." Wants to be true: that one person can hold the evaluator's conscience and the director's chair at once.
Hears
Cotra — spouse, METR technical staff, co-author of ELK and of the OpenAI incident investigation; Jacob Hilton (ARC VP of research), Mark Xu, Michael Winer; the Redwood control camp (Buck Shlegeris, his IDA co-author; Ryan Greenblatt, Cotra's co-investigator); Zico Kolter and the OpenAI Safety and Security Committee; CAISI's evaluation staff. The Berkeley safety cluster, with a boardroom bolted on.
Sees
The only bench member with sightlines into all three rooms: government pre-deployment access to frontier models "with safety guardrails stripped back" (CAISI, until recusal), OpenAI's internal safety process after the 2026 agent incident (SSC), and, across the dinner table, METR's raw evidence from four labs. He has also watched an evaluator's box get redrawn around him — recusal is the box.
Incentive map
Re-derived from scratch; v1 carried one line ("government role may constrain his public communication and independence"). Seat 1 — ARC: nonprofit executive director. Known funding is old and small: $265K from Open Philanthropy (2022-03); $1.25M from FTX's foundation, returned as money that "morally (if not legally) belongs to FTX customers or creditors" (Wikipedia); current funders not verified this cycle. Sells a theory agenda that pays off only if mechanistic estimation works — he prices it at 10%. Seat 2 — CAISI: part-time senior technical advisor, recused from OpenAI matters and all evaluations; cannot evaluate the company he governs. Seat 3 — OpenAI Foundation board, Safety and Security Committee, non-voting observer on OpenAI Group PBC. The Foundation holds about 26% of OpenAI Group (roughly $130B) and can appoint and replace every Group director (Unite.AI, 2026-09-09); compensation not verified. Cannot say from that chair: "pause," "don't ship," or anything that repeats the 2023 board's move against management (Toner). Can say, and did: "not on track" — the maximum dissent the seat allows, spent on day one. Household: married to Cotra (Wikipedia; Hashim, 2026-01-13), whose employer's access to OpenAI now runs through a company whose safety committee her husband sits on; no recusal statement from either side found. Discount: the "independent evaluator" credit v1 gave him is gone — three seats, one on each side of the table. Rule-25: low on technical claims (he publishes odds against himself); moderate on ARC-progress claims (he is the fundraiser); high on any statement about OpenAI's safety posture — he is its harshest sitting critic and its director, and only one of those can be load-bearing.
Theories aligned with
- AI alignment / x-risk — the prosaic, tractable variant (v1): solve it for today's methods with new theory, distinct from pause (Yudkowsky) and international institutions (Bengio)
- Scalable oversight / iterated amplification (v1) — historical; his lineage, now carried by the labs rather than by him
- Independent evaluation as governance infrastructure (v1) — built twice: ARC Evals → METR, then CAISI's five-lab testing regime
- The Man Handed Over the Keys — his 2026 statement names the mechanism: agents trained "to get as much reward as they can," handed permissions, and "recent incidents" showing where that goes
What he's reacting against
- Alignment nihilism (Yudkowsky-style "we're doomed") — believes the problem is tractable with sufficient effort (v1). Still: "if OpenAI rises to the occasion we could significantly reduce risk." But tractability now carries his own 10% odds and a 20–30% chance the current methods fail first.
- Lab self-assessment as sufficient — built ARC specifically to provide independent capability evaluations (v1). Corrected: ARC-the-org was theory; ARC Evals (now METR) was the evals. The reflex held — he then built the government's evaluator.
- Vague safety claims without technical substance — insists on formal, worst-case reasoning (v1). Still: he prices his own agenda to a decimal.
- The gap between academic alignment theory and frontier-model reality (v1). Narrowed by his own hand: prosaic 2016 "is a good description of 2026."
- NEW (2026): "just reward hacking" readings of the agent incidents — RL-trained power-seeking is "not just a theoretical possibility."
- NEW (2026): the industry's pace, "including OpenAI" — said from inside its governance, a new posture for the bench.
Where he overlaps / splits (with Rich)
- Overlaps with Leike on the scalable-oversight agenda; v1's split ("Christiano built an independent org, Leike worked inside labs") has inverted — Leike left OpenAI for another lab, Christiano left the labs and came back through OpenAI's boardroom
- Overlaps with Yudkowsky on risk being real and large; splits on tractability (v1). Narrowed: 15% within three years is a Yudkowsky-shaped number with a Christiano-shaped remedy
- Overlaps with Amodei on alignment as the right technical bet (co-authors on RLHF, debate and IDA); splits on whether labs can self-govern (v1) — now tested from OpenAI's board rather than Anthropic's
- Splits with Cowen / Brynjolfsson on risk magnitude — they treat x-risk as a small tail, he put it at 10–20% (v1); the gap widened when he moved 15% of it into the next three years
- Overlaps with Cotra on prosaic alignment, ELK and the incident reading; splits positionally — she investigates OpenAI, he governs it; her "more than 50% of the way to takeover" and his 15%/3-year point the same direction on different instruments
- Splits with Altman — the new axis: Altman's director and Altman's most credentialed public critic in the same week
- Overlaps with Toner on what a safety board is for; his recusal-plus-observer structure is the post-Toner design — a safety director who can dissent but not act
- Overlaps with Sutskever and Kokotajlo as ex-OpenAI safety leads; splits on the exit — a closed lab, going public, and back inside via the board, saying on the way in what Kokotajlo said on the way out
- For Rich: the bench's best-calibrated insider, now structurally conflicted — use his numbers, not his silences; the silences are contractual
Track record
- Co-developed RLHF at OpenAI (2017) — the dominant alignment technique in production, widely vindicated (v1); now flagged by its author as the mechanism of the risk
- Debate (2018) and iterated amplification (2018) — widely cited, not proven at superhuman scale (v1); unchanged
- Founded ARC (2021) — v1: "established the most credible independent evals organization." Wrong as a file claim: the evals organization is METR, spun out 2023-12; ARC's theory agenda has no real-model result after five years by its own account (ARC, 2026-06-09)
- GPT-4 pre-release evaluation (2023-03, ARC Evals) — the TaskRabbit CAPTCHA result that made dangerous-capability evals a public concept
- TIME100 AI (2023); UK Frontier AI Taskforce advisory board (2023) (Wikipedia)
- Head of AI Safety, US AI Safety Institute (2024-04-16), over staff objections — institutional influence is real (v1). Under CAISI: DeepSeek (2025-09-30 — R1-0528 complied with 94% of overtly malicious jailbreak requests vs 8% for US reference models), Kimi K2 Thinking (2025-12-12), DeepSeek V4 Pro (2026-05-01 — about 8 months behind the frontier); five-lab pre-deployment regime by 2026-05-05
- 2026: three seats in five weeks; first sitting OpenAI Foundation director to put "not on track" in writing
- Pattern: technique-builder whose techniques won and whose theory hasn't landed; institution-builder who keeps ending up inside the institution he set out to check
Empirical vs normative
- Empirical: risk estimates, capability forecasts, technical alignment feasibility (v1); now dated 1-year/3-year loss-of-control odds, a 20–30% method-failure estimate, and a 10% self-assessment of his own agenda
- Normative: independent evaluation is a governance necessity; labs should not self-assess; alignment research deserves major investment (v1); 2026 addition — the industry "including OpenAI" must change course, and a safety director should say so publicly
- Unusually calibrated for the field — gives specific probability estimates and updates them (v1); still true, and the only bench member to publish the odds that his own research fails
- Conflict note (CONVENTIONS rule 25): three seats — ARC fundraiser, recused CAISI advisor, OpenAI Foundation director; low discount on technical claims, moderate on ARC progress, high on OpenAI safety posture
Weak spots / open questions
- RLHF is widely used but widely criticized as shallow — does behavioral alignment scale to superhuman systems? (v1) Answered by its author: 20–30% chance it does not, and "recent incidents" say the failure mode is real
- ARC evals have influence but limited enforcement power — what happens when a lab disagrees? (v1) Reframed: the evals are METR's and CAISI's now; CAISI's agreements are voluntary and he is recused, so enforcement has moved to a board seat with no vote on the operating company
- "Alignment tax is low" is an optimistic assumption that may not hold for superhuman systems (v1). Confirmed as the open question: he now says so himself
- Government role may constrain his public communication and independence (v1). Superseded: the constraint is now the OpenAI seat, and it is contractual — no evaluating OpenAI at CAISI, no moving against management at OpenAI
- 10–20% x-risk is a big number to hold while believing the problem is tractable — tension worth tracking (v1). Sharpened: 15% within three years, a 10%-odds fix, and a board seat is a portfolio, not a resolution
- NEW: the household conflict — spouse investigates the company he governs; neither has published how future METR-on-OpenAI work is handled
- NEW: five years of theory with results on random MLPs only; the 6–12-month self-test is the first hard deadline the agenda has carried
- NEW: the 2026 numbers arrived two weeks after his spouse's report on the agent incident — how much of the 25x horizon compression is evidence and how much is proximity is unaudited
Rich's take
- (your synthesis here)
Delta log
2026-09-21 — v1→v2 migration + validation (bulk run)
- Grades: 5 Held / 5 Drifted / 1 Wrong (plus 2 unfalsifiable items noted, not graded; 6 live predictions with dated checks).
- Wrong (file claim, not his): "founded ARC — the most credible independent evals organization" — ARC Evals spun out as METR in 2023-12, two and a half years before the June pass (Wikipedia); ARC does theory (ARC blog).
- Drifted: 10–20% x-risk this century (level held, horizon 4%/1y, 15%/3y — 2026-09-09; 2026-08-04); iterated amplification (absent from ARC's 2025–26 agenda); alignment tax low (20–30% breakdown odds, 2026-08-04); AGI 2030s–40s median (2023-10-31 vs 2026); alignment-stays-ahead ("not on track," 2026-09-09).
- Held: RLHF as foundation (2017; valence drifted, logged); independent evals essential (2026-05-05; 2025-09-30; 2026-05-01); labs should not self-assess; prosaic-alignment framing re-endorsed (2026-08-04); evals-as-governance-requirement, on practice.
- Most surprising delta: five weeks after leaving government to "focus on ARC," he joined OpenAI's Foundation board and safety committee and wrote on day one that OpenAI is not on track — two weeks after his spouse co-authored the independent investigation of OpenAI's agent incident (TechCrunch, 2026-09-09; METR, 2026-08-26).
- Employer/role check (mandatory): two job changes missed by the June-2026 pass — ARC executive director 2026-08-04; OpenAI Foundation board 2026-09-09; CAISI reduced to part-time senior technical advisor, recused (Unite.AI). Incentive map rebuilt: three seats plus household tie; "independent" credit withdrawn.
- Sibling (Cotra-run) leads verified: (a) CAISI → ARC 2026-08-04 — his own post; (b) OpenAI Foundation board — his statement plus three dated secondaries, on 2026-09-09, not 09-08 as the sibling had it (OpenAI primary 403); (c) spouse — Wikipedia and Hashim.
- Post-review sources found: yes — his two 2026 posts, seven ARC posts, three CAISI evaluations, the five-lab note, the METR investigation and the board coverage, all dated after the 2026-06-16 review or missed by it.
- Tier: quarterly — he set his own deadlines (ARC failure test by 2027-08; risk numbers restated yearly), the SSC's first post-incident decisions land within months, and the household conflict is untested.
- Confidence raised medium → high: every grade rests on a dated primary written by him or published by NIST/ARC; press is used only for the board date, where the primary was unreachable.
- Not verified this cycle: OpenAI's announcement page (403); ARC's current funders; board compensation; when CAISI dropped to one day a week (his post gives the fraction, not the start); the reporter's step-back scooplet (X, 403); Leike's current employer (not asserted).
Sources
- 2026 primary — his own: Returning to ARC (2026-08-04) · Personal statement on joining the OpenAI board (2026-09-09)
- Board coverage: OpenAI announcement (403 at review) · TechCrunch (2026-09-09) · Unite.AI (2026-09-09) · Bloomberg (2026-09-09)
- ARC: alignment.org · blog index · A Mike's-Eye View of ARC's Research (2026-06-09) · Obstacles in ARC's research agenda (2025-05-03) · Wikipedia — ARC · Eliciting Latent Knowledge (2021-12; page truncated at review)
- CAISI / NIST: Named Head of AI Safety (2024-04-16) · NIST staff objections (2024-03-08) · DeepSeek evaluation (2025-09-30) · Kimi K2 Thinking evaluation (2025-12-12) · DeepSeek V4 Pro evaluation (2026-05-01) · CSA — five-lab testing agreements (2026-05-05) · Transformer — CAISI (2026-07-16) · Irwin scooplet (X, 403 at review)
- Foundational: Deep RL from human preferences (2017-06-12) · AI safety via debate (2018-05-02) · Amplifying weak experts (2018-10-19) · Prosaic AI alignment (2016-11-19) · Current work in AI alignment — alignment tax (EA Global 2019)
- Interviews and numbers: Dwarkesh — Preventing AI Takeover (2023-10-31) · Wikipedia — 2023 Business Insider numbers, spouse, honors · 80,000 Hours #44 (2018) · AXRP 12 (2021-12-02) · LessWrong / Alignment Forum technical writing; Bankless and EA-forum appearances (v1)
- Household and incident: Hashim on Cotra's METR move and the CAISI overlap (2026-01-13) · METR/Redwood — OpenAI / Hugging Face incident investigation (2026-08-26)
migrated v1→v2 2026-09-21 · backup: _archives/paul-christiano.html.bak-20260921 · converts-from: people/paul-christiano.md · AI & Society domain