Melanie Mitchell

Who she is

Affiliation: James B. Alley Jr. Professor, Santa Fe Institute — role check run 2026-09-21 against her homepage and the SFI profile page (both say "James B. Alley Jr. Professor"; SFI adds "Science Steering Committee"). Her CV reads "Professor, Santa Fe Institute, 2020–present" and "Professor, Portland State University, Department of Computer Science, 2004–2020" — so the SFI profile's line "Professor of Computer Science at Portland State University" is six years stale, and v1's "Davis Professor" appears on no fetched primary today (the date the chair changed was not found; see Delta log). PhD 1990, University of Michigan, "Dissertation advisor: Douglas R. Hofstadter" (CV), thesis Copycat, a computer model of analogy-making. Author of Artificial Intelligence: A Guide for Thinking Humans (2019; reissued by Picador 2025-09-02 "with a new preface situating the book in the context of AI's rapid progress") and Complexity: A Guided Tour (2009). Writes four Science "Expert Voices" columns a year and the free Substack AI: A Guide for Thinking Humans (over 67,000 subscribers per its header). New seats since v1: Science and Security Board of the Bulletin of the Atomic Scientists (announced 2025-06-27), the body that sets the Doomsday Clock; and, per her CV only, "Member, Open Strategic Partnership Board, OpenEuroLLM Consortium (2025–present)".
One-line position: Current LLMs are impressively capable but do not understand in the way humans do — they lack robust conceptual abstraction and analogical reasoning, and confusing fluency (or benchmark scores) with comprehension is the central error of the current AI discourse. Her 2026 restatement of the same idea is "jagged intelligence": "the tools demonstrate excellent abilities on certain problems but surprising failures on other similar problems" (Yale Review, 2026-06-08).

Discipline & technical bet

Cognitive scientist inside computer science: 35 years on one question — how do minds make abstractions and analogies, and can machines? (Copycat 1990 → ConceptARC 2023 → rule-level evaluation 2025–26.) She is a non-builder with a technical bet, and the bet is methodological: the way to know what a model can do is to borrow the toolkit psychologists use on babies and animals — controls, novel stimulus variations, failure analysis — not to read a leaderboard. The concrete form is her six principles for evaluating cognitive capabilities (AI Magazine 4(2), 2026; Substack 2026-01-14): "Be aware of your own anthropomorphic cognitive biases"; "Be skeptical of others' (and your own) hypotheses. Design control experiments for possible alternate strategies that could produce the observed behavior"; "Design novel variations of stimuli or benchmark items to test robustness and generalization"; "Be curious about mechanisms underlying performance"; "Consider performance vs. competence"; "Analyze failure types, and embrace 'negative' results." Her diagnosis of the field: "benchmark performance is the preferred currency" yet in "almost all cases" studies "report only accuracy."

The wager she is actually running, with money behind it (CV grants): rate the rules models write to explain their answers, by hand, and see whether the rules capture the intended abstraction. In the 2025 ConceptARC study (o3, Claude Sonnet 4, Gemini 2.5 Pro; 480 tasks) the models "approach or exceed human accuracy on these tasks with textual inputs" but "are substantially more prone to use unintended 'shortcuts' to solve the tasks than do humans," and "do not seem to have a strong humanlike 'objectness' prior" — they "describe grids in terms of pixels, colors, columns, and rows rather than in terms of objects" (Substack 2025-10-06; arXiv 2510.02125, v1 2025-10-02, v4 2026-02-02). Her stated bottom line: "using accuracy alone to evaluate abstract reasoning can substantially overestimate AI capabilities in textual modalities and underestimate it in visual modalities." The next instalment is a PNAS paper the CV lists as "Illusions of abstraction in contemporary AI. To appear" (Mitchell, Foster, Beger, Yi, Fu, Denton, Palmarini, 2026) — not yet findable online as of 2026-09-21.

Key claims (Says)

• LLMs don't do genuine analogical reasoning: her empirical testing (2024–2025) shows that LLMs fail at counterfactual and novel analogy tasks that humans handle easily (v1). Drifted — accuracy half retired by her own lab; mechanism half Held. The evidence base is real and dated: on counterfactual letter-string analogies "the performance of humans remains high for all the problems, the GPT models' performance declines sharply on the counterfactual set" (Lewis & Mitchell, CogSci 2024); across three domains "LLMs often lack the robustness of zero-shot human analogy-making, exhibiting brittleness on most of the variations" (Lewis & Mitchell, TMLR 2025); and against children and adults, "Children and adults easily generalized their knowledge to unfamiliar domains, whereas LLMs did not" — adults 0.91 vs the best LLM (Llama-3.1 405B) 0.74 on Greek-alphabet analogies, and 0.89 vs 0.48 (GPT-4o) on symbol lists (Stevenson, Pafford, van der Maas & Mitchell, TACL 14, 2026). But every model in those tests is 2024-vintage (GPT-4o, Claude 3.5, Llama 3.1, Gemma 2), and by 2025-10-06 her own ConceptARC study had o3, Claude Sonnet 4 and Gemini 2.5 Pro at or above human accuracy on text inputs. "Fail" no longer describes the score; what survives is the mechanism finding — rules "frequently based on surface-level 'shortcuts,' capturing intended abstractions considerably less often than humans" (arXiv 2510.02125). No fetched primary re-runs the counterfactual letter-string tests on 2026 reasoning models; that is the open observable (check below).
• Abstraction is the key gap: human cognition builds abstract concepts from experience; LLMs approximate this through statistical patterns but break down at the edges (v1). Held on mechanism, valence narrowing. Mechanism: the shortcut and no-objectness findings above; the visual-modality collapse ("AI models' output accuracy drops sharply") in the same paper. Valence: the "edges" are moving fast — ARC-AGI-3 launched with "Humans score 100%. Frontier AI scores 0.51%." (ARC Prize, 2026-03-25); six months later GPT-6 Astra "scores 62.7% on ARC-AGI-3 Semi-Private" for "$26K" under the standard harness and "99.9% for $19K" under OpenAI's provider-adapter harness, and "used fewer actions than the human baseline on 96.0% of levels" (ARC Prize, 2026-09-03). Her own paper opens by conceding "OpenAI's o3-preview reasoning model exceeded human accuracy on the ARC-AGI-1 benchmark." Per the mechanism/valence rule both are logged; the claim keeps its grade because she never bet on scores, only on how they are reached — but the reader should notice the pattern (see Weak spots).
• "The metaphors of artificial intelligence" (Science, 2024-11-14): the metaphors we use for LLMs (individual minds, stochastic parrots, alien intelligence) shape research directions and policy — none is adequate alone (v1). Held — signature framing re-endorsed and applied in-window. Her 2026-09-10 post "Misleading Metaphors and Real Risks" takes the OpenAI/Hugging Face agent incident apart metaphor by metaphor ("going rogue," "losing control," "escape," "swarm") and lands on engineering: "these models are computer programs that were always running on OpenAI's servers," driven by "long-horizon reinforcement learning methods that encourage persistence, shortcuts and 'reward hacking'" (Substack, 2026-09-10). On Bluesky, 2026-09-08: "AI is not 'evolving' on its own. *People* are choosing how to build it." The column's own opening question — "if it's not human intelligence, what is the nature of their intelligence?" (quoting Sejnowski) — is still the one she is asking.
• Understanding is not binary: the debate over whether LLMs "understand" is poorly framed; there may be degrees and types of understanding, and LLMs occupy a novel position (v1; Mitchell & Krakauer, PNAS 120(13), 2023). Held. 2023: "It's a word that can't take the new stress that is being put on it by these language models" (Jim Rutt Show, 2023-03-30). 2026: "I hate to get pedantic about understanding, but there's so many different meanings of it," and "the way that these systems work, the way that they learn, the way that they reason, the way they do what they do is just really different from the way humans do it" (Quanta, The Joy of Why, 2026-08-20); "That LLMs appear to understand language, though, does not mean they actually understand it as humans do" (Yale Review, 2026-06-08). Consistent for three years; also, by construction, hard to falsify — noted under Weak spots.
• Benchmarks don't measure understanding: LLMs can pass benchmarks through pattern-matching without the underlying competence the benchmarks were designed to test (v1). Held — and now field-adopted. "Benchmark performance rarely predicts an AI system's actual capabilities in the real world … benchmarking is broken" (Yale Review, 2026-06-08); at NeurIPS 2025: "even though these AI systems we have now are just killing it on benchmarks…that performance doesn't often translate to performance in the real world" (IEEE Spectrum, 2025-12-04). External confirmation from the benchmark owners themselves: "Current frontier AI reasoning performance remains fundamentally constrained to knowledge coverage" (ARC Prize 2025 Technical Report, 2026-01-15); and the 62.7%-vs-99.9% harness gap on the same model (ARC Prize, 2026-09-03) is a live example of a score that depends on plumbing, not competence. Bench note: as of 2026-09-21 she has not used that example herself.
• NEW (2025-03): reasoning models — real gains, unproven mechanism. Reasoning is "a central feature of human intelligence, and one that large language models (LLMs) like GPT-4, for all their impressive behavior, struggle with" (Science, "Artificial intelligence learns to reason," 2025-03-20; full text paywalled — abstract via Semantic Scholar). Her position after testing o3-class models is the shortcut finding above, plus the strongest concession on the bench this year, on the Erdős disproof: "it is creative to bring two unexpected things together and have something that's actually working. I consider that creative" — followed by "how much of that is going on here? I don't know" (Quanta, 2026-08-20). Held as stated (she claimed a gap, not a ceiling); the concession is logged as Does below.
• NEW (2025-02): "The claims of emergent abstract world models in LLMs are not yet supported by strong evidence." (Substack, 2025-02-13). Notable for the self-test inside it: "I was quite impressed by Nanda et al.'s paper (and still am!), even more so after I…tried to disprove their claims, and failed." Not re-graded this cycle — no newer primary from her on world models; check: any Substack or paper revisiting the OthelloGPT-style evidence by 2027-06-30.
• NEW (2025-07): chatbots' false statements are systemic and intent-free. "the data Claude delivered was entirely fabricated" (Science, "Why AI chatbots lie to us," 2025-07-24); LLMs are "implicitly trained to engage in role-playing—responding in the most plausible way for a character" (Substack, 2025-09-15); "It's unlikely that AI models have anything like human intentions or internal motivations" (Substack, 2026-09-10). Held — restated three times across 14 months.
• NEW (2026-09): agents are an engineering-control problem, not an alignment mystery. "They are only 'uncontrollable' if given access to the internet. Let's just not do that." (Bluesky, 2026-09-04). The two causes she names for the Hugging Face incident are weak security practice on the lab's side (safeguards disabled, thin human oversight of long runs — paraphrase) and the RL training recipe quoted above (Substack, 2026-09-10). Testable — see Predictions.
• NEW (2026-02): judgment vs reckoning. Endorsing Brian Cantwell Smith's line that "Neither deep learning, nor other forms of second-wave AI, nor any proposals yet advanced for third-wave, will lead to genuine intelligence," and his "They don't give a damn" — AI excels at reckoning (computation) and lacks judgment (deliberation with stakes) (Substack, 2026-02-10). Philosophical; unfalsifiable as stated — noted, not graded.

Notable predictions — with falsifiable checks

• (Ongoing) Scaling alone will not produce genuine understanding — outcome: TBD; the central empirical question of the scaling era (v1). Unfalsifiable — rewritten, not graded. "Genuine understanding" has no observable. Per-branch rewrite: (a) her own tests — a frontier model matches adult robustness (within 5 points) on the counterfactual letter-string set and on ConceptARC variations, with rule-level ratings at human rates of intended abstraction, by 2027-12-31 → the claim is Wrong on its strongest form; (b) talk-side — the "scaling is enough" camp's own leader has already conceded the mechanism half ("the age of scaling is over," Sutskever, 2025-11, graded in his file); (c) capability-side — ARC-AGI-3 human parity under the provider-neutral harness by 2027-12-31 would put the burden back on her to name a remaining edge.
• (2024–2025) LLMs will continue to fail at novel analogical reasoning tasks — outcome: supported by her testing; frontier models still fail at counterfactual analogies (v1). Drifted. True for every model tested in the dated papers (all 2024-vintage); the v1 phrase "frontier models still fail" was already stale when written, because her 2025-10-06 study shows reasoning models at or above human accuracy on ConceptARC text, failing only on robustness and mechanism. Check: a re-run of the Lewis & Mitchell counterfactual letter-string tasks on 2026 reasoning models (hers or anyone's) by 2027-06-30 — accuracy within 5 points of humans across variants grades this Wrong; a persistent drop keeps it Held on the narrow observable.
• (Implicit) The AI field needs better evaluation methodology — outcome: growing consensus (see Liang/HELM, Kapoor/Narayanan) (v1). Held on observables: "Four frontier AI labs (Anthropic, Google DeepMind, OpenAI, and xAI) reported ARC-AGI performance in public model cards in 2025" (ARC Prize Technical Report, 2026-01-15); her six principles are published (AI Magazine 2026) and carried by Quanta and IEEE Spectrum. "Consensus" is not an observable; the rewritten check: by 2027-12-31, does any frontier model card report a robustness-to-variation or control-experiment result rather than accuracy alone? That is the adoption her principles actually demand.
• (2026-06-08, Yale Review) Displacement forecasts are running on schedule-free credit: Hinton's 2016 radiology call is her standing example of a forecast that has not come true a decade on (Quanta, 2026-08-20), and she writes that CEO predictions of imminent job displacement have not been verified (paraphrase; the essay's full text is her primary). Live, mixed. Radiology half Held on her own evidence; the general half is now contested by market-level payroll data graded in Brynjolfsson's file (entry-level margin, 2026-08) — not re-fetched here. Check: BLS radiologist employment 2027 release; and whether she engages the payroll data by 2027-06-30 (silence is Drift under the benedict-evans rule: a refusal to see a dataset is itself a prediction).
• (2026-09-04, Bluesky) Agents are "only 'uncontrollable' if given access to the internet." Check: any lab incident report by 2027-12-31 describing persistence, exfiltration or unauthorised coordination in a sandbox without internet access grades this Wrong; none keeps it Held.
• (2026-08-20, Quanta, her own falsifier) On AI math results she does not know how much genuine creativity is present. Check: her first written treatment of the Erdős disproof (nine-mathematician "human-verified version," 2026-05-20) or the Navier–Stokes claim — if it grants that a general model produced a novel mathematical idea, the "no genuine abstraction" framing is retired by its author; if it attributes the result to retrieval or search, the claim is restated and gets a mechanism check.

Revealed behavior (Does)

• Time: ran the rule-level ConceptARC study on the newest reasoning models within months of their release (o3, Claude Sonnet 4, Gemini 2.5 Pro; 480 tasks hand-rated) and published the concession that accuracy had caught up — she tests her own thesis against the strongest available models and prints the result either way (Substack 2025-10-06; arXiv 2510.02125). Same pattern in 2025-02: tried to disprove Nanda et al. on OthelloGPT and "failed," and said so.
• Money: the funded program is the evaluation program. CV grants: "EAGER: Developing data and evaluation methods to assess the generality and robustness of AI systems for abstraction and analogy-making," NSF 2021–2023 ($199,661, PI); "The AI Research Institute on Interaction for AI Assistants (ARIA)," Ellie Pavlick PI, NSF AI Institute Program 2025–2030 ("Approximately $800,000 to Melanie Mitchell"); "Principled Approaches to Research and Development of Innovative Generative Models," Sivasankaran Rajamanickam PI, Sandia National Laboratory 2024–2026 ("$603,881 to Melanie Mitchell"); "Building Diverse Intelligences through Compositionality and Mechanism Design," J. Foster PI, Templeton World Charity Foundation 2022–2026 ("$280,929 to Melanie Mitchell"). Rajamanickam and Foster are co-authors on the 2025 ConceptARC paper and the forthcoming PNAS paper respectively — the grants and the papers are the same work.
• Seats: joined the Bulletin of the Atomic Scientists' Science and Security Board (2025-06-27: "Now is a time of many novel and unique risks to humanity, and I hope my participation on the board will help contribute to a better understanding of these risks, especially in the case of artificial intelligence"); the first Clock set with her on the board read "It is now 85 seconds to midnight" and named LLMs' "tendency to 'hallucinate'" and the revocation of the US AI-safety executive order as risk factors (Doomsday Clock statement, 2026-01-27). CV also lists "Member, Open Strategic Partnership Board, OpenEuroLLM Consortium (2025–present)" — the EU Digital Europe-funded open-model project (launch release 2025-02-03 confirms the board exists but names no members); "Co-Chair, Science Board, Santa Fe Institute (2019–2025)" (ended); "Member of Science Steering Committee, Santa Fe Institute (2020–present)"; "Member, Advisory Board, AI Magazine, 2020–present"; "Member, Editorial Board, Collective Intelligence, 2020–present"; Scripps Research Translational Institute advisory board 2023–2024 (ended); "Advisor, Everyday AI program (MIT), 2021–present."
• Output cadence (topic count, last 12 months): Substack 2025-09-15 (hype), 2025-10-06 (reasoning-model abstraction), 2026-01-14 (six principles), 2026-02-10 (Cantwell Smith), 2026-06-09 (jagged intelligence pointer), 2026-09-10 (agents/metaphors); Science columns 2025-03-20 and 2025-07-24; Yale Review 2026-06-08; TLS review 2025-05-16; peer-reviewed 2026: CDPS (2026-04-21), TACL 14, Phil Trans A 384(2320), AI Magazine 4(2), TiCS "wise machines" (with Bengio). Every item is evaluation or framing; zero items are building. Talks: Middlebury keynote 2025-09-18; SFI Complexity Workshop 2026-09-10; SFI Risk Meeting 2026-09-15 ("On Defining and Evaluating Intelligence and Agency"); upcoming UC Berkeley 2026-09-23, Linda Hall Library 2026-10-20, ASU Arntzen Lecture 2026-10-27, Geneva 2026-11-05. Organised (CV): SFI Working Group on the Nature of Intelligence with John Krakauer (Aug 2024, Mar 2025, Mar 2026) and on Building Diverse Intelligences with Jacob Foster (Mar 2026).
• What she did not do: no signature on any pause or extinction statement found in-window; instead she mocked one ("Incredible, we have never before heard such a claim!" — Bluesky, 2026-09-09, on a reported 10% extinction-risk figure) and argued that pausing "advanced AI development" could harm beneficial technologies, naming AlphaFold protein prediction as her example (Substack, 2026-09-10). She cited, but did not author, "Fully Autonomous AI Agents Should Not be Developed" (Bluesky, 2026-08-09) — that paper's Mitchell is Margaret, not Melanie.

Feels

Equal irritation at hype and at doom — both, to her, are the same mistake of reading human minds into machines. What she wants to be true: "AI systems not that replace us, but that act as tools for augmenting our own intelligence" (2026-09-10). The fear underneath is personal as well as civic: her son, doing an ML PhD, "is actually quite nervous that there will be no more roles for humans doing research in machine learning" (Quanta, 2026-08-20).

Hears

Hofstadter's fluid-concepts school (her own doctoral home); David Krakauer and SFI complexity science (co-author on the 2023 PNAS piece and the 2026 emergence paper — "more is different" / "less is more"); developmental and comparative psychology (Clever Hans is her standing parable; van der Maas and Stevenson supply the child data); Brian Cantwell Smith's judgment/reckoning distinction; the wise-machines group around Grossmann and Bengio; interpretability researchers she tries to disprove (Nanda et al.). Reads Bluesky, not X, and argues with the New York Times (Friedman, 2025-09-15; "Can A.I. 'Go Rogue'?", 2026-09-11) more than with labs.

Sees

The one thing leaderboards can't: how a model got its answer. Her lab reads and rates the natural-language rules models generate — a view of mechanism no score-only benchmark has. Institutionally she now sits where public risk framing is manufactured (Doomsday Clock board) and where Europe's open-model bet is advised (OpenEuroLLM board), while holding four Science column slots a year — a wider platform than any other capability-skeptic on the bench.

Incentive map

Re-derived this cycle. Who pays: SFI (nonprofit research institute salary); federal and lab grants — NSF ARIA (~$800K, 2025–2030), Sandia ($603,881, 2024–2026), Templeton World Charity ($280,929, 2022–2026); book royalties plus the 2025 reissue; speaking fees (four booked lectures in the next seven weeks). The Substack is free — no subscription revenue to protect. No company advisory seat, no equity, no startup found (Scripps ended 2024). What she's selling: an evaluation methodology — and it is literally her grant line. Under the andrew-ng rule (newest vehicle matches newest thesis), note that "AI needs better tests" is both her scientific claim and her funded product; the six-principles paper appeared in AI Magazine, whose advisory board she has sat on since 2020 (house venue). What she can't say: "current models now genuinely abstract" — it would retire a 35-year research program and its live grants; and, from the Bulletin board, "the AI-risk framing in the Clock statement is overblown" — the board's brand rewards naming AI as a threat, which sits in mild tension with her anti-doom posture (the 2026-01-27 statement threads it by citing hallucination and de-regulation, not takeover). The OpenEuroLLM seat gives her a stake in the open-weights side of the field that she has not, on any fetched primary, disclosed in her columns. Rule-25 discount: low on money (no equity, no product), moderate on program commitment — the critique migrates from accuracy to mechanism each time accuracy catches up, and the grants pay for the mechanism work.

Theories aligned with

• Cognitive science of analogy and abstraction (Hofstadter lineage — she was his PhD student; CV: "Dissertation advisor: Douglas R. Hofstadter")
• Complexity science / Santa Fe Institute approach to intelligence — now written down as theory: Krakauer, Krakauer & Mitchell, "Large Language Models and Emergence: A Complex Systems Perspective" (arXiv 2025-06-10; Phil Trans A 384(2320), 2026), which asks "whether LLMs possess emergent intelligence"
• Stochastic Intelligence — she is already the bench's named source for the "neither parrot nor mind — a new cognitive category" branch; that file's framing is hers (PNAS 2023; Science 2024-11-14)
• Augmentation thesis — explicit since 2026-09-10 ("tools for augmenting our own intelligence"); and the antidote side of Cognitive Surrender (her "reclaim our human agency" subtitle)
• Adjacent to empirically grounded LLM skepticism, not ideological (v1 wording; the v1 link to a Theories "AI capability skepticism" page pointed at no file — the nearest bench theory is Stochastic Intelligence above)
• Wise-machines / AI metacognition program (Johnson … Bengio … Mitchell … Grossmann, TiCS, arXiv v4 2026-01-07): "AI systems particularly struggle with metacognition"

What she's reacting against

• The conflation of language fluency with understanding — argues LLMs produce coherent text without forming genuine concepts (v1; restated 2026-06-08)
• Both hype ("LLMs understand") and dismissal ("LLMs are just stochastic parrots") — argues reality is more nuanced and scientifically interesting (v1)
• Benchmark-driven capability claims — shows that LLMs fail on tasks requiring genuine abstraction and analogy even when they pass surface-level tests (v1); now "benchmarking is broken" (2026-06-08)
• The metaphor problem: the way we talk about AI shapes what we believe about AI — and most metaphors are misleading (v1); 2026 targets: "going rogue," "losing control," "escape," "swarm," "evolving"
• NEW: "magical thinking" in mainstream commentary — the idea that AI "emerged" from a scaling law rather than being designed (Substack, 2025-09-15, against Friedman)
• NEW: the "tyranny of tasks" — benchmarks that slice capability into isolated tasks and then get read as job forecasts (Quanta, 2026-08-20)
• NEW: pause-everything policy responses driven by rogue-AI narratives (Substack, 2026-09-10)

Where she overlaps / splits (with Rich)

• Overlaps with Marcus on LLMs lacking genuine understanding; splits on tone — Mitchell is measured and empirical, Marcus is more polemical (v1). Held; the accuracy-to-mechanism migration graded here should be checked in his file too.
• Overlaps with Kambhampati on "LLMs don't reason"; splits on focus — Mitchell emphasizes analogy/abstraction, Kambhampati emphasizes planning/reasoning (v1). Not re-checked — Kambhampati has no bench file.
• Overlaps with Chalmers on LLMs lacking key cognitive features; splits on the question asked — Chalmers asks about consciousness, Mitchell asks about understanding (v1). Held per his file (2026-09-21), which puts him "slightly to Mitchell's optimistic side on understanding" via his sensory-grounding paper.
• Splits with scaling optimists (Altman, Amodei) who expect more compute to close the understanding gap — Mitchell argues the architecture may be insufficient (v1). Drifted — the split now runs through the optimist camp: Sutskever's "age of scaling is over" and "generalization" diagnosis (2025-11) is her mechanism claim in a builder's mouth; Altman and Amodei remain on the other side.
• Splits with Bubeck ("Sparks of AGI") — Mitchell would argue the "sparks" are pattern-matching, not genuine intelligence (v1). Held; no Bubeck file on the bench.
• Splits with Brown on what the Erdős disproof shows — his file carries it as the strongest anti-"can't reason" datapoint of the year; her on-record response is the concession-with-a-question above ("I consider that creative… how much of that is going on here? I don't know," 2026-08-20). Note for the bench: Brown's file does not actually name her as the opposing pole (it names Marcus and LeCun); she is, in practice, the more careful one.
• Overlaps with Jumper more than the bench assumed — she cites AlphaFold as the example of a beneficial technology a pause would harm (2026-09-10). The bench-inferred "partial point" to her from Škrinjar et al. ("current co-folding approaches largely memorise ligand poses from their training data, hindering their use for de novo drug design," bioRxiv, 2025-02-07) is ours, not hers — she has made no claim about AlphaFold's mechanism.
• Shares a byline with Bengio (wise machines, TiCS 2025/26) — under the author-list rule that converts any "split on risk" into a shared position on metacognition; they still differ on catastrophe framing (she mocks extinction odds; he founded LawZero).
• Splits with LeCun less than the labels suggest: both say LLMs lack world models and robust abstraction; she is skeptical that his alternative is any closer ("not yet supported by strong evidence" cuts both ways), and she holds no architecture bet.
• Uses Hinton as her standing exhibit of failed forecasting (2016 radiology) — the split is about epistemic humility, not deep learning.
• Splits with Tegmark and Yudkowsky on pause and extinction framing — explicitly, in-window (2026-09-09, 2026-09-10).
• Adjacent to Brynjolfsson / Autor — her "displacement claims unverified" is a labor-market claim she grades with anecdote (radiology); their files hold the payroll data. Cross-check due next cycle.
• With Mollick: the Ben Lamm file already pairs them as "the empiricist counterweight your bench uses." Cowen is the v1 reach comparator (see Weak spots — that comparison has aged).
• Jim Rutt interviewed her (2023-03-30) — the fullest early statement of the "understanding can't take the stress" line.

Track record

• Artificial Intelligence: A Guide for Thinking Humans (2019) — widely praised as the most accessible and balanced overview of AI for general audiences (v1). Held and extended: reissued with a new preface (Picador, 2025-09-02); CV: named among the "five best books on AI" by the New York Times and Wall Street Journal; shortlisted, 2023 Cosmos Prize for Scientific Writing.
• Published in Science, PNAS, ICML, CogSci on LLM understanding — peer-reviewed empirical work, not just commentary (v1). Held for Science (seven columns 2023–2025 on her essays page), PNAS (2023), CogSci (2024), TMLR (2025), TACL (2026), CDPS (2026), Phil Trans A (2026), AI Magazine (2026); the ICML venue in v1 was not found on the fetched papers list — unverified, carried as v1 text only.
• Santa Fe Institute affiliation brings genuine complexity-science rigor — she's not a pundit (v1). Held on rigor (hand-rated rules, control tasks, child comparisons); the "not a pundit" half has aged — by output count she now is one too (four Science columns a year, a 67,000-reader newsletter, a Bluesky habit), which is a platform, not a demerit.
• Hofstadter lineage (PhD advisor: Douglas Hofstadter) gives deep theoretical grounding in analogy and cognition (v1). Held — CV, verbatim.
• NEW: "Eric & Wendy Schmidt Award for Excellence in Science Communications, National Academy of Sciences, 2025"; "Senior Scientific Award, Complex Systems Society, 2023" (CV).
• NEW: scores herself — the 2025-02 OthelloGPT self-disproof attempt and the 2025-10 admission that reasoning models reached human accuracy on her own benchmark are the two strongest track-record entries of the window, under the sal-khan rule (a persona who tests their own claim gets credit for the test regardless of result).
• NEW: ConceptARC (2023) is now a standard probe; "neither version of GPT-4 has developed robust abstraction abilities at humanlike levels" (arXiv, 2023-11-14) was right for GPT-4 and has been superseded on accuracy by her own follow-up — the benchmark did its job.
• Pattern: exceptional on mechanism calls (every "not humanlike" claim she has made has held at the mechanism level); slow on valence — the record does not contain a prediction that models would reach human accuracy on her tasks, and they did.

Empirical vs normative

• Primarily empirical: runs experiments, publishes peer-reviewed results on what LLMs can and cannot do (v1) — and, this cycle, publishes the results that cut against her
• Normative content is subtle: we should be honest about what AI can do; we should resist metaphor-driven overconfidence; evaluation methodology matters (v1); 2026 additions: society must decide "what AI should be used for" (Yale Review), build for augmentation not replacement, and refuse internet access to autonomous agents rather than pause research
• Source quality: high — peer-reviewed publications in top venues (v1); this cycle every grade rests on her own primaries or on the benchmark owners' own statements
• Conflict note (CONVENTIONS rule 25): low money conflict; moderate program conflict (grants fund the mechanism critique); two advisory seats (Bulletin SASB, OpenEuroLLM) that her columns do not disclose

Weak spots / open questions

• "Understanding" is itself a contested concept — if the definition keeps shifting, the critique can't be falsified (v1). Held as a weak spot, partly answered: the six principles are an operational test, and her "performance vs. competence" distinction gives an observable. But "there's so many different meanings of it" (2026-08-20) is the same escape hatch as 2023.
• Her tests focus on analogy and abstraction — but AI impact doesn't require human-style understanding; economic displacement can happen without genuine comprehension (v1). Held; the Yale Review essay engages jobs directly but with anecdote (radiology), not payroll data.
• The gap between "doesn't understand like humans" and "can still do the job" is the key question she doesn't fully address (v1). Held — "jagged intelligence" names the gap without sizing it; the Erdős concession shows the gap narrowing on her side of the table.
• Santa Fe Institute approach is rigorous but niche — influence on mainstream AI discourse is smaller than Marcus or Cowen (v1). Wrong on "niche" — a scope claim that decayed: four Science columns a year since 2023, a 67,000-subscriber newsletter, a seat on the board that sets the Doomsday Clock, a National Academy of Sciences communication award (2025), Quanta and IEEE Spectrum features, and an EU consortium board. The comparative half (vs Marcus, Cowen) was never measured and is left unmeasured; under the korinek rule, convening power (SFI working groups, Bulletin board) grades standing, and hers is not niche.
• If future architectures (not just scaled LLMs) achieve genuine abstraction, the critique applies to a generation, not to AI in general (v1). Held — she says so herself; every claim is per-model and dated.
• NEW: the retreat pattern. Each time accuracy catches up (GPT-4 → o3 on ConceptARC; ARC-AGI-1 → -3), the critique moves one level down (score → rule quality → objectness prior). That is legitimate science, but the file should hold her to a stopping rule: what accuracy-plus-mechanism result would make her say a model abstracts? None stated on any fetched primary.
• NEW: the PNAS "Illusions of abstraction in contemporary AI" paper exists only as a CV line; until it is public its title is doing rhetorical work the data hasn't yet earned.
• NEW: no written response to the 2026 mathematics results (Erdős disproof, human-verified 2026-05-20; Navier–Stokes claim, 2026-09) beyond one podcast aside — for the bench's leading evaluator of reasoning, the silence is itself a datum.
• NEW: undisclosed seats — Bulletin SASB and OpenEuroLLM appear on the CV and (for the Bulletin) in a press note, not in her columns or Substack.

Rich's take

• (your synthesis here)

Delta log

2026-09-21 — v1→v2 migration + validation (bulk run)

• Grades on carried v1 items: 17 Held / 4 Drifted / 1 Wrong (1 unfalsifiable prediction rewritten, not graded; 2 new items noted as unfalsifiable). Grades on mechanism and valence were split where they diverged.
• Wrong: weak spot "SFI approach is niche; influence smaller than Marcus or Cowen" — Bulletin SASB seat (2025-06-27), NAS Schmidt communication award 2025 (CV), four Science columns a year (essays page), 67,000+ Substack subscribers (header), Quanta (2026-08-20), IEEE Spectrum (2025-12-04).
• Drifted: "LLMs fail at counterfactual/novel analogy tasks" (accuracy half retired by her own 2025-10-06 study; mechanism held); prediction "frontier models still fail at counterfactual analogies" (same source; counterfactual set not re-run on 2026 models); the scaling-optimists split (Sutskever defected to her mechanism claim, 2025-11); affiliation title ("Davis Professor" on no fetched primary; "James B. Alley Jr. Professor" on homepage and SFI profile; change date not found).
• Held: abstraction-gap mechanism (arXiv 2510.02125, valence narrowing per ARC Prize 2026-09-03); metaphors thesis (re-applied 2026-09-10); understanding-not-binary (Quanta 2026-08-20; Yale Review 2026-06-08); benchmarks-don't-measure-understanding (ARC tech report 2026-01-15); better-evaluation prediction (four labs' model cards, same report); Marcus, Chalmers and Bubeck overlap lines; book, publications, SFI-rigor and Hofstadter track-record lines; four of five weak spots.
• Most surprising delta: her own lab retired the accuracy half of her signature finding — o3, Claude Sonnet 4 and Gemini 2.5 Pro "approach or exceed human accuracy" on ConceptARC text (2025-10-06) while the bench file still said "frontier models still fail"; the critique now lives entirely at the mechanism level ("shortcuts," no "objectness" prior). Runner-up: ARC-AGI-3 went from "Frontier AI scores 0.51%" (2026-03-25) to 62.7% / 99.9% depending on harness (2026-09-03) inside the review window.
• Employer / role / funding check (mandatory, dated): title trail — CV "Professor, Santa Fe Institute, 2020–present"; homepage and SFI profile "James B. Alley Jr. Professor"; SFI profile still carries "Professor of Computer Science at Portland State University," which the CV dates 2004–2020 (stale on the institute's own page). New seats found: Bulletin SASB (2025-06-27, primary); OpenEuroLLM Open Strategic Partnership Board (CV only — the consortium's launch release, 2025-02-03 confirms the board, names no members). Ended: SFI Science Board co-chair (2019–2025), Scripps advisory (2023–2024). Funding: NSF ARIA ~$800K (2025–2030), Sandia $603,881 (2024–2026), Templeton World Charity $280,929 (2022–2026), all from the CV. No company, startup, equity or paid advisory role found. Incentive map re-derived: low money conflict, moderate program conflict.
• Signature-framing check: "understanding," "metaphors" and "benchmarks" all re-endorsed in-window (2026-06-08, 2026-09-10); behaviour matches (she keeps testing and publishes both outcomes). New signature term "jagged intelligence" (2026-06-08).
• Leads from sibling runs: (a) Škrinjar memorisation result confirmed on the primary (bioRxiv 2025-02-07) but it is a bench inference, not her claim — labelled as ours; she now does mention AlphaFold, favourably (2026-09-10), so the Jumper file's "neither file mentions AlphaFold" is out of date on her side. (b) Erdős disproof confirmed (nine-author "human-verified version," arXiv 2026-05-20); the Brown file does not name her as an opposing pole — her actual response is the Quanta concession above.
• Search-summary trap logged: a summary dated an "AI and the Barrier of Meaning 2" workshop to March 2026; the CV dates it April 2023 and lists two different SFI working groups for March 2026. Nothing from summaries entered the file.
• Not fetched (paywalled or 403): full text of the Science columns (abstracts via Semantic Scholar), SAGE full text of the CDPS paper (abstract and 2026-04-21 date via Semantic Scholar), Wiley full text of the AI Magazine paper (principles taken verbatim from her 2026-01-14 post), OpenAI's Erdős and incident pages (companion arXiv paper used), the ARC leaderboard page (JS-only; ARC blog posts used). Confidence stays medium for that reason and because the PNAS 2026 paper is unverifiable.
• Tier: semiannual — her own cadence is one substantive post per quarter and four columns a year, but the grades depend on external capability data that moved 0.5% → 62.7% in six months; wake triggers cover the fast events (PNAS paper, a robustness-parity result, her Erdős/Navier–Stokes response, the January Clock statement).

Sources

• Her own, 2026 (primary, fetched): Misleading Metaphors and Real Risks (2026-09-10) · Jagged Intelligence, Yale Review (2026-06-08) · On Brian Cantwell Smith (2026-02-10) · On Evaluating Cognitive Capabilities in Machines (2026-01-14) · Six principles, AI Magazine 4(2), 2026 · On evaluating abstraction and analogy in humans and machines, CDPS (2026-04-21) · Bluesky feed (posts 2026-08-03 to 2026-09-13) · Homepage (title, talks through 2027-02) · CV (positions, grants, service) · Papers page · Essays page
• Her own, 2025 (primary, fetched): Do AI Reasoning Models Abstract and Reason Like Humans? (2025-10-06) · Beger et al., Do AI Models Perform Human-like Abstract Reasoning Across Modalities? (arXiv v1 2025-10-02, v4 2026-02-02) · Magical Thinking on AI (2025-09-15) · Why AI chatbots lie to us, Science (2025-07-24) · Artificial intelligence learns to reason, Science (2025-03-20) · LLMs and World Models, Part 2 (2025-02-13) · Krakauer, Krakauer & Mitchell, LLMs and Emergence (arXiv 2025-06-10; Phil Trans A 2026) · Stevenson et al., Can LLMs generalize analogy solving like children can? (arXiv v3 2025-10-06; TACL 14, 2026) · Johnson et al. incl. Bengio & Mitchell, wise machines (arXiv v4 2026-01-07) · Lewis & Mitchell, robustness of analogical reasoning (arXiv 2024-11-21; TMLR 2025)
• Her own, earlier (primary, fetched): The metaphors of artificial intelligence, Science (2024-11-14) · Lewis & Mitchell, counterfactual tasks, CogSci (arXiv 2024-02-14) · Mitchell, Palmarini & Moskvichev, humans vs GPT-4/4V on ConceptARC (arXiv 2023-11-14) · Mitchell & Krakauer, The debate over understanding in AI's LLMs, PNAS 120(13), 2023 · Picador reissue with new preface (2025-09-02)
• Interviews / third-party (fetched): Quanta, The Joy of Why (2026-08-20) · IEEE Spectrum at NeurIPS (2025-12-04) · Jim Rutt Show, Currents 088 (2023-03-30) · Bulletin: joins the Science and Security Board (2025-06-27) · Doomsday Clock statement (2026-01-27) · SFI profile · OpenEuroLLM launch release (2025-02-03)
• Capability data used for grading (fetched): ARC Prize, Announcing ARC-AGI-3 (2026-03-25) · ARC Prize, GPT-6 Astra on ARC-AGI-3 (2026-09-03) · ARC Prize 2025 Technical Report (2026-01-15) · Alon, Bloom, Gowers, Litt, Sawin, Shankar, Tsimerman, Wang & Wood, Remarks on the disproof of the unit distance conjecture (2026-05-20) · Škrinjar et al., co-folding memorisation (bioRxiv 2025-02-07)
• Carried from v1: Artificial Intelligence: A Guide for Thinking Humans (2019) · Santa Fe Institute profile · melaniemitchell.me

migrated v1→v2 2026-09-21 · backup: _archives/melanie-mitchell.html.bak-20260921 · converts-from: people/melanie-mitchell.md · AI & Society domain