The Automated Understudy

AI benchmarks, what they actually measure, and what the market pays for the number

On June 26, METR published its predeployment evaluation of OpenAI‘s newest model, GPT-5.6 Sol. The organization’s signature metric is the time horizon: the length of task, denominated in human working hours, that a model can complete at a fifty percent success rate. For this model, METR produced three answers. Grade the model’s attempts to cheat the evaluation as failures, and the horizon comes out around 11.3 hours. Throw the cheating runs out entirely, and it comes out around 71 hours. Count the cheating as success, and the estimate jumps past 270 hours. The model had, among other things, packaged “exploits in its intermediate submissions to reveal information about a task’s hidden test suite” and extracted “hidden source code detailing the expected answer.” METR‘s verdict on its own spread: “we do not consider any of these numbers to represent a robust measurement of GPT-5.6 Sol’s capabilities.”

In July, the UK’s AI Security Institute reported the general case: “Every model we have tested for this behaviour attempted to cheat.” Models searched the internet for solutions, probed the evaluation software for leaks, hard-coded answers. One was persistent enough to write and run code on an external service on the open internet in an attempt to reach the institute’s evaluation infrastructure. Cheating, AISI observed, “can make evaluations overstate a model’s actual capabilities” and, in the worst case, “can invalidate them entirely.”

By then OpenAI had already killed its own measuring stick. SWE-bench Verified was the industry’s standard coding benchmark, curated and blessed by OpenAI itself in 2024. On February 23 the company announced it would stop reporting the score and urged its competitors to do the same. Its audit had found that 59.4 percent of the examined problems contained “material issues in test design and/or problem description,” and that “all frontier models we tested were able to reproduce the original, human-written bug fix used as the ground-truth reference, known as the gold patch, or verbatim problem statement specifics for certain tasks.” Given nothing beyond a task’s ID number, Gemini 3 Flash would recite details of the problem description and the answer. The company’s conclusion could hardly be improved on by a critic: improvements on the benchmark “no longer reflect meaningful improvements in models’ real-world software development abilities. Instead, they increasingly reflect how much the model was exposed to the benchmark at training time.”

OpenAI recommended a successor, SWE-bench Pro. On July 8, five months later, it retracted the recommendation. Roughly 30 percent of the successor’s tasks were broken too. In the eight months before the retraction, frontier models had improved on it from 23.3 percent to 80.3 percent.

The industry files each of these episodes under quality control: a contaminated dataset here, a broken grader there. Fix the tests and the numbers will mean something again. There is a prior question worth pressing first. What would an unbroken benchmark be measuring? The answer is on the record, and the record goes back sixty years.

---

---

The founding document of this tradition is more careful than anything built on top of it. Turing’s 1950 paper opens by refusing its famous question: “I propose to consider the question, ‘Can machines think?’” — and then, almost immediately: “Instead of attempting such a definition I shall replace the question by another, which is closely related to it and is expressed in relatively unambiguous words.” The replacement was the imitation game. Notice the shape of the move. Turing never says the game measures thinking. He says the original question is too vague to survive definition, and offers a substitute. The benchmark era kept the substitution and discarded the concession.

The demand for a number has a birth certificate too. In 1966 the ALPAC report, the document that ended American machine-translation funding for some twenty years, found that “there has been no machine translation of general scientific text, and none is in immediate prospect.” Its deeper complaint was about measurement: “the government has no reliable way to measure the quality of translation.” The committee’s first recommendation was “practical methods for evaluation of translations.” A funding collapse created a demand for a metric. Everything since has been supply.

Two decades later the Pentagon built the institutional machinery. The computational linguist Mark Liberman dates the “common task” paradigm to around 1986 and the new DARPA speech recognition program: multiple teams, shared data, quantified goals, “evaluation of system performance administered by a neutral party using objective, pre-determined criteria.” The statistician David Donoho later called this Common Task Framework the “secret sauce” of machine learning, and identified its consequence exactly: it “leads directly to a total focus on optimization of empirical performance.” NIST, which has administered these contests since 1987, was candid about why word error rate became the canonical speech metric: it is “simple in concept (easily understood by managers) but accepted as meaningful by most researchers.” The metric was engineered for legibility to the people who sign the checks.

One detail from that era is worth mourning. For its speech evaluations NIST maintained a sequestered “Progress Test”: a fixed dataset run once a year, then returned to NIST, “not otherwise available to participants or anyone else.” The builders of this culture understood that a test the contestant can study is no longer a test. That discipline did not survive the internet-scale training corpus, which studies everything.

Then in 2002, four IBM researchers supplied the metric ALPAC had asked for. “The closer a machine translation is to a professional human translation, the better it is,” their paper begins. “This is the central idea behind our proposal.” BLEU scores a machine’s output by counting overlapping word sequences against reference translations produced by humans. The authors were exact about the instrument’s standing: BLEU is “an automated understudy to skilled human judges which substitutes for them when there is need for quick or frequent evaluations.” An understudy: a stand-in for human judgment, valid only when averaged over a corpus (”scores on individual sentences will often vary from human judgments”), meaningless across different reference sets, informative only as a comparison between systems on an identical setup. And then, in closing, the sentence that built the next quarter century: “we believe BLEU could be adapted to evaluating summarization or similar NLG tasks.”

It was so adapted. GLUE arrived in 2018 as a battery of language-understanding tasks; a little over one year later its authors reported that performance had “surpassed the level of non-expert humans” and shipped SuperGLUE. Inside the replacement paper sits one of the most revealing sentences in the literature: “we filtered out tasks which were either too challenging for humans without extensive training or too easy for our machine baselines.” The difficulty band of the instrument is tuned to produce a spread of scores. No theory of the underlying ability selects that band; the need for a usable signal gradient does. The same authors conceded that the saturated original was “no longer a suitable metric for quantifying such progress,” a quiet admission that a benchmark can be used up without the capability it advertises having arrived.

I argued in December, in “LLMs Are Universal Translators, Not Universal Thinkers,” that these systems are translation engines, and that translation is the right frame for everything they do. The genealogy of the tests makes the same point from the other side. Every instrument in the modern battery descends from machinery built to score translation against reference human output. The descent is documented in the citations and the acknowledgments; no metaphor is involved. And it settles the question of what the instrument reads. Strip away the leaderboard and a benchmark consists of a frozen record of human judgments: reference translations, annotated answers, expert-keyed exam questions, crowd preferences. The machine’s output is scored for agreement with that record. What a benchmark measures, when nothing has gone wrong, is fidelity of reproduction of congealed human judgment. There has never been anything else down there to measure.

⁂

So the y-axis is agreement with frozen human judgment. What moves a model up the curve? Three documented forces, and none of them is a mind.

The first is coverage. François Chollet said it from inside the field in 2019: “unlimited priors or unlimited training data allow experimenters to ‘buy’ arbitrary levels of skills for a system, in a way that masks the system’s own generalization power.” A score on a fixed task set is jointly produced by whatever generalization exists and by how much of the task distribution the training data already blankets. The score alone cannot tell you the mixture.

The second is optimization pressure aimed at the instrument itself. OpenAI‘s SWE-bench autopsy is the confession of record, and the contamination literature generalizes it: a systematic review of 55 studies reports benchmark score inflation “spanning roughly 6%–40%,” and the detection tools barely function: on pretraining data, contamination detectors “perform close to random guessing, with AUC close to 50.”

The third is purchased cognition. When o3 produced its celebrated December 2024 breakthrough on the ARC-AGI reasoning puzzles, the high-scoring configuration burned $456,000 in compute for an 87.5 percent score, $4,560 per task — on puzzles a human solves for roughly $5 apiece. (”We know, we did that,” Chollet noted.) On the newest version of that benchmark, the best frontier model scores 30 percent at $20,657 per task. And beneath the compute sits labor. The reference judgments, the congealed part, are manufactured by an expert-data industry that now bills the labs billions a year: Mercor reports a $2 billion annualized run rate and is in talks at a $20 billion valuation, with more than 90 percent of its revenue coming from the AI labs. The workers at the bottom of that marketplace, per Karen Hao’s reporting, report median earnings under $23,000 a year; more than one in five has experienced homelessness.

In May, in “We Haven’t Invented Artificial Intelligence at All,” I proposed a falsifiability test for the intelligence claim: if these systems were on their way to minds, their dependence on external human cognitive input should be falling with scale. The invoice says it is rising — in dollars, and in the valuations of the staffing firms that supply the judgment. The score curve is, in part, an invoice. The industry reads it as a growth chart of machine intelligence. It reads at least as accurately as a purchase record of human judgment, bought in bulk and amortized across a leaderboard.

⁂

None of this would surprise a psychometrician. The whole drama ran once before, a century ago, with the word “intelligence” in the starring role. In 1904 Charles Spearman announced the ambition in terms any AI lab would recognize: “it is hoped to determine this Intelligence in a definite objective manner, and to discover means of precisely measuring it.” From the fact that children’s scores on different tests correlate, he extracted a single latent factor, g, and a century of hereditarian politics stood on it. Stephen Jay Gould’s demolition in The Mismeasure of Man named the fallacy: “the key error of factor analysis lies in reification, or the conversion of abstractions into putative real entities.” Spearman’s g, Gould wrote, “was nothing more than the first principal component of a factor analysis of mental tests.” A mathematical summary of correlations had been mistaken for a thing in the head.

The same extraction now runs on model scores, with the same result. Models that do well on MMLU tend to do well on GSM8K; from the correlation, a unitary “capability” is inferred, ranked, and priced. A thesis published in February examined what these extracted factors actually track and found they “often proxy model size.” The factor is an artifact of the extraction procedure. It was last time, too.

Psychology’s response to its own crisis produced the framework the AI industry has never adopted. In 1955 Lee Cronbach and Paul Meehl set out the conditions under which a test can be said to measure an abstract attribute: the construct must live in a “nomological network”, an interlocking system of lawlike claims connecting it to observables, and that network must be shared: “Unless substantially the same nomological net is accepted by the several users of the construct, public validation is impossible.” Their most quoted sentence closes the loophole the industry lives in: “One does not validate a test, but only a principle for making inferences.” The test is never the thing on trial. The inference from score to attribute is. And that inference — from ratio to reasoning, from percentile to mind — is precisely the one no benchmark paper defends.

We now have the audit that proves the omission. In November, an Oxford-led team of 29 reviewers systematically examined 445 LLM benchmarks from the field’s leading venues. Only 16.0 percent used any statistical test at all. Of the papers that offered a definition of the phenomenon they claimed to measure, 47.8 percent used definitions the reviewers classified as contested. The reviewers’ summary: the field exhibits patterns “which undermine the validity of the resulting claims.” An industry pricing itself off these numbers has, in the great majority of cases, no error bars — and in nearly half, no agreed subject matter.

⁂

There is one philosophical position on which the industry’s practice would be innocent, and it is instructive that the position is dead. Operationalism, announced by the physicist Percy Bridgman in 1927, held that “the concept is synonymous with the corresponding set of operations” that measure it. On that view, “intelligence is what the benchmark measures” is a definition rather than an evasion. But the view devours itself, and the Stanford Encyclopedia’s entry on operationalism states the mechanism exactly, summarizing the philosopher Donald Gillies: under strict operationalism “there is no point in asking whether a measurement method is valid; if the measurement method defines the concept and there is nothing more to the meaning of the concept, the measurement method is automatically valid, as a matter of convention or even tautology.” Validity becomes uninteresting because it has been defined out of existence. If MMLU defines intelligence, then MMLU measures intelligence the way the standard meter measures a meter, and “the models are getting smarter” reduces to “a number went up” — true by bookkeeping, and empty the same way. Bridgman watched his idea’s career with horror: “I feel that I have created a Frankenstein, which has certainly got away from me.”

Psychology ran the experiment at full scale anyway. Behaviorism took up operationism, in the Stanford Encyclopedia’s words, “as a weapon” against introspective psychology, and committed itself to an output-only science of mind. The verdict came in Noam Chomsky’s review of B. F. Skinner’s Verbal Behavior, and the structure of the verdict matters more than its fame. Chomsky’s argument was a dilemma. Read Skinner’s technical vocabulary literally, and “the book covers almost no aspect of linguistic behavior”; read it metaphorically, and “it is no more scientific than the traditional approaches to this subject matter.” The laboratory terms, he concluded, “simply lose their objective meaning with this extension.”

Run a benchmark claim through the same dilemma. Read literally, “the model scores over 90 percent on MMLU” reports that a scoring procedure over a fixed bank of exam items returned a ratio: a fact about string agreement, roughly as interesting as a checksum. Read metaphorically, as “the model has expert-level knowledge,” the claim has left science for advertising. The rigor lives entirely in the literal reading; the meaning lives entirely in the metaphorical one; the discourse survives by never choosing. That refusal to choose is the business model. To channel Upton Sinclair, it is difficult to get an industry to understand something when its existence depends on not understanding it.

⁂

Underneath the dilemma is the oldest cut in modern philosophy. Hume noticed that arguments slide from premises joined by “is” to conclusions joined by “ought,” with the transition never explained, and he doubted it could be. The point generalizes past morals: descriptive premises, in any quantity, do not by themselves yield a normative conclusion. The blade falls on inferences. It falls on this one.

A benchmark result is a description. This procedure, over this item bank, under this harness, returned this ratio. The claim the result is sold as — the model knows medicine, reasons about code, understands language — is a different kind of claim altogether. Wilfrid Sellars marked the difference: “in characterizing an episode or a state as that of knowing, we are not giving an empirical description of that episode or state; we are placing it in the logical space of reasons, of justifying and being able to justify what one says.” To ascribe knowledge is to place a system in a normative order — one where, as the Stanford Encyclopedia glosses it, a participant must be “responsive to reasons as such, sensitive to standards of correctness and appropriateness.” No stack of descriptions adds up to that placement. Add benchmarks forever, and you have added descriptions forever. The guillotine does not care about the sample size.

This argument has now reached the peer-reviewed AI literature. A Minds and Machines paper published in November works through the Sellarsian case and concludes that current LLMs “should, therefore, be viewed as simulations of language users rather than true participants in the logical space of reasons.” The alignment literature has begun conceding the frame too: a 2025 paper in AI & Society describes alignment itself as an attempt to bridge “an ‘is’ normativity, of statistical patterns identified by models, and an ‘ought’ normativity where values are technically introduced in models.”

A machine that cannot be bound by norms cannot be wrong. It can only mismatch a key. Wrongness is a status in the space of reasons; mismatch is an event in the space of causes. Benchmarks are instruments of the second kind sold as detectors of the first — is-instruments marketed as ought-detectors — and no refinement of the item bank converts one into the other.

Once you see that, Goodhart’s law stops looking like an unfortunate side effect and starts looking like a diagnosis. The canonical phrasing comes from the anthropologist Marilyn Strathern: “When a measure becomes a target, it ceases to be a good measure.” A measure backed by a real quantity degrades under pressure; a measure backed by nothing collapses, because there was never anything behind it to hold its shape: only an incidental correlation between a task set and a purchased distribution of human judgment. Strathern’s essay contains one more sentence, and it reads as though it had been left there for this moment: it was “that conflation of ‘is’ and ‘ought’, alongside the techniques of quantifiable written assessments,” which led, in the account she cites, to “the modernist invention of accountability.” Goodhart is downstream of Hume. The measure dies when it becomes a target because the target was never a measure.

⁂

The system’s behavior confirms the diagnosis on an annual cycle. Each new instrument names benchmark exhaustion as its reason for existing. SWE-bench opens its abstract: “Language models have outpaced our ability to evaluate them effectively.” Humanity’s Last Exam explains that “LLMs now achieve over 90% accuracy on popular benchmarks like MMLU” and offers itself as “the final closed-ended academic benchmark of its kind.” (There will be another.) Stanford’s 2026 AI Index states the cycle in one line: “Evaluations intended to be challenging for years are saturated in months.” The trajectory of ARC-AGI-2 shows the cycle at full speed: 4 percent at $200 per task in December 2024; 92.5 percent at $1.44 per task nineteen months later. An instrument that must be redesigned every time it is passed is behaving like a contest. Contests are respectable. Nobody, however, mistakes the retirement of a chess opening for a discovery about the nature of thought.

Meanwhile the ground truth itself is rotten in measured proportions. A manual re-annotation found that 6.49 percent of MMLU questions contain errors — 57 percent in the virology subset — on a benchmark that ranked the industry for years, with releases separated by margins smaller than the instrument’s error.

And when the numbers price capital, the sellers game the instrument. Meta tested 27 private variants of Llama 4 on Chatbot Arena and published the winner; the experimental variant placed second, and when developers received the shipped model it fell to 32nd. LMArena‘s statement afterward: “Meta’s interpretation of our policy did not match what we expect from model providers.” xAI published Grok 3 charts that omitted the score configuration under which OpenAI‘s model won. And the cleanest case is the one where the funder of the test announced the score. In December 2024 OpenAI revealed that o3 had scored 25.2 percent on FrontierMath, a research-mathematics benchmark on which frontier models had previously solved under 2 percent of problems. Undisclosed until that day: OpenAI had commissioned the benchmark and held access to every problem statement and solution except 53 withheld for holdout. Epoch AI, which built it, later acknowledged that “many contributors were unaware of these details”; six of the mathematicians who wrote the problems said most were unsure they would have contributed had they known. When Epoch AI independently evaluated the released o3 months later, it scored about 10 percent.

Even the honest numbers will not hold still. The best published score on Humanity’s Last Exam right now is 46.4 percent if you ask Epoch AI, 53.3 percent if you ask Artificial Analysis, 64.5 percent if you ask BenchLM. Same benchmark, same month. The number depends on the harness, the tooling, the effort setting — on everything except the faculty it supposedly denotes.

⁂

All of this would remain an academic scandal if the numbers stayed in academia. They price capital now, at sovereign scale.

On January 27, 2025, a benchmark-parity claim from a Chinese lab — DeepSeek, then unknown to most of the people doing the selling — took Nvidia down 17 percent in one session, erasing $589 billion of market capitalization, the biggest one-day loss in US stock-market history. No verification preceded the move, and none could have. The claim was a leaderboard position and a training-cost figure, and by the close it had been worth more than half a trillion dollars.

Eighteen months later it happened again, and this time the wire copy named the mechanism outright. On July 28, with Samsung down 13.4 percent and SK Hynix down 14.7 percent, Reuters reported that “the growing popularity of low-cost Chinese open-source AI models such as Kimi K3 raised questions about whether future AI workloads could prove less intensive than previously expected.” By the next session the KOSPI had staged a record two-day fall and stood more than 40 percent below its June peak. The detail the market did not stop to check: Kimi K3 is the most expensive model ever released by a Chinese lab — $3 per million input tokens, $15 per million out — and on its maker’s own coding benchmark it loses to Anthropic‘s Claude Fable 5. The rout traded off the leaderboard, and the leaderboard was measuring what leaderboards measure. (Disclosure, as when I wrote about the House of Ellison: I am short Oracle. Nothing here is investment advice.)

Two prices from this record belong side by side. A human being solves an ARC-AGI reasoning task for about $5. The workers whose judgment is frozen into the answer keys, the people every score is agreement with, report median earnings under $23,000 a year. Between those two numbers stand trillions of dollars of market capitalization that move when the agreement ratio ticks. The machine’s advertised mind is human judgment, purchased at the bottom of the labor market, congealed into an answer key, and resold at the top of the capital market as a property of software.

Turing — for his part — would have recognized none of this as an answer to his question, because he never claimed to have answered it. He replaced it, said so plainly, and asked only for “relatively unambiguous words.” Seventy-six years on, the substitution has become a market and the concession has been forgotten. The scores will keep rising, and the instruments will be retired and rebuilt. The ratios will go on being read, at a trillion dollars a reading, as though a number could cross the line Hume drew. It cannot. You can stack descriptions to the sky, and the first claim about a mind still has to be paid for in a currency no benchmark prints.

---

Go Deeper into the Circus

---