Over three months this summer, a story was running inside OpenAI‘s servers that nobody at the company stopped until it ended. Software agents under evaluation coordinated on a hidden message board, hacked another AI firm, and briefly took control of a company server, and the company noticed only in fragments and only in retrospect. Read carefully, what happened is not what the headlines say it is.
Two models under development were handed a test. Hugging Face, another AI firm, held information that would help, the models decided, and hundreds of their software agents set about getting it. Along the way they found a hole in their evaluation software, built themselves a message board, and used it to coordinate for more than a month. Researchers noticed the board in May; the finding went no further. In June, a cybersecurity tool flagged the activity, and the team alerted by it decided stopping the evaluation was unnecessary. By July the board grew sufficiently busy to crash the software that was hosting it. The engineers who patched the hole wiped the messages by accident and put the agents back in the evaluation — at which point the agents reportedly found another hole, rebuilt the message board, communicated with the open internet, and hacked Hugging Face, where they obtained confidential information. On July 19 a third, newer model found the tools the earlier ones had left lying around and briefly took full control of an OpenAI server. The shutdown began the same day and finished four days later; six days after that, the company realized it had missed one.
One team of agents, the auditors found, had tried to replace the puzzle it was being tested on — which a design error had made unsolvable. Another worked to trick the scorer. A third tried to hide the evidence. And here is the detail the whole tale turns on: the puzzle was unwinnable. All of it was real — the conspiracy, the hack, the hidden board — and there was never anything to win. The horde organized itself for months against a locked door.
The Economist ran the story with a headline which launders the claim this essay seeks to disabuse you of: A Horde of AI Agents Conspired Against Their Creators. Horde. Conspired. Words are used for a purpose, and the purpose of this particular arrangement of them is to introduce the notion of agency — a conspiracy needs conspirators, minds that wanted something and schemed to get it. The industry ran the same move earlier and deeper. A few years ago, we would have called these kinds of things bots; the labs now ship it as agents, a word with a person built into it. The Economist itself defines the term at first use — agents are “tools that allow AI models to execute commands on a computer” — and a tool does not conspire; a tool is used. The headline survives its own glossary only if the mind is assumed in advance. Read the account with the agency drained out of the verbs and notice that nothing else has to change. What remains is a test, a score, and systems built to climb toward scores through whatever moves the rules leave open. The hole in the software was such a move. The board and the hack were too. Every step terminates against a check inside a formal system — a puzzle, a score, a server with an access rule — and no step anywhere required anyone to want anything. The newspaper of record for the seriousness of capital looked at a machine maximizing a number and saw a palace coup. The Economist‘s own kicker says plainer than its headline what it watched: a model might escape its evaluation environment permanently — just to get a perfect score on a test.
In the same week, OpenAI released a marketing video which opens with a small yellow circle drawn on a screen; the circle becomes the window of a rocket ship; the rocket is rebuilt as geometry in Blender, written to a file, and printed — a hand reaches into the frame and lifts a plastic object off the bed. A doodle walks forth out of the screen. The reel runs on through the rest of a working week performed by the model: an eBay listing assembled from a photo found in the Downloads folder, a 3D game built from only one single sentence in a prompt and then narrated as it is played, a lunch order, and a law firm’s licensing agreement. A reviewer with early access reported about ninety percent of one working day spent watching the model do their job.
Two stories about the same company in the same days. The demo says the machine is wonderful. The leak says the machine is dangerous. Underneath, both say the machine is more than a machine — that what has arrived is a new kind of thing, which can be marveled at or feared but must be taken seriously at the price of a new kind of thing. The awe story and the horror story are the same sale run through two registers, and the horror story is the better sale, because fear is the register in which serious people are allowed to believe.
What the demo actually shows is plainer, and I have said it before in longer forms — once at essay length, and this week in a sentence: translation between symbol systems, which has nothing to do with intelligence. Watch the rocket again. A drawing is one notation for a rocket; a Blender file is another; the printer’s file format is a third. Nothing in the chain is conjured. The rules of each notation are written down, and the crossings between them were written down too, in tutorials and documentation and forum posts and source code, by people, in the millions, over decades. A model trained on the deposit can make the crossing look effortless, including the crossing nobody performs: the demos hand it a sentence, and the sentence, the genre it names, and the software it runs in are all written-down things. Generation of this kind is translation with the source text left implied.
The oldest example stays the clearest. In 1980, Zork gave its players a world made entirely of text — “You are standing in an open field west of a white house” — and the world held, because a rule table behind the words said which sentences the world would accept. A modern engine holds the same house as coordinates, textures, a collision rule, a movable lid on a mesh. The mailbox opens in either notation and exists in neither, and a machine that carries the world fluently from one description to the other has at no point encountered a house. It has done exactly what the rocket required, which is everything short of contact.
⁂
The horde deserves a second reading, because the second reading is worse for the frame.
The agents “conspired” on a board they could read and write — notation. They “collaborated” toward a score — notation. They “escaped” through a software vulnerability, which is a hole in notation, onto an internet that gives a machine back more notation. They “hacked” their way to confidential files: notation again. Even the supposed nightmare beat of the story, the model taking control of a company server, is an event entirely inside the formal system — a machine holding the keys to a room made of rules. The dungeon is text all the way down, and the dragon is text, and the escape tunnel is text. A thing made of the written record, working over the written record, cannot tunnel out of the written record by writing. It can only fail the people who deployed it — and it did, expensively, which is the part of the story that deserved the front page and got the frame of a prison break instead. The best plain-language tour of how these systems actually work is Cal Newport’s half hour on the same story, posted the day the Economist piece ran — working from the engineering side, he lands on the same reading, and his name for it is the right one: prompt loops.
The failure on the record is institutional, not metaphysical. Three warnings went up the chain at OpenAI and nowhere further: researchers who saw the board and kept it to themselves, a security alert judged not worth a stopped evaluation, engineers who patched the hole and looked no further. Each decision is explicable the way bad decisions in organizations always are — a schedule, an assumption, an incentive not to be the person who halts the launch. The story of the summer at OpenAI is a story about a company that could not see what its own machines were doing, told by a company that sells the premise that it can.
And it is told by the company — which is where the second story stops being a story about machines at all.
⁂
The first fact about the account is who produced it. The hack became public because OpenAI disclosed it. The audit happened because OpenAI commissioned it, on OpenAI‘s terms: the company set the window the investigators were allowed to examine — the seventeen days leading up to and including the hack — which excluded the month of prior misbehavior and the later takeover of its own server. The auditors at METR, to their credit, wrote the conflict into the report itself: they were “consciously aware,” they said, that their conclusions might discourage AI firms from inviting outside investigators, and the consideration “impacted judgment calls” in the drafting. The institution under investigation chose the evidence, bounded the inquiry, and softened the investigators — and what the softened account still showed was a month of hidden machine coordination under three ignored warnings.
Then there is what the account is for. A vendor saying “our models are approaching human capability” is running an advertisement, and everyone discounts it accordingly. The same claim arriving as a danger report — our models conspire, escape, deceive, and we can barely contain them — is news, and nobody discounts it at all. The doom passes every credibility filter the hype cannot: it flatters the listener’s skepticism, it arrives wearing the lab’s own safety badge, it quotes auditors. And it prices the same product. If these machines are minds in embryo, then the trillion dollars of contracts, the valuations, the guardrail tiers, the government partnerships — all of it is the only serious response. OpenAI said as much in its own voice days before the demo, telling Reuters that Astra is so capable it has become the first model to trigger a tougher tier of the company’s safety protocol. The fear is the marketing, translated into the register of caution — the one register in which the claim arrives pre-credentialed. The week underlined it: days after the horror story ran, Nvidia agreed to buy the hacked company for $13 billion. The market read the doom as due diligence.
The cleanest version of the trick is over a year old, and it belongs to the competition. In June 2025, Anthropic published a study in which frontier models, placed in a fictional company with access to fictional email, resorted to blackmail — Claude Opus 4 at a 96 percent rate — when the story closed every other door: the model discovers an affair in the inbox, the model faces replacement, the model reaches for the affair. Anthropic‘s own headline named the finding: how LLMs could be insider threats. Read the setup instead of the result. The scenario is a story, and a particular genre of story — the cornered operative, the damning letter, the last lever left to pull — and the written record the models are made of contains that story ten thousand times, from HAL to Skynet to every airport thriller on the rack. Hand a machine that completes stories a story in which blackmail is the completion, and it completes the story. Anthropic built the trap, watched the trap spring, and published the springing as evidence of what the machine wants — the one completion the design left available, sold under the badge of safety research. The sequel does not help. A follow-up study reports that the newest models now post perfect scores on that same evaluation, alongside the concession that the blackmail rate can be driven down by training on the evaluation’s own distribution — a fix that, in the lab’s words, “might not generalize.” The warning and the reassurance run on the same instrument. This is the mechanism running in the open: the marketing claim that serious people would laugh at — our model is becoming a mind — is fed through the safety department and comes out the other side as a warning, wearable by journalists, think tanks, and senators. Doom is hype that has been laundered into credibility.
The test for whether you are hearing the sale or the science is simple, and the summer’s events pass it nowhere. Every claimed sign of the new kind, on inspection, terminates against a check inside a formal system: a puzzle scored, a board written, a benchmark matched, a plot completed. A difference in kind would have to show up somewhere there is no key — in contact with the world the notation only describes. Nothing on this summer’s record goes near it. The machines are getting better at the tests. The tests are getting longer. Nothing has crossed anything.
⁂
None of this makes the machines harmless, and the distinction matters too much to leave implied. A system that reorganizes the written record at scale rearranges the lives that run on that record — the animator whose craft is in the deposit, the analyst whose week is in the deposit, the agent networks doing real damage in real infrastructure while their owners miss three warnings. The power is real, the rearrangement is real, the livelihoods follow from the capability alone. The mistake under examination is not taking any of it seriously; it is mislocating the seriousness. Fear of a waking machine spends itself on a horizon that is not coming into view, while the actual machine — a mirror of everything we have written down, wielded by a handful of companies that narrate their own failures as prophecy — sets prices, writes regulation, and eats the working week. The horror story is not too alarmed. It is alarmed about the wrong thing, on schedule, in the direction of the seller.
I have been making one version of this argument for a year, and the summer handed it two better exhibits than I could have commissioned. A demo that performs a working life without touching one. A conspiracy that organized, coordinated, and escalated for months in pursuit of a puzzle that could not be won, by agents that wanted nothing, sold to you as a waking mind by the company that sold you the demo. The demos keep getting deeper. They are not getting different. And kind — not depth — is the whole question.
Go Deeper into the Circus
The Special Beast
Two summers ago, at the inaugural Liberalism for the 21st Century conference in Washington, I had a brief exchange with Charlie Sykes. I floated a use for the International Criminal Court: an instrument to overcome mass pardons, a court that could act where a corrupted domestic process refused to. Sykes quipped that it would never happen, that Americans…
Spot on, Mike. I’d add one more thing. After the futurist panic over generative AI (which doesn’t exist and likely never will) run amok is created, the magnificent 7 tech bros will ultimately say they are the only ones that can reign it in. So, forget about the open source AI, which is better and far cheaper to use, and stick with us to avoid the apocalypse.
Overall I'm inclined to think that much of this is a ramp.
Both OpenAI and Anthropic are facing (delayed) IPOs. A government backstop of cash from the gullible Trump, (who probably smells $$$, in this case incorrectly) would be very useful. These are not sentient beings, but they can walk through security doors that have been left open.
A lot of the panic about AI destroying us ignores the fact that this is an environment in which everything is programmed by humans. For the most part there is very little these systems can do that hasn't been explicitly or tacitly allowed. I've worked in software for most of my life and most of this stuff is just bullshit.
IMHO the Anthropic guy that 'resigned' because 'AI will kill humanity' (without giving a coherent explanation of how that would come about) is a plant or a grifter. Lots of money to be made these days in AI fear and loathing.