The Doomsday Moat

The AI Industry Cannot Deliver the Returns It Promised. So It Made Extinction Part of the Pitch

At some point early in the discussions, somebody at OpenAI proposed giving the United States government 5 percent of the company. The proposal has been presented as a public-spirited response to the extraordinary wealth artificial intelligence will create. Sam Altman and other OpenAI executives have reportedly discussed placing the stake in a public investment vehicle modeled on the Alaska Permanent Fund, through which ordinary Americans might share in the gains. At OpenAI’s reported valuation of $852 billion, the government’s share would be worth approximately $42.6 billion. OpenAI has suggested that other leading American AI companies should surrender comparable stakes.

It sounds generous until you ask why one of the most valuable private companies in the world is volunteering to place the federal government on its capitalization table. OpenAI is not known for surrendering control to the public. Its history has been a long migration away from the nonprofit structure through which it first claimed to hold advanced artificial intelligence in trust for humanity. The sudden enthusiasm for public ownership is a transaction, even if no cash changes hands.

The sovereign-shareholder model is already operating. Last August, Washington converted previously awarded CHIPS Act and Secure Enclave funds into an $8.9 billion investment and a 9.9 percent stake in Intel. Nvidia and AMD separately agreed to remit 15 percent of certain Chinese chip-sale revenues to the government in exchange for export licenses. The state has begun exchanging permission, subsidy, and protection for claims upon private technology companies.

OpenAI made its offer under pressure. It surfaced less than a week after the company delayed the broad release of GPT-5.6 at the government’s request and weeks after a coalition of 42 state attorneys general opened an investigation into the company. The offer is tribute as well as entanglement: political insurance offered to a government already demonstrating its capacity to interrupt the business. Anthropic, importantly, has held no comparable discussions with the administration. This is OpenAI’s proposal, not a position that can be imputed to the industry as a whole.

What OpenAI receives is more valuable than the equity it gives away. The federal government becomes a shareholder in the company’s valuation, a political participant in its continued solvency, and a beneficiary of the future returns OpenAI has promised but has not yet produced. A restriction that reduces OpenAI’s revenue would then reduce the value of a public asset. A competitor that displaces OpenAI would threaten an investment notionally held on behalf of every American. The regulator would acquire a financial interest in the regulated company before anyone had established that the company could earn returns commensurate with its valuation. This is not a conventional bailout. It is the architecture of a future bailout assembled in advance.

OpenAI wants the government on its capitalization table because of a problem the company cannot say plainly. The artificial-intelligence industry has promised economic returns that the technological roadmap it is following cannot deliver. The models will become more useful. They will reshape software and many forms of knowledge work. But they will not become the autonomous general intelligences required to justify the scale of investment now being made in their name, and much of the value they do create will be captured outside the companies paying to build them.

The industry did not invent the doomsday argument. Eliezer Yudkowsky was writing about artificial intelligence as a global catastrophic risk in 2008, and Nick Bostrom published Superintelligence in 2014, before the present valuations or companies existed. Geoffrey Hinton left Google so he could warn freely about dangers he sincerely believed the technology posed. The corporate AI industry found an intellectual tradition built in earnest and gave it a new institutional function. The gap between promised returns and attainable technology did not create the theology. It made the theology useful to capital.

⁂

I have spent a great deal of time in these pages arguing about what large language models are. The argument has changed as the systems have improved, but its central conclusion has held. These models are extraordinary representational machines. They compress human symbolic production, translate between domains, generate useful artifacts, and allow a skilled person to move through accumulated knowledge at speeds that would have been unimaginable a decade ago. Their value is real, which is precisely why the financial distortion surrounding them is harder to see.

In Why I’m Betting Against the AGI Hype, I made the engineering case that the scaled-LLM roadmap faces a compound set of architectural barriers. Learning and inference remain separated. Catastrophic forgetting can worsen with scale. Energy requirements remain grotesquely different from biological intelligence. Systems that perform impressively within trained distributions remain brittle when forced to adapt continuously in an open world. The industry needs several independent breakthroughs to arrive together, at production scale, while spending remains economically supportable. My estimate was that scaled LLMs had between a 1 and 15 percent chance of producing AGI over a five-year horizon, dramatically below the probability embedded in market valuations.

The later argument in LLMs Are Universal Translators, Not Universal Thinkers clarified the category mistake underneath those engineering constraints. A model can translate between symbol systems, narrow domains, reorganize human knowledge, and produce fluent language without possessing judgment. It can show us the patterns through which human beings have discussed courage, grief, justice, or responsibility. It cannot experience what any of those words weigh, choose among the values they encode, or bear responsibility for the choice. Translation can improve indefinitely without becoming the conscious activity for which meaning matters.

I sharpened the point in Only Conscious Beings Can Reason Ontically. A language model has representations but no independent cognitive contact with what those representations are about. It has no ontic floor from which to discover that a plausible sequence of symbols has failed to describe the world. This is why hallucination persists even as fluency improves. The model becomes better at producing the surface form of reasoning while remaining categorically unable to know whether the representation it has generated is true because there is no subject inside the system standing in relation to a world.

The three objections describe the same failed roadmap at different levels. The engineering problems reveal an architecture being asked to perform functions it was not built to perform. The philosophical problem explains why adding more representations cannot bridge the gap. The economic problem follows because the industry has capitalized itself on the assumption that quantitative improvement will eventually produce a qualitative transformation. More compute can produce a better model. It does not follow that enough compute produces a mind, yet the entire investment structure depends upon pretending that it does.

---

---

The mistake would be easier to correct if the technology were worthless. A useless product produces a clean failure. Investors stop funding it, executives move on, and the capital finds another story. Large language models are useful enough to keep the story alive while being structurally incapable of delivering the particular future used to justify their cost.

The distinction matters. The internet was real during the dot-com bubble. Railroads were real during the railroad bubbles. Fiber-optic cable was useful during the telecom bubble. A bubble forms when capital is deployed on terms that the underlying technology cannot repay, not when the technology itself is imaginary. The product can transform society while destroying the investors who paid too much for the wrong part of the transformation.

That is the argument I made in The Capital Is Misaligned and the Crash Is Coming. Large language models radically reduce the cost of producing bespoke software. A competent engineer can now translate a business process into functioning software in a fraction of the time previously required. That is a profound technological revolution, but it threatens the software-as-a-service rents of the same hyperscalers and enterprise vendors financing the revolution. If a company can cheaply build a system fitted to its own workflows, the economic logic of paying perpetual per-seat fees for generic software begins to dissolve.

Inference creates a second problem. The current buildout assumes that enormous volumes of model use will continue flowing through hyperscaler datacenters, generating the cloud revenue needed to justify the chips, electricity, land, and financing. But quantization, distillation, specialized silicon, and better local hardware steadily move useful inference onto devices. The consumer who can run a sufficiently capable model on a phone or laptop has little reason to rent every interaction from a remote datacenter. The model remains valuable while the hyperscaler loses the toll.

The industry is building infrastructure for a transformation that attacks its own mechanisms of value capture. Models make software cheaper to produce, which weakens software rents. Models become cheaper to run locally, which weakens cloud-inference rents. Open models and falling inference costs commoditize capabilities that private laboratories need to keep scarce. The technology diffuses value through the economy while the capital structure assumes that a few firms will capture it.

AGI is the bridge across this contradiction. If the companies produce autonomous systems capable of replacing large portions of human labor, conducting science, operating firms, and recursively improving their own successors, then almost any present expenditure can be justified. The prize becomes so large that current revenue ceases to matter. A trillion dollars in infrastructure looks cheap when measured against ownership of the intelligence that will perform a substantial share of the world’s economically valuable cognition.

Without AGI, the arithmetic becomes ordinary again. The companies must earn returns from subscriptions, advertising, enterprise contracts, cloud services, and cost savings. They must explain why increasingly expensive models produce revenue that cheaper models cannot. They must show that customers will pay enough for inference to recover the cost of building and operating the infrastructure. They must confront the possibility that the most valuable use of their technology is to destroy the margins of businesses they already own. The technology is revolutionary. The returns promised to the people financing its present form are not there.

The financing now makes that absence harder to see because the industry has begun financing its own demand. Microsoft invested more than $13 billion in OpenAI, which became one of its largest cloud customers and later committed to purchase $250 billion in Microsoft cloud services. Amazon and Google invested in Anthropic, which in turn committed to using their cloud infrastructure and chips. Bloomberg’s map of these circular deals shows the same pattern across the industry: the supplier funds the customer, the customer spends the capital on the supplier, and the resulting revenue is received as evidence that demand independently exists.

Nvidia has brought the arrangement closest to self-parody. Its proposed investment of up to $100 billion in OpenAI was designed to arrive in tranches as OpenAI deployed Nvidia systems. When that deal failed to close in its original form, Nvidia moved toward a $30 billion equity investment whose proceeds OpenAI was expected to spend primarily on Nvidia chips. The chipmaker would finance one of its largest customers, the customer would return much of the money through chip purchases, Nvidia would report the revenue, and OpenAI’s increased computing capacity would support the valuation that made the investment appear prudent.

AMD devised an even more explicit loop. It issued OpenAI warrants for as many as 160 million AMD shares, vesting them as OpenAI’s purchases scale from one gigawatt of AMD systems to six, with additional vesting tied to AMD’s share price and OpenAI’s technical and commercial milestones. Nvidia, meanwhile, invested $2 billion in CoreWeave, a cloud company built on Nvidia infrastructure that plans to deploy more than five gigawatts of Nvidia-powered capacity. Nvidia has also agreed to buy $6.3 billion of CoreWeave’s cloud services. Capital, procurement, equity appreciation, reported demand, and future capacity have been wired into a single circuit.

These transactions are real, and each may be defensible when considered alone. The circularity lies in what they can no longer prove. Supplier-financed purchases do not establish the same demand as customers spending cash generated by economically productive use of the product. Equity warrants tied to procurement do not establish the same price signal as an unconflicted buyer choosing among vendors. The industry points to booked capacity and chip revenue as proof that the transformation is accelerating, while the financing behind those purchases depends upon the transformation eventually arriving. Telecom equipment makers used vendor financing to produce a similar illusion during the fiber buildout. The equipment was real, the networks were useful, and the demand had been pulled forward by capital that vanished when the promised revenue failed to appear.

⁂

Every additional dollar committed strengthens the perverse incentive. The executives cannot revise the destination without impairing the assets built to reach it. They cannot tell investors that LLMs may become universal translation infrastructure rather than autonomous minds because translation infrastructure does not justify the same valuations. They cannot admit that local inference may consume the profitable edge of cloud inference after announcing datacenter plans premised on explosive demand. They cannot tell markets that the systems will improve incrementally after markets have priced them as the next industrial substrate.

Every participant behaves rationally within the trap. The hyperscaler CEO must invest because refusing to invest would be interpreted as surrendering the future to a rival. The portfolio manager must own the beneficiaries because underweighting them while the story rises is professionally dangerous. The analyst must extend the revenue curve because calling the top too early is more costly than missing it with everyone else. The journalist must cover each model release as a milestone on the road to AGI because the roadmap has become the organizing narrative of the industry.

No conspiracy is required. Everyone receives an incentive to preserve a proposition that nobody is individually positioned to falsify. The capital expenditure becomes evidence that the technology must be consequential because serious companies would not spend so much on an illusion. The rising valuation becomes evidence that informed investors believe the roadmap. The number of researchers working on alignment becomes evidence that the destination is near. Each institution points to the others as confirmation.

The system closes upon itself when it becomes too expensive to stop. Too much has been raised, ordered, built, promised, and priced. A modest reassessment of the technology would now produce an immodest reassessment of the companies. Executives who might once have admitted uncertainty become cornered rats defending the only path by which their previous commitments remain rational. The rhetoric must grow more extreme because the economics have not grown strong enough. Doomerism becomes useful here.

⁂

Telling the public that your product may kill them is terrible marketing. It invites regulation, liability, labor resistance, consumer distrust, and the possibility that a government might respond to an extinction warning by prohibiting the dangerous activity. The doomsday narrative damages the back end of the business strategy, which is one reason I do not believe it began as a carefully designed industry-wide influence operation.

The primary audience is more opportunistic. To capital, the claim that a model may escape human control communicates power. Nobody creates an international safety institute for ordinary software. Nobody compares a customer-service tool to nuclear weapons. Nobody asks whether a better search engine will exterminate the species. The apocalypse is a capability claim carrying the force of a warning.

The investor hears that the scaling curve is real, the transformation is inevitable, and the company building the dangerous system occupies the frontier of the most important technology in history. The possibility of extinction certifies the magnitude of the opportunity. Even the critics who demand that development stop reinforce the premise that the asset possesses world-altering power.

The accelerationist promises that artificial general intelligence will transform everything. The doomer warns that artificial general intelligence may destroy everything. Their moral disagreement is real, but the investment thesis underneath it is identical. Both positions tell capital that scaled computation is approaching an intelligence with powers beyond ordinary economic measurement. The choice between paradise and extinction occurs after inevitability has already been conceded.

I argued in There Is No AI Apocalypse that the boom and the doom perform the same operation. The scaling hypothesis predicts that greater model capability should reduce the auxiliary human apparatus needed to produce competent output. The opposite has occurred. Frontier systems require expanding networks of human annotators, evaluators, domain experts, reinforcement-learning workers, and verifiers. Synthetic data cannot freely replace them without degradation, while self-play succeeds mainly where a reliable reward signal already exists. The field’s labor economics reveal increasing dependence on human intelligence at the same time its public story promises machine independence from it.

The doomsday story absorbs this failure. Growing human scaffolding becomes a temporary alignment requirement. Hallucination becomes evidence of an alien intelligence learning to deceive. Brittleness becomes unpredictability. A benchmark success demonstrates emergence, while a benchmark failure demonstrates that the system remains inscrutable. Every outcome is translated into a reason to finance the next scaling run.

The bull case says the model is becoming a god. The bear case says the model is becoming a demon. Nobody in the valuation conversation is permitted to say it is becoming a better tool.

⁂

The money now surrounding the doomsday belief is real and substantially disclosed. Anthropic, Google, Microsoft, and OpenAI established a $10 million AI Safety Fund for research into frontier-model risks. It has distributed more than $5 million to eleven grantees. OpenAI committed £5.6 million to a British alignment program supported by Microsoft, AWS, and Anthropic, bringing the initiative’s total funding above £27 million.

None of this proves that researchers are being paid to reach conclusions dictated by the companies. Catastrophic-risk research is a legitimate field of inquiry, and many of its participants plainly hold their beliefs with great sincerity. Institutional influence rarely requires the vulgarity of a script. Money decides which questions support careers, conferences, institutes, journals, fellowships, and policy organizations. It allows one account of the technology to become durable enough that the existence of the field begins to serve as evidence for the premise that created it.

Anthropic is the clearest example because its executives appear to believe the warnings they issue. Dario Amodei told the Senate that advanced AI could pose “extraordinarily grave threats” to American national security and advocated mandatory testing before deployment. The company has supported increasingly strong state legislation while arguing that voluntary transparency is insufficient. Anthropic made two $20 million corporate contributions to Public First Action, a nonprofit policy organization, with the contributions barred from use to influence candidates. Separately, Amodei personally gave $1 million and five other Anthropic employees gave more than $2 million in total to the affiliated Public First super PAC.

I take Amodei’s concern seriously. Sincerity is part of what makes the system effective. Capital selects leaders whose convictions support continued investment, gives them the resources to build institutions around those convictions, and then points to their honest alarm as independent confirmation of the capability premise. The warning makes continued scaling appear responsible, the safety apparatus makes the company appear indispensable, and the company’s valuation gives the warning an audience. At no point must anyone consciously lie. Perverse incentives are more durable when they operate through honest people.

⁂

Regulatory capture follows the valuation story rather than causing it. Once companies have described their systems as potentially apocalyptic, they need a political answer to the apocalypse. OpenAI’s leadership moved from “the possibility of existential risk” to a proposal for an international authority resembling the International Atomic Energy Agency. Systems above specified capability or compute thresholds would face inspections, audits, safety testing, and deployment restrictions. Microsoft endorsed licensing highly capable models and the datacenters used to develop and deploy them.

The companies define scale as the path to dangerous intelligence. Scale then becomes the threshold for regulation. The firms possessing the greatest concentrations of chips, electricity, datacenters, lawyers, safety researchers, and political access are naturally best equipped to comply with the rules. The European Union has already adopted a systemic-risk presumption tied to a training threshold of 10 to the 25th floating-point operations, a technical boundary that also happens to describe a class of firms.

This is the doomsday moat in its regulatory form. The private accumulation of scale creates the alleged danger. The danger justifies public restrictions. The restrictions certify the institutions that achieved private scale as the only responsible custodians of the technology. A startup can comply with ordinary consumer-protection and liability law. It cannot easily maintain a hundred-person alignment organization, negotiate national-security agreements, run constant catastrophic-capability evaluations, and satisfy an international licensing authority.

The industry is not unified around this strategy. OpenAI opposed California’s SB 1047, while Meta resisted binding safety rules and declined to join the famous extinction statement. Build American AI, a dark-money group linked to a super PAC backed by OpenAI president Greg Brockman and other technology investors, offered creators $5,000 per TikTok to deliver fear-based messages about China. At least one influencer labeled the post as an advertisement without disclosing the organization that paid for it. This was an anti-regulatory campaign, not a doomer campaign.

Opposition to regulation does not itself prove the existence of a doomsday moat. It proves that the industry is not operating through a single strategy, much less a coordinated conspiracy. The narrower claim is that catastrophic-risk rhetoric acquires a moat function when it is joined to licensing, compute thresholds, and compliance systems whose fixed costs established firms can absorb more easily than challengers. A faction organized around defeating those rules is evidence of a different political strategy, not additional evidence for the same one.

The claim must admit the possibility of being wrong. A frontier laboratory that accepts binding rules imposing substantial costs on its current systems, lowers rather than raises barriers for smaller competitors, and surrenders market power without receiving a compensating procurement advantage or government privilege would count against the moat. So would a safety position that produced a durable loss of valuation or market share without certifying the company’s capabilities or political importance.

The technical and economic argument has a clock as well. By July 2031, frontier models should still require expanding human evaluation and scaffolding to operate reliably in open-ended domains; useful inference should continue migrating toward cheaper and more local hardware; and revenue attributable to model use should remain insufficient to justify the industry’s infrastructure commitments at ordinary market rates. If scaled models instead learn continuously, operate autonomously across novel domains while reducing their dependence on human verification, and generate returns commensurate with the capital invested, then the central claim of this essay will have failed. Let the record score it.

⁂

OpenAI’s proposed government stake converts technological determinism into a public financial interest. The company has accumulated private capital by arguing that artificial general intelligence is approaching. It has spent against a future in which intelligence becomes a service delivered from infrastructure it controls. The technology has improved, but the autonomous economic agents needed to justify the buildout have not arrived. Useful inference is becoming cheaper, more local, and more commoditized, while the cost of remaining at the proclaimed frontier continues to rise.

OpenAI cannot retreat to the claim that it has built a valuable tool. Valuable tools are governed by ordinary expectations about profit, competition, liability, and return on invested capital. OpenAI has been valued as the institution through which history will pass. If that claim weakens, the company does not merely lose prestige. The financing logic supporting its infrastructure, partnerships, and prospective public offering begins to fail.

So the company offers the government a piece of the prophecy. The public-wealth language asks Americans to imagine themselves as beneficiaries of gains that remain speculative. The equity stake asks Washington to treat the continuation of OpenAI’s valuation as a national interest. The state is invited to share the upside before the market has determined whether the upside exists, and by accepting the invitation it acquires a reason to prevent the downside.

An investor in both OpenAI and Anthropic told Axios the proposal was a political move to gain favor with the administration rather than a meaningful mechanism for sharing wealth with the public. The tribute reading is correct, but it understates the elegance of the proposal. A government shareholder does more than favor the company. It becomes institutionally entangled with the company’s survival. A collapse in valuation becomes a loss on the public balance sheet. Regulation that impairs OpenAI becomes regulation that destroys an asset notionally owned by citizens. The government does not have to transfer cash today. It only has to acquire an interest that makes transferring cash tomorrow appear responsible. This is how a speculative private project becomes too publicly owned to fail.

⁂

The executives are now trapped between audiences whose requirements cannot be reconciled. They must tell investors that the systems are becoming powerful enough to replace large categories of labor, automate science, operate firms, and perhaps escape human control. They must tell customers that the products are reliable enough to purchase and safe enough to place inside consequential workflows. They must tell governments that the systems are dangerous enough to require the companies’ expertise but manageable enough to remain legal. They must tell courts that harms were unforeseeable after spending years warning that the technology could alter civilization.

The ontology changes with the audience. The machine becomes an uncontrollable alien when capital needs awe, a dependable assistant when customers need reassurance, a strategic national asset when government threatens restraint, and merely a neutral tool when liability appears. The contradictions are no longer rhetorical accidents. They are the different stories required to sustain a business whose technology, revenue model, capital requirements, and political position no longer fit together.

That is why the executives sound increasingly frantic. They cannot abandon inevitability without threatening investment. They cannot continue speaking of extinction without threatening their freedom to operate. They cannot admit that the roadmap may be wrong because uncertainty is fatal to a valuation premised upon destiny. The public sees inconsistent claims because consistency would require one of the audiences to hear the truth.

The models are useful and the capital is misaligned. The systems will change software without becoming conscious sovereigns. They will create enormous social value while failing to return that value to the companies in proportion to what those companies have spent. The infrastructure will remain useful after the valuation story breaks, just as fiber survived the telecom collapse and rail survived the railroad bubbles. The people holding the inflated claims will absorb the difference.

The industry has begun trying to transfer that difference to the public before it is recognized. The doomsday narrative makes the companies appear too consequential to ignore. The China narrative makes them appear too strategic to constrain. The government stake would make them too publicly entangled to permit to fail. We should refuse the sequence before its final step becomes inevitable merely because the industry says it is.

⁂

The politics become simpler once we reject the metaphysics embedded in the investment pitch. We do not need to believe that a language model will become a god before regulating the firms that build it. We do not need to solve alignment before enforcing antitrust law, protecting workers, assigning liability, taxing wealth, defending privacy, or governing the enormous demands datacenters place upon public infrastructure. Those are questions about present power exercised by identifiable human beings.

The doomsday story points beyond the horizon because the horizon is where the returns have been placed. It asks us to fear a machine that may someday escape its creators while the creators attempt to escape the ordinary disciplines of markets and democratic law. It turns a failed roadmap into a national emergency, a speculative valuation into a strategic asset, and the people who made the bad bet into the indispensable authorities on how the public must absorb it.

The technology will remain. It will become cheaper, more useful, more local, and more deeply integrated into ordinary software. That future does not require the companies currently building the largest datacenters to own the intelligence of the world. It requires us to understand what the systems actually do, where their value actually lands, and which institutions are trying to make their private exposure into our public obligation.

The doomsday belief may be sincere. Its function is now unmistakable. It tells capital that the capability is real, tells government that the company is indispensable, and tells the public that there is no time to deliberate about who should bear the cost. OpenAI is offering Washington a share of the future it promised. The offer is arriving because the future cannot pay the bill.

---

Go Deeper into the Circus

---