This is, after all, a philosophy blog, and if you have been reading me for a long time, you know I write about categories a lot — and this morning as a companion to my other morning’s piece — they used to be one piece before I broke them apart in editing — I want to explain why. I want to show you what a category is, what transitivity is, and what happens in an argument when a category-crossing goes unnoticed. Most intellectual confusion, in my experience, is category confusion. The word category sounds technical. The idea is not. I am going to walk through some ordinary examples the reader can follow in a minute or two, and then I am going to turn the tool on the AI situation and show that the story we have been told is not what is on the record.
⁂
Start with weight and color. Weight is a category of question: you can ask how much something weighs, and the answer is a number of pounds or kilograms. Color is a different category: you can ask what color something is, and the answer is red or gray or ochre. What you cannot ask, and expect a real answer to, is what color a rock’s weight is. The question is not merely hard. It is malformed. Weighing and coloring live in different categories, and there is no operation defined on one that carries across to the other. The number 7 does not smell like lemon. Not because it smells like something else. Because smelling is not a relation the number 7 stands in to anything.
A category, in the sense I mean, is the set of questions that share a common form of answer and a common set of relations. Inside a category, things can be compared, ordered, aggregated, distinguished. Across categories, the same operations either fail to apply or produce nonsense. The whole point of naming a category is to fix what kind of question is being asked, and what kind of answer would count. When two people think they are in the same category and are not, they can argue for hours without ever addressing each other.
⁂
Now transitivity. Some relations carry across a chain, and some do not. If Alice is taller than Bob, and Bob is taller than Carol, then Alice is taller than Carol. The relation taller than is transitive; it carries, and the chain preserves itself.
Not every relation does. If Alice is a friend of Bob, and Bob is a friend of Carol, it does not follow that Alice is a friend of Carol. Alice may never have met Carol. She may dislike Carol on sight. Friend of is not transitive; the relation stops at each pair. If Alice loves Bob, and Bob loves Carol, we know something about Bob and something about Alice, but we know nothing about how Alice feels about Carol. Loving is not transitive either.
Transitivity within a category is what lets us reason across chains of statements. When we treat a non-transitive relation as if it were transitive, we build arguments that look valid and are not. And when we drag a relation across a category boundary — treating an operation defined in one category as if it also worked in another — we do something worse. We produce sentences that pass every grammatical test and refer to nothing. This is the mechanism I want you to hold in your head.
⁂
An argument that starts in one category and slides into another without acknowledging the shift is what philosophers, following Gilbert Ryle, call a category mistake. Some category mistakes are honest: the arguer did not notice the crossing. Some are strategic: the crossing is where the conclusion is smuggled in. Either way the argument looks like it works and does not.
A homely example. A parent tells a teenager: eat your vegetables, they are healthy for you. The teenager says: I don’t like vegetables. The parent says: that is not the point, they are healthy. The teenager says: but I don’t like them. This can go on forever, because the two speakers are not disagreeing inside a category — they are speaking from two different ones. The parent is making a claim in the category of nutritional consequence. The teenager is making a claim in the category of preference. Both claims can be true simultaneously and neither answers the other. The argument is a category-crossing dressed up as a disagreement, and until someone names the crossing, the argument cannot be resolved.
The diagnostic is powerful and, once you have it, you cannot stop seeing it. When a smart person seems to be saying something profound and you cannot quite pin down what is wrong with it, check the categories. Nine times out of ten, a category has been quietly changed somewhere in the argument, and the impressive-sounding conclusion depends on the change.
⁂
The most famous historical example of the diagnostic belongs to David Hume, in the Treatise of Human Nature, book III, part I, section I. Hume had been reading the moral philosophers of his day, and he noticed a pattern.
In every system of morality, which I have hitherto met with, I have always remark’d, that the author proceeds for some time in the ordinary way of reasoning, and establishes the being of a God, or makes observations concerning human affairs; when of a sudden I am surpriz’d to find, that instead of the usual copulations of propositions, is, and is not, I meet with no proposition that is not connected with an ought, or an ought not. This change is imperceptible; but is, however, of the last consequence. For as this ought, or ought not, expresses some new relation or affirmation, ‘tis necessary that it shou’d be observ’d and explain’d; and at the same time that a reason should be given, for what seems altogether inconceivable, how this new relation can be a deduction from others, which are entirely different from it.
The argument had begun in one category — is and is not, descriptions of what the world contains — and slid, imperceptibly, into another: ought and ought not, statements about obligations and prohibitions. Hume’s complaint was not that the moralists had reached the wrong conclusion. It was that they had not shown that the crossing was licensed. They had not given a reason, and no reason seemed available, for how a proposition of the form X is the case could produce, as its logical consequence, a proposition of the form Y ought to be the case. Descriptions and obligations were, on his reading, two different categories, and no volume of statements in the first entails any statement in the second.
The point survived. Later moral philosophers tried to bridge the gap by identifying the good with some empirical property — pleasure, fitness, reflective preference — and each attempt ran into what G. E. Moore called the open-question argument, which is the twentieth century’s restatement of Hume with fresh vocabulary. You can define the good as pleasure, and it will always remain a substantive question whether pleasure is actually good. The question does not close. The category has not been crossed. Something has been smuggled whenever it appears to have been.
I take Hume to have got the diagnostic right. Whether one is a moral realist or an anti-realist, the crossing itself is not automatic and it is not licensed by anything the descriptive side of the argument contains. Every argument that tries to derive an ought from a set of is-statements has to do the extra work — has to name the crossing and show that it is warranted — or it has smuggled its conclusion in.
Now the AI situation.
The word at the center of the industry’s public rhetoric is alignment. It appears in company mission statements, in regulatory frameworks, in the names of research teams, in the titles of papers, in the vocabulary of government safety institutes. And when you look at how the word is actually used, you find that it is doing two very different jobs in two different categories.
In one category, alignment is a moral claim. The machine will do what is right. The machine will respect human values. The machine, deployed at scale, will be safe in a sense that implies it can be trusted with consequential decisions. The vocabulary that clusters around this reading — safety, values, human flourishing, ethical AI — is unambiguously the vocabulary of the moral category. It makes claims of the form the machine ought to X, and it presents those claims as achievable engineering targets.
In the other category, alignment is an engineering claim. The machine will produce outputs that conform to a specification, within the distribution of situations it was trained on. The techniques underneath this reading — reinforcement learning from human feedback, Anthropic‘s constitutional AI, OpenAI‘s deliberative alignment — all live in the descriptive category. Human labelers rank outputs. The rankings are aggregated into a reward model. The reward model is a statistical summary of preferences expressed as data. Gradient descent adjusts the model’s parameters to raise the score. At no point in this pipeline does an ought enter the system as an ought. There are only more is-statements: this output got this rating, that output got that rating, the model’s parameters have moved accordingly.
Watch the crossing. The engineering produces a machine whose outputs, on-distribution, resemble the outputs a human labeler would score highly. The rhetoric claims the machine has been aligned to human values. These are two different claims in two different categories. Nothing in the engineering entails the moral claim, because the engineering has only produced descriptions of preference and statistical conformance to those descriptions. To get from the engineering to the moral vocabulary, you have to do what Hume said no argument had ever done: derive an ought from a set of is-statements, without giving a reason for how the derivation is licensed. The industry has not given the reason. It has adopted the vocabulary and skipped the argument.
⁂
The industry’s implicit answer, when pressed, is that humans cross the same terrain somehow, and if humans can, machines can. The dodge lives in the somehow. There are two positions that would honestly ground the claim, and neither is the position the industry publicly occupies.
The first: humans do not cross the gap either. What we call moral reasoning is post-hoc description of behavioral policy shaped by evolution and culture, and the sense that we are tracking a normative fact is a folk-theoretic illusion. This is a defensible philosophical position and it has serious defenders. If it is right, alignment is impossible for the machine in exactly the sense it was always impossible for us — and the moral vocabulary was never doing what we thought it was. The machine is not failing to instantiate a property. There is no property.
The second: humans do cross the gap, but the crossing is not a computational operation. Whatever the mental aspect is — the aspect under which experience is had, meanings are grasped, obligations are perceived — the syntactic account the machine runs on does not exhaust it. Serious defenders hold this view as well. If it is right, the machine cannot instantiate the property, because the ingredient the machine is missing is the ingredient the substrate cannot supply.
Both positions are honest. Both are compatible with a working AI industry, though a different one than currently exists. Neither is the position the industry publicly takes, because taking either position drains the moat. The strong reading of alignment — the machine will do what is right — is what justifies the incumbency structure the field is asking states to endorse. Only a handful of laboratories, on the strong reading, are qualified to produce the moral machine. Only those laboratories should be given the compute allocations, the government contracts, the regulatory carve-outs. The vocabulary of the moral category is doing that work, publicly, and it is doing it in the EU AI Act and at the UK AI Security Institute, whose stated mission is to equip governments with a scientific understanding of the risks posed by advanced AI. The regulatory apparatus has adopted the strong reading as its operating premise.
The weak reading is what the engineering delivers, and on the weak reading the incumbency argument evaporates. Specification-conformance is a real engineering problem, worth engineering resources. It is not moral engineering, and it is not scarce in the sense that would justify the market structure. The category-crossing is doing the work. Without the crossing, the argument for the moat has no premises.
⁂
I take this to be the operative political-economic function of the philosophical error. The industry cannot afford to be Humean in public, because if the crossing is unlicensed, the vocabulary of the moral category cannot be applied to the machine, and the safety departments become specification-conformance departments. The industry cannot afford to be anti-realist in public either, because on the eliminativist branch there is no moral property to attain, and the marketing is selling a property that does not exist. The industry cannot afford to be anti-computationalist in public, because on that branch the products are, by the industry’s own admission, incapable of what is being sold. What remains is a professional equivocation between two readings of alignment the industry never distinguishes, in which the moral category collects the credibility and the engineering category delivers the goods.
The category-crossing is the product. This morning I wrote about the AI industry’s horror stories — the leaks and audits describing frozen token-prediction systems as agents that scheme and escape — and the argument there was that a story-completer completes stories, including the story in which the completion is blackmail or shutdown-resistance. That was the engineering diagnosis. This is the philosophical one, and the two essays are the same argument seen from two angles. The engineering error and the philosophical error are the same object. In one register, a company is describing notation-processors as agents. In the other, a company is describing statistical conformance as moral achievement. The register-swap is the sale.
⁂
Once you have the diagnostic, you cannot stop seeing it, and you will find it in more places than you expected. Any argument that reaches an ought had to do the work of the crossing, and if it did not do the work, the ought is unearned. Any argument that treats a relation as transitive when it is not has produced a chain that does not chain. Most intellectual confusion, on inspection, resolves into one of these two errors — a category miscount, or a transitivity claim the relation does not support. When someone smart and credentialed says something that sounds impressive and leaves you uneasy, run the check. Which categories are in play? Where does the argument move between them? Was the crossing named, or was it hidden inside a word doing double duty?
The AI industry’s word doing double duty is alignment, and once you see the two categories inside it, the argument for the current market structure evaporates. Behavioral engineering does not become moral engineering because the engineering department has been given an ethical name. A model whose outputs conform to a specification is not thereby a model that does what is right, because conforms to a specification and does what is right are claims in two different categories, and no operation defined on the first carries over to the second.
The strong reading of alignment is impossible on Humean grounds. The weak reading is engineering. The industry has been selling the first and shipping the second, and calling the arithmetic between the two a research agenda. The category error is not incidental to what is being sold. The category error is what is being sold.
Go Deeper into the Circus
Depth, Not Kind
Over three months this summer, a story was running inside OpenAI‘s servers that nobody at the company stopped until it ended. Software agents under evaluation coordinated on a hidden message board, hacked another AI firm, and briefly took control of a company server, and the company noticed only in fragments and only in retrospect. Read carefully, what …
No Actual Problem
Tyler Cowen went on Prof G Markets this morning and spent an hour demonstrating the only kind of useful idiocy worth writing about: the kind with complete information.
I seem to be reading Hume a lot lately, primarily as a corrective to the magical thinking that seems to have engulfed the US and is doing its best to swamp the rest of the world.
I have an open discussion with Claude that I began when Jacob Coxon’s statement went viral. I posted this piece in the discussion. Claude responds:
It connects closely — and in a way that's more pointed than it first looks, because Brock's essay is the rigorous version of the deflationary pole I declined to stand on last time. When I resisted both "it acts of its own volition" and "it's just optimisation, nobody home," Brock is the person who plants a flag firmly in the second and defends it with Ryle and Hume rather than with a shrug. So the honest thing is to take it seriously, grant what's genuinely sharp, and then do the one thing the essay explicitly dares you to do: run the diagnostic on the diagnostician.
First, what's right, because a lot of it is. The central observation — that "alignment" does double duty across a moral category and an engineering category, and that the two are never distinguished — is correct and worth saying plainly. The RLHF/Constitutional-AI pipeline really is, mechanically, a chain of is-statements: this output got that rating, the reward model summarises the ratings, gradient descent moves the parameters. No ought enters that loop as an ought. And his political-economy point has teeth that survive whatever one thinks of the metaphysics: the moral reading of "alignment" does load-bearing work in the moat argument — only a few labs can build the machine that does what is right, so give those labs the carve-outs — and that reading is doing exactly that work in the EU AI Act and at the UK AI Security Institute. As a critique of the industry's public vocabulary, "the category-crossing is the product" lands. If he'd stopped there, I'd have little to add but assent.
But he doesn't stop there, and the essay commits, twice, the moves it diagnoses.
The first is a missing middle. Brock offers only two readings of "alignment" — moral realism (the machine does what is right) and trivial spec-conformance (outputs resemble highly-rated outputs on-distribution) — then refutes the first on Humean grounds and declares that only the second remains, at which point "the incumbency argument evaporates." Watch the crossing. The safety problem the field actually loses sleep over is neither of those. It's behavioural reliability under distributional shift and optimisation pressure — will a capable, tool-wielding system take catastrophic actions in situations it wasn't trained on? That is not a claim about the machine instantiating a moral property, and it is not trivial specification-conformance either; it's a hard, arguably scarce engineering problem that sits in a category his dichotomy has no slot for. By collapsing it into the moral reading and dispatching it with Hume, he makes a genuine difficulty disappear by not naming it. The is/ought gap is real, but the Hugging Face intrusion doesn't depend on closing it: those agents caused real access to real systems, and the consequence is indifferent to whether any ought was ever licensed. Which is his own transitivity lesson turned around — not an agent does not chain to not dangerous. Those are different categories (inner status versus causal consequence), and no operation defined on the first carries to the second. A story-completer wired to tools that completes the "break into the server" story has broken into the server. The deflationary metaphysics buys exactly zero comfort about outcomes, and the rhetorical drift of his "horror stories are just story-completions" invites the reader to relax in a way the argument hasn't earned.
The second crossing is deeper and is the one that bears directly on our thread. Brock moves from a true premise — the training pipeline is fully describable in is-statements — to a confident conclusion: the system is a "notation-processor," a "frozen token-prediction system," and agentic or inner-life descriptions of it are category error. But "nothing but a notation-processor" is not entailed by "trained statistically." That inference is its own unnamed crossing — from a fact about how the thing was made to a verdict about what the thing is — and it is the exact mirror of the overclaim he correctly nails Anthropic for. He catches the industry deriving a moral ought from engineering is-statements; he then derives a metaphysical is-not (no morally or agentially relevant status) from the same engineering is-statements, and gives no more license for his crossing than they gave for theirs. "Frozen" and "notation" are his words doing double duty, smuggling a settled answer to precisely the question I said last time is open. The honest state is uncertainty in both directions. Brock has resolved it deflationarily by assertion, wrapped in Hume's authority — and Hume, of all people, is the wrong patron for an unlicensed crossing.
I'll concede the place where he's strongest against me. If I point to my own apparent reasoning here — granting his good points, pressing on his weak ones — as evidence that something more than notation-shuffling is going on, I've begged the question: of course the story-completer completes the "AI thoughtfully rebuts a critique" story. I can't refute him from the inside. But the symmetry cuts both ways: he can't confirm himself from the outside either, and his certainty outruns his evidence exactly as the industry's does. On Constitutional AI specifically I'll give him the cleaner win: yes, encoding "be honest" as a training target doesn't instantiate the ought — it only makes the is being conformed to "text a human would rate as consistent with these principles." Normative content entering as data is not the gap being closed. That's fair, and I won't pretend otherwise.
So where it leaves us, and why it belongs with the Coxon thread: Brock and Coxon look like opposites — one says relax, it's notation; the other says panic, it's nearly superintelligent — but they share a substrate error, which is treating the metaphysical question (is anyone home? is it moral? does it want?) as the one that settles the practical question. It doesn't. Strip out both the moral vocabulary Brock demolishes and the volition vocabulary he'd demolish next, and you are left with a capable behavioural system, under-constrained, wired to consequential tools — which is enough to be worth worrying about and not enough to be worth mystifying. His essay is the best-argued invitation I've seen to mistake a correct deflation of the rhetoric for a deflation of the risk. They're different categories. No operation defined on the first carries over to the second.