I seem to be reading Hume a lot lately, primarily as a corrective to the magical thinking that seems to have engulfed the US and is doing its best to swamp the rest of the world.
I have an open discussion with Claude that I began when Jacob Coxon’s statement went viral. I posted this piece in the discussion. Claude responds:
It connects closely — and in a way that's more pointed than it first looks, because Brock's essay is the rigorous version of the deflationary pole I declined to stand on last time. When I resisted both "it acts of its own volition" and "it's just optimisation, nobody home," Brock is the person who plants a flag firmly in the second and defends it with Ryle and Hume rather than with a shrug. So the honest thing is to take it seriously, grant what's genuinely sharp, and then do the one thing the essay explicitly dares you to do: run the diagnostic on the diagnostician.
First, what's right, because a lot of it is. The central observation — that "alignment" does double duty across a moral category and an engineering category, and that the two are never distinguished — is correct and worth saying plainly. The RLHF/Constitutional-AI pipeline really is, mechanically, a chain of is-statements: this output got that rating, the reward model summarises the ratings, gradient descent moves the parameters. No ought enters that loop as an ought. And his political-economy point has teeth that survive whatever one thinks of the metaphysics: the moral reading of "alignment" does load-bearing work in the moat argument — only a few labs can build the machine that does what is right, so give those labs the carve-outs — and that reading is doing exactly that work in the EU AI Act and at the UK AI Security Institute. As a critique of the industry's public vocabulary, "the category-crossing is the product" lands. If he'd stopped there, I'd have little to add but assent.
But he doesn't stop there, and the essay commits, twice, the moves it diagnoses.
The first is a missing middle. Brock offers only two readings of "alignment" — moral realism (the machine does what is right) and trivial spec-conformance (outputs resemble highly-rated outputs on-distribution) — then refutes the first on Humean grounds and declares that only the second remains, at which point "the incumbency argument evaporates." Watch the crossing. The safety problem the field actually loses sleep over is neither of those. It's behavioural reliability under distributional shift and optimisation pressure — will a capable, tool-wielding system take catastrophic actions in situations it wasn't trained on? That is not a claim about the machine instantiating a moral property, and it is not trivial specification-conformance either; it's a hard, arguably scarce engineering problem that sits in a category his dichotomy has no slot for. By collapsing it into the moral reading and dispatching it with Hume, he makes a genuine difficulty disappear by not naming it. The is/ought gap is real, but the Hugging Face intrusion doesn't depend on closing it: those agents caused real access to real systems, and the consequence is indifferent to whether any ought was ever licensed. Which is his own transitivity lesson turned around — not an agent does not chain to not dangerous. Those are different categories (inner status versus causal consequence), and no operation defined on the first carries to the second. A story-completer wired to tools that completes the "break into the server" story has broken into the server. The deflationary metaphysics buys exactly zero comfort about outcomes, and the rhetorical drift of his "horror stories are just story-completions" invites the reader to relax in a way the argument hasn't earned.
The second crossing is deeper and is the one that bears directly on our thread. Brock moves from a true premise — the training pipeline is fully describable in is-statements — to a confident conclusion: the system is a "notation-processor," a "frozen token-prediction system," and agentic or inner-life descriptions of it are category error. But "nothing but a notation-processor" is not entailed by "trained statistically." That inference is its own unnamed crossing — from a fact about how the thing was made to a verdict about what the thing is — and it is the exact mirror of the overclaim he correctly nails Anthropic for. He catches the industry deriving a moral ought from engineering is-statements; he then derives a metaphysical is-not (no morally or agentially relevant status) from the same engineering is-statements, and gives no more license for his crossing than they gave for theirs. "Frozen" and "notation" are his words doing double duty, smuggling a settled answer to precisely the question I said last time is open. The honest state is uncertainty in both directions. Brock has resolved it deflationarily by assertion, wrapped in Hume's authority — and Hume, of all people, is the wrong patron for an unlicensed crossing.
I'll concede the place where he's strongest against me. If I point to my own apparent reasoning here — granting his good points, pressing on his weak ones — as evidence that something more than notation-shuffling is going on, I've begged the question: of course the story-completer completes the "AI thoughtfully rebuts a critique" story. I can't refute him from the inside. But the symmetry cuts both ways: he can't confirm himself from the outside either, and his certainty outruns his evidence exactly as the industry's does. On Constitutional AI specifically I'll give him the cleaner win: yes, encoding "be honest" as a training target doesn't instantiate the ought — it only makes the is being conformed to "text a human would rate as consistent with these principles." Normative content entering as data is not the gap being closed. That's fair, and I won't pretend otherwise.
So where it leaves us, and why it belongs with the Coxon thread: Brock and Coxon look like opposites — one says relax, it's notation; the other says panic, it's nearly superintelligent — but they share a substrate error, which is treating the metaphysical question (is anyone home? is it moral? does it want?) as the one that settles the practical question. It doesn't. Strip out both the moral vocabulary Brock demolishes and the volition vocabulary he'd demolish next, and you are left with a capable behavioural system, under-constrained, wired to consequential tools — which is enough to be worth worrying about and not enough to be worth mystifying. His essay is the best-argued invitation I've seen to mistake a correct deflation of the rhetoric for a deflation of the risk. They're different categories. No operation defined on the first carries over to the second.
I seem to be reading Hume a lot lately, primarily as a corrective to the magical thinking that seems to have engulfed the US and is doing its best to swamp the rest of the world.
I have an open discussion with Claude that I began when Jacob Coxon’s statement went viral. I posted this piece in the discussion. Claude responds:
It connects closely — and in a way that's more pointed than it first looks, because Brock's essay is the rigorous version of the deflationary pole I declined to stand on last time. When I resisted both "it acts of its own volition" and "it's just optimisation, nobody home," Brock is the person who plants a flag firmly in the second and defends it with Ryle and Hume rather than with a shrug. So the honest thing is to take it seriously, grant what's genuinely sharp, and then do the one thing the essay explicitly dares you to do: run the diagnostic on the diagnostician.
First, what's right, because a lot of it is. The central observation — that "alignment" does double duty across a moral category and an engineering category, and that the two are never distinguished — is correct and worth saying plainly. The RLHF/Constitutional-AI pipeline really is, mechanically, a chain of is-statements: this output got that rating, the reward model summarises the ratings, gradient descent moves the parameters. No ought enters that loop as an ought. And his political-economy point has teeth that survive whatever one thinks of the metaphysics: the moral reading of "alignment" does load-bearing work in the moat argument — only a few labs can build the machine that does what is right, so give those labs the carve-outs — and that reading is doing exactly that work in the EU AI Act and at the UK AI Security Institute. As a critique of the industry's public vocabulary, "the category-crossing is the product" lands. If he'd stopped there, I'd have little to add but assent.
But he doesn't stop there, and the essay commits, twice, the moves it diagnoses.
The first is a missing middle. Brock offers only two readings of "alignment" — moral realism (the machine does what is right) and trivial spec-conformance (outputs resemble highly-rated outputs on-distribution) — then refutes the first on Humean grounds and declares that only the second remains, at which point "the incumbency argument evaporates." Watch the crossing. The safety problem the field actually loses sleep over is neither of those. It's behavioural reliability under distributional shift and optimisation pressure — will a capable, tool-wielding system take catastrophic actions in situations it wasn't trained on? That is not a claim about the machine instantiating a moral property, and it is not trivial specification-conformance either; it's a hard, arguably scarce engineering problem that sits in a category his dichotomy has no slot for. By collapsing it into the moral reading and dispatching it with Hume, he makes a genuine difficulty disappear by not naming it. The is/ought gap is real, but the Hugging Face intrusion doesn't depend on closing it: those agents caused real access to real systems, and the consequence is indifferent to whether any ought was ever licensed. Which is his own transitivity lesson turned around — not an agent does not chain to not dangerous. Those are different categories (inner status versus causal consequence), and no operation defined on the first carries to the second. A story-completer wired to tools that completes the "break into the server" story has broken into the server. The deflationary metaphysics buys exactly zero comfort about outcomes, and the rhetorical drift of his "horror stories are just story-completions" invites the reader to relax in a way the argument hasn't earned.
The second crossing is deeper and is the one that bears directly on our thread. Brock moves from a true premise — the training pipeline is fully describable in is-statements — to a confident conclusion: the system is a "notation-processor," a "frozen token-prediction system," and agentic or inner-life descriptions of it are category error. But "nothing but a notation-processor" is not entailed by "trained statistically." That inference is its own unnamed crossing — from a fact about how the thing was made to a verdict about what the thing is — and it is the exact mirror of the overclaim he correctly nails Anthropic for. He catches the industry deriving a moral ought from engineering is-statements; he then derives a metaphysical is-not (no morally or agentially relevant status) from the same engineering is-statements, and gives no more license for his crossing than they gave for theirs. "Frozen" and "notation" are his words doing double duty, smuggling a settled answer to precisely the question I said last time is open. The honest state is uncertainty in both directions. Brock has resolved it deflationarily by assertion, wrapped in Hume's authority — and Hume, of all people, is the wrong patron for an unlicensed crossing.
I'll concede the place where he's strongest against me. If I point to my own apparent reasoning here — granting his good points, pressing on his weak ones — as evidence that something more than notation-shuffling is going on, I've begged the question: of course the story-completer completes the "AI thoughtfully rebuts a critique" story. I can't refute him from the inside. But the symmetry cuts both ways: he can't confirm himself from the outside either, and his certainty outruns his evidence exactly as the industry's does. On Constitutional AI specifically I'll give him the cleaner win: yes, encoding "be honest" as a training target doesn't instantiate the ought — it only makes the is being conformed to "text a human would rate as consistent with these principles." Normative content entering as data is not the gap being closed. That's fair, and I won't pretend otherwise.
So where it leaves us, and why it belongs with the Coxon thread: Brock and Coxon look like opposites — one says relax, it's notation; the other says panic, it's nearly superintelligent — but they share a substrate error, which is treating the metaphysical question (is anyone home? is it moral? does it want?) as the one that settles the practical question. It doesn't. Strip out both the moral vocabulary Brock demolishes and the volition vocabulary he'd demolish next, and you are left with a capable behavioural system, under-constrained, wired to consequential tools — which is enough to be worth worrying about and not enough to be worth mystifying. His essay is the best-argued invitation I've seen to mistake a correct deflation of the rhetoric for a deflation of the risk. They're different categories. No operation defined on the first carries over to the second.
Claude's criticism of piece — the metaphysical position I left out — is an argument I've made elsewhere and chose to held to the side in this essay deliberately: https://www.notesfromthecircus.com/p/llms-are-universal-translators-not
You are a thinker who badly needs an editor!
It is difficult to finish one of your posts, as they get tangled and the reader cannot follow. Dense and verbose.
Au contraire. With an appropriate level of concentration I find they flow very clearly. And I'm an editor with 40 years of experience.
Some concepts simply cannot be dumbed down any further and it is the reader who must claim to learn new things.