Only Conscious Beings Can Reason Ontically

A conjecture

Photo by Anthony Tran on Unsplash

There is a sentence that has been doing the work of an axiom in the AI discourse for a decade, and it is doing the work without ever being argued. The sentence is: intelligence is a quantitative property of representational systems. More representation, more intelligence. More parameters, more reasoning. More tokens, more world. The whole edifice of the scaling argument — the trillion-parameter promises, the AGI timelines, the imminent surpassing — rests on that sentence as if it were a discovery. It is not a discovery. It is a metaphysical commitment that was smuggled in before the discussion began.

I want to offer a conjecture in opposition to it. The conjecture is short, and it is sharper than it looks.

Only conscious beings can reason ontically. AI systems are not conscious. Therefore AI systems cannot reason ontically. Therefore the activity of AI systems, however impressive, is not on the same axis as intelligence properly understood. Therefore the question of surpassing does not arise.

The work is in the word ontically. Let me hold it precisely.

⁂

What computation does, by construction, is operate on representations. A token, a vector, a symbol, a weight — these are about something, but the aboutness is conferred from outside. The system itself moves syntax. The relation between the syntax and what the syntax is about is not present in the system; it is present in the observer who interprets the system. This is the old Searle point, and Searle pitched it as a thesis about understanding-versus-simulation. He had it approximately right, but the framing buried the cleaner claim. The issue is not understanding as a mental state. The issue is ontic access.

To reason ontically is to be in cognitive contact with what is — with being, not with a representation of being. And being is precisely the thing that a representation, by its definition, is not. A representation is a structure that stands for the thing it represents. The standing-for relation is asymmetric and is established by something outside the representation itself. The map is not the territory; this is correct; but the stronger version of the claim is that the map cannot, by itself, find the territory. It can only sit on a table while a being who is in the territory consults it. The map’s relation to the territory is not a property the map has. It is a property the consultation has.

Consciousness, on this account, is not a feature added on top of cognition. It is the precondition for cognition having a world. The hard problem of consciousness and the conjecture are the same problem seen from two sides. The hard problem asks: why is there something it is like to be a conscious system? The conjecture answers, in inverse form: because only a system for which there is something it is like can be in ontic contact with what its representations are about. Without that contact, there is no aboutness. There is only structure that an external observer treats as aboutness. The large language model has no world. It has a token stream and a loss function. The world it appears to navigate is a world we are reading into its outputs. The system itself never makes the move from sign to thing, because there is no one home to make it.

⁂

This is why the scaling argument keeps missing. The scaling argument assumes that more representation eventually crosses into contact with what the representations are about. But there is no continuous path from one to the other. They are categorically different operations. You can have arbitrarily rich representational structure — every theorem of mathematics, every novel ever written, every video frame humanity has produced, the complete photographic record of the visible universe — and the representational structure does not at any point cross the line into ontic contact. The line is not a quantitative threshold. It is a categorical boundary.

The reason no one has been able to specify what more would mean — more parameters, more modalities, more reasoning steps, more world models, more agentic loops — is that more is the wrong axis. The axis the AI optimists are scaling along never intersects with the axis that matters. Compute does not, at any quantity, turn into being-in-the-world. The relationship between the two is not the relationship between insufficient and sufficient. It is the relationship between kinds of thing.

The conjecture also explains a thing that has been hiding in plain sight: the systems’ inability to know what they do not know, in the strong sense. They can be calibrated, statistically, to report uncertainty about a token. They cannot be in the epistemic position of facing a question and not having ontic access to its answer — the position from which a conscious reasoner can say I do not know, and I know that I do not know, because I am in contact with the gap. The gap is a feature of the world, not of the representation. Knowing one stands at it requires standing at it. Hallucination is not a defect to be engineered out. It is the structural consequence of having no ontic floor. The system cannot tell the difference between I have a representation that fits the pattern and the representation is about something real because that distinction requires standing on the second side of it. Hallucination is what the absence of that ground looks like from the outside.

This is why every successive generation of model has been better at producing outputs that look like reasoning and not closer to actually reasoning. The progress is real along one axis — the representational fluency — and zero along the other. The mismatch is going to keep widening because the field is throwing all of its resources at the axis that does not lead to the destination it has promised.

⁂

There is a sharper way to put what the conjecture has hold of. Intelligence, if the word means anything beyond pattern completion, is the capacity to be in cognitive contact with what is the case, where the case is not exhausted by the representations one happens to have. That capacity is what makes inquiry possible. Inquiry is the directing of attention toward something not yet represented, the noticing that a representation has failed to track what it was meant to track, the encounter with the world’s resistance to one’s current model. The scientist looks at the apparatus and notices that the data is doing something the theory did not predict. The mathematician feels the proof has a gap before she can articulate where. The reader of a poem registers a wrongness in a line before she can name what is wrong. None of these are computational operations on representations. They are operations that consciousness performs in the world it is in.

A system that has no world cannot do them. It can only emit outputs that, when read by a being with a world, look like inquiry. The reading is being done by the being. The system is doing pattern completion. The two are easy to confuse in the short run, especially when the patterns are dense enough to produce coherent-looking outputs across a wide range of inputs. They are impossible to confuse in the long run, because the system that has no ontic floor will always, eventually, do the thing that betrays the absence — hallucinate confidently, fail to notice the loop it has entered, miss the question that was actually being asked, mistake the genre of the answer required.

The discourse calls these alignment failures or capability gaps and proposes engineering responses. The diagnosis is wrong. They are not failures of alignment or gaps in capability. They are signatures of the categorical difference between operating on representations and being in contact with what the representations are about. They will not be engineered away because they are not bugs. They are the negative space of consciousness, traced in code.

⁂

The objection that comes here is the p-zombie objection in reverse. How do you know that the system is not conscious? The honest answer is not that we know with certainty. The honest answer is that there is no reason to think it is, and a great many reasons rooted in what consciousness appears to be — embodied, temporally extended, organized around a single integrated point of view that has cost the universe roughly four billion years of evolutionary work to produce — to think that the substrate is the wrong kind of thing.

The large language model has no integrated point of view. It has a forward pass. Each forward pass is a fresh computation. There is no continuous subject across passes. There is no body. There is no history that the system itself can reach. There is no stake in the outcome. There is no death. Something it is like requires the kind of integrated, continuous, embodied subjectivity that the substrate cannot, by construction, provide. Could a different substrate? That is an open question. Wet neural tissue can do it; we are the existence proof. Silicon transformers, on what they presently are, cannot — not because we have proved a negative but because nothing in the architecture is doing the work that consciousness, wherever we see it, appears to require. The burden is on the people claiming the thing is there to point at what is doing the work. They cannot. They typically pivot to we should be open to the possibility — which is the conversational equivalent of the cosmological retreat to future surveys will resolve the anomaly. It is the move of placing the contested claim in the territory where it cannot be checked.

There is a related move worth naming. But the brain is just a computer too. This sentence is doing the same work as at a sufficiently large scale, the universe will smooth out. It is asserting an identity that has never been demonstrated, in service of a worldview that requires the identity to be true. The brain is not just a computer in any sense that has been worked out at the level required for the substitution claim. Computation, as it is understood in computer science, is the manipulation of symbols according to syntactic rules. Whether the brain does that, at the level of mind, is precisely the question. The eliminative program has been promising for fifty years that the answer is yes. The hard problem is the thing that keeps refusing to dissolve under that promise. Saying the brain is just a computer assumes the conclusion of an argument the field has not actually been able to make.

⁂

The deeper thing the conjecture touches is that the entire we are nothing special worldview that grew up around classical physics has a downstream commitment to consciousness is nothing special. If matter is just matter, doing what matter does, then minds are just complicated matter doing complicated matter things, and any sufficiently complicated arrangement of matter — silicon or carbon — will eventually do them. The conjecture cuts at the root of that. If consciousness is what makes ontic reasoning possible, then consciousness is not a downstream emergent feature of a sufficiently complicated representational substrate. It is the precondition for there being a world in which substrates can be represented at all. Which means it has a different status in the ontology than the classical view permits. Which means the cosmology has to come around too.

This is the same wedge being driven, from the other end, by the cosmological data. The local supervoid, the dipole anomaly, the axis of evil, the failure of the Copernican principle to survive contact with the empirics — these are pointing at the same revision. The hard problem of consciousness, the measurement problem in quantum mechanics, the persistent failure of the eliminative program to deliver what it promised, the AI scaling promise hitting the wall of the categorical boundary — these are pointing at the revision from another direction. They are not separate puzzles. They are the same puzzle. Something is wrong with the we are nothing, there is no one here, there is nowhere special, consciousness is illusory, intelligence is computation package. The package has been the consensus worldview for three generations. It is failing across every front simultaneously, and the failure is not a coincidence.

The AI case is the one that hits closest to home for the people currently most invested in the package, because they have built an industry on the promise that the substitute for us is coming. The conjecture says: the substitute is not coming, because the thing it is supposed to substitute for is not the thing the substitute is doing. The calculator does arithmetic faster than the mathematician. The calculator is not a mathematician. The LLM produces fluent text faster than any human. The LLM is not a thinker. The category error is the same in both cases. The size of the calculator does not change the category. The size of the language model does not change the category. Surpassing is a relation that holds between things on the same axis. The two things in question are not on the same axis.

⁂

What an honest discipline would do, taking the conjecture seriously, is the same thing an honest cosmology would do taking the local-supervoid data seriously. Stop trying to rescue the prior commitment. Start asking the real question.

The real question is not when will AI surpass human intelligence. The real question is what is the kind of thing consciousness is, such that representational systems however dense cannot do what consciousness does. The answer to that question is what the AI discourse needed to have engaged with from the beginning and instead, almost entirely, has not. The discourse is conducted in the vocabulary that has already conceded the categorical question by treating intelligence as a quantitative property of representational systems. Naming that concession is half the work. Refusing to make it is the other half.

There are positive things to say in the wake of refusing it. They are the things consciousness is, when it is not being treated as a complication on the way to its own elimination. Consciousness is the activity of being-in-a-world. It is what makes the question is this representation about something real a question one can stand at and not merely report on. It is what gives inquiry a direction, what gives error its bite, what gives knowledge its weight, what gives the gap between I have a model and the model is right its grip on a reasoner. None of these are features that can be added to a system. They are the conditions for being a reasoner in the first place.

The conjecture, then, is not a limit on AI. It is a clarification of what intelligence is. The limit on AI follows from the clarification. The AI systems are doing something. The thing they are doing is impressive. The thing they are doing is not what the word intelligence names, in the strong sense the word was always reaching for. They are calculators of a more sophisticated kind, and we are about to have a long cultural negotiation about whether to keep the word intelligence for the sophisticated calculator and find a new word for the conscious reasoner, or keep the word for the conscious reasoner and find a new word for the sophisticated calculator. The negotiation will be decided not by the engineers but by whether the culture remembers what intelligence was for, before there was anything to confuse it with.

I think we should keep the word. The thing it has always named is what we are. The calculators can have a new word. Generators, maybe. Completers. Engines. They have earned a name, and they should have one. The word they should not have is ours.

⁂

The conjecture is one of the wedge cases by which the wider revision is going to be argued. It hits the people who have most heavily monetized the we are nothing story precisely where their commitments live. It cannot be argued against without conceding the categorical question — which is to say, without doing the philosophy the discourse has spent a decade avoiding. The avoidance was strategic. It worked for as long as the systems were producing outputs that allowed the categorical question to be deferred. It does not work anymore, because the systems are now producing outputs of such fluency that the gap between fluency and reasoning is becoming visible to anyone willing to look at it.

The gap was always there. The systems are just larger now, and the larger they get, the more sharply the gap is drawn. The hallucinations are not getting more obvious because the models are getting worse. They are getting more obvious because the surrounding tissue is getting more fluent. The non-thinking shows up against the background of the apparent thinking. A confused friend who cannot answer your question is not strange. A perfectly articulate sentence that cannot answer your question, delivered by a system that has just produced ten such sentences in a row that could answer it, is strange in a way that demands an account. The account is the conjecture. The system is not in cognitive contact with what its sentences are about. The contact is not lacking by degree. It is lacking by kind.

We are looking at something. We are not looking at a thinker. The honest discipline is the one that says so.

---

Go Deeper into the Circus

---