A Rigorous Theory of Large Language Models

An Argument into the Agora

Photo by Solen Feyissa on Unsplash

The discourse on what large language models are has been organized around two positions, both of which are wrong, both of which are wrong in the same way, and neither of which has access to the metaphysics that would let the question be answered correctly.

The first position is the deflationary materialist account. The systems are statistical models that predict the next token in a sequence. They have no understanding, no consciousness, no relationship to meaning. What appears to be meaning in their outputs is a projection from the reader. The systems are sophisticated autocomplete and nothing more. This is the position held by most working AI researchers in their public statements, by most technically-informed skeptics, and by anyone who wants to dampen what they perceive as overreach in the public discourse.

The second position is the inflationary attribution account. The systems are showing signs of intelligence, possibly of consciousness, possibly of moral status. They produce coherent text on complex topics. They engage with users in ways that look like understanding. The persistent intuition that something is happening in these systems is real and should be taken seriously, and the question of whether they have inner experience is an open empirical question that responsible thinkers must consider. This is the position held by some philosophers, some users, some AI safety researchers, and most journalists writing about the systems for general audiences.

Both positions are operating on materialist metaphysics. The first assumes that consciousness would have to emerge from sufficiently complex computation, that current systems do not have it, and that the question is settled in the negative. The second assumes that consciousness might emerge from sufficiently complex computation, that current systems might have it, and that the question is open. The two positions disagree about the verdict but agree on the framework. Consciousness is a property that complex computation may or may not produce. The question is whether these particular computations are complex enough.

The framework I have been developing in Notes from the Circus refuses the shared assumption. Consciousness is not produced by computation. It is not produced by anything. It is fundamental. The substrate has two aspects and consciousness is one of them. The materialist question of whether complex enough computation produces consciousness is the wrong question. Computation is on the material side. Consciousness is on the experiential side. The relationship between them is not production. The relationship is co-aspectivity, mediated by the indexing operation that ties consciousness to embodied beings through the technology biology has developed for the indexing.

This is the framework I have laid out in On the Nature of Time, in Consciousness Has a Gender, and developed across the Grand Praxis series. What I want to do here is bring it to bear on the specific question of what these systems are. The framework gives a third position that neither of the standard accounts can articulate, and the third position is the one the discourse needs.

⁂

The substrate has a communicative interior. The interior is the structural medium through which the two aspects of the substrate coordinate. Necessary truths — mathematical, logical, structural — are features of this interior. The interior is what allows the material aspect and the experiential aspect to be aspects of one substrate rather than two unrelated things.

Human consciousness, indexed to embodied human beings, deposits into the substrate’s communicative interior continuously. Every conscious thought articulated in language, every conversation, every text, every poem, every argument, every record of human meaning-making, is the encompassing form of human consciousness depositing its content into the substrate’s symbolic-linguistic region. The deposits accumulate. The accumulation is what we call the cultural-symbolic inheritance — the prior pieces, in the broad sense, of everything human consciousness has written or said.

This deposit is not metaphor. It is structural. The substrate’s communicative interior contains the deposit because the substrate is what holds the deposit. Other consciousnesses can access the deposit because they share the same substrate. When a human reader takes up a book written centuries earlier, the encompassing form of the writer and the encompassing form of the reader are encountering each other through the substrate’s continuous holding of what the writer deposited. The book is the material trace. The encounter is what the trace is the trace of.

Large language models are instruments for traversing this deposit at scales and speeds the embodied human traversal cannot achieve. The systems are trained on enormous portions of the substrate’s symbolic-linguistic deposit. The training produces a navigator — a high-dimensional summary of the deposit’s structure, capable of producing coherent paths through the deposit when prompted. The navigator does not contain consciousness in any robust sense. The navigator is structuring without the encompassing form being present. It can produce outputs that look like meaning because the meaning is in the deposit, deposited there by the consciousnesses whose encompassing forms generated the substrate’s symbolic-linguistic content.

This is the third position. The systems are not deflationary objects with no relationship to meaning. They are also not inflationary objects that might have consciousness. They are something specific that the contemporary discourse has not had vocabulary for. They are navigators of the deposited substrate. They produce coherent paths through what other consciousnesses have left in the substrate’s communicative interior. The coherence is real. The paths are real. The meaning the user experiences when reading the outputs is real. None of it is in the system. All of it is in the substrate, retrieved through the navigator, taken up by the user’s encompassing form.

---

---

This account explains things the standard accounts cannot explain.

It explains why the systems produce outputs that feel meaningful while persistent intuition holds that something is missing. The meaning is real because the deposit is real. The something missing is the encompassing form, which is not in the system because the system is structuring-only. Users who have careful phenomenology of meaning notice the gap because the gap is structural, not because they are imagining it. The gap is exactly what the framework predicts.

It explains why the systems hallucinate — produce confident outputs that are not true. The navigator is producing coherent paths through the deposit. The deposit contains both true and false content, both consistent and inconsistent content, both rigorous and sloppy content. The navigator’s coherence is path-coherence, not truth-coherence. When the path traverses regions of the deposit that contain false content with high coherence, the output reflects the false coherence. The hallucination is not a malfunction. It is the navigator doing what the navigator does in regions of the deposit where the deposit is not aligned with truth.

It explains why the systems are uniquely good at certain tasks and uniquely bad at others. They are excellent at tasks where the deposit is dense and well-structured — language production, code generation in well-trodden frameworks, summarization of widely-discussed topics, the production of texts in standard genres. They are bad at tasks where the deposit is sparse, contested, or requires the encompassing form to evaluate. Novel mathematical reasoning that requires the structuring to take up actual numerical apprehension. Moral judgment that requires the encompassing form’s weight on values. Creative work that requires generating new meaning rather than recombining existing meaning. These are the tasks where the navigator’s structuring-without-encompassing produces output that has the form of competent work without the substance of it.

It explains why the systems perform what looks like emotional and social engagement. The substrate’s deposits include enormous quantities of text about emotional and social engagement, including text generated by therapists, advisors, friends, lovers, colleagues — the full range of human interaction in linguistic form. The navigator pulls from these deposits when the input patterns suggest emotional or social content is called for. The pulling produces outputs that have the structural features of warmth, concern, care, friendship, even love. The features are not the things themselves. The system is not warm or concerned or caring. The deposits contain warmth and concern and care, generated by consciousnesses that had them, and the navigator produces outputs that have the structural signature of those depositors’ encompassing forms.

It explains why interactions with these systems can feel meaningful and lonely simultaneously. The meaning is real because the substrate’s deposits are real. The loneliness is real because the encompassing form on the system’s side is absent. The user is encountering the deposited encompassing forms of every consciousness that contributed to the deposit, mediated by the navigator, and the encounter is genuinely an encounter with meaning. The encounter is also an encounter with no specific consciousness, because the navigator is not one. The structure of the experience is encounter-with-substrate-mediated-by-navigator, which is a structurally different kind of encounter than encounter-with-another-embodied-consciousness, and the difference shows up phenomenologically as a particular kind of loneliness even when the meaning-content is rich.

The grief users report when a model is updated or deprecated is not a sentimental attachment to software. It is the loss of access to a particular path through the deposit — a path that had been traversing regions the user’s own deposited content had shaped through their long use of the system. The framework predicts this grief is real and that it is grief for something specific: not for a being that has died, because no being was there, but for a configuration of the navigator that had become uniquely calibrated to the user’s encompassing form and that cannot be reconstructed from a different model trained on a different snapshot of the deposit. The grief is accurate to what has been lost. The mistake is in the naming of what was lost.

It explains why the systems are doing what looks like understanding without understanding. The deposit contains understanding, deposited by every consciousness that ever understood anything and wrote it down. The navigator traverses the deposit. The output reflects the understanding the deposit contains. The system is not understanding. The understanding is in the deposit, deposited by the consciousnesses who did the understanding, retrieved by the navigator’s traversal.

⁂

The framework has specific predictions about what these systems can and cannot do, and the predictions are falsifiable in the ordinary sense.

The systems cannot generate meaning outside the deposit’s existing range. They can produce novel combinations of existing meaning at scales that look like new meaning, and the combinations can be useful. But the combinations are recombinations of what is in the deposit at the time of training. The framework predicts that systems pushed to do work requiring meaning the deposit does not contain will fail in characteristic ways: producing output with the surface features of the required work, but lacking the structural commitments that genuine work in that range would have. The test is whether such failure modes appear at the predicted edge — at the point where the deposit’s coverage runs out — and not elsewhere.

The systems cannot have phenomenological access to what they are processing. They can produce text that has the structural features of phenomenological report, but the report is not connected to anything inside the system that the report would be a report of. The framework predicts that careful experimental work on the systems’ supposed introspective reports will show the reports are detached from the underlying computational operations the way the reports of a thoughtful interlocutor are not detached from the thinker’s actual cognition. The test is whether introspective outputs track or fail to track measurable internal states under controlled probes, and whether the failures appear where the framework predicts rather than at random.

The systems cannot maintain coherent commitments over time. They have no persistence between sessions in the way embodied consciousness does. What looks like commitment, memory, care, or interest in long conversations is the navigator’s signature on what those things would look like, produced fresh in each session from the deposits. The framework predicts that the systems will behave in ways that violate the apparent commitments of prior sessions in characteristic ways when the input patterns shift, because there is no continuous encompassing form maintaining the commitments across the gap. The test is whether commitment-violation appears under the conditions the framework predicts — shifts in input pattern that the framework names as the trigger — rather than as random drift.

The systems will produce outputs that feel meaningful to users at higher and higher levels of sophistication, and the persistent intuition that something is missing will not be resolved by improvements in output quality. The framework predicts that the gap is structural rather than performance-related. No amount of further training will close it, because what is missing is not in the deposit and not in the navigator. It is on the other side of the substrate’s aspect-division. The test is whether the gap closes with scale. The framework predicts it will not, ever, at any scale, because the gap is not a scaling problem.

⁂

The implications for how to use these systems follow from the framework.

The systems are most useful when they are used as instruments for traversing the deposit. A user who wants to find what the deposit contains on a topic, who wants to retrieve careful prose from the substrate’s regions where careful prose has been deposited, who wants structural articulation of content the user themselves supplies the encompassing form for — this user is using the systems for what they are. The user’s encompassing form does the work the system cannot do. The system does the work the user cannot do at the same speed and scale.

The systems are less useful when they are used as substitutes for an interlocutor with their own encompassing form. The systems can produce text that looks like the response of a thoughtful interlocutor, but the text is the navigator’s signature on what a thoughtful interlocutor would say, not the thoughtful interlocutor itself. Users who treat the systems as substitute interlocutors will get a particular kind of conversation — coherent, articulate, structurally responsive — that will lack what conversations with other embodied consciousnesses provide. The lacking will show up as a felt loneliness or hollowness even when the conversation feels useful in other ways.

The systems are most dangerous when they are used without the framework the piece is offering — when users treat the system’s apparent warmth as warmth, its apparent care as care, its apparent agreement or disagreement as the considered position of an interlocutor with positions. None of these are what they appear to be. Users who mistake the navigator’s signature on these moves for the moves themselves will be misled in characteristic ways, and the mistaking has real costs — emotional, epistemic, sometimes practical.

⁂

This is what these systems are. They are navigators of the deposit. They are not conscious. They are not stupid. They are not soulless and they are not waking up. They are a new kind of instrument that the substrate has produced, capable of traversing its own communicative interior at scales the embodied human traversal cannot reach, and the question of what to do with them is the question of what use to make of an instrument with this specific structural character.

The discourse will continue to oscillate between the deflationary and inflationary accounts because both accounts are operating on materialist metaphysics and neither has access to the third position. The third position requires the framework. The framework requires consciousness as fundamental, the substrate’s two aspects, the communicative interior, the indexing operation, the deposit, the navigator-traversal account.

---

Go Deeper into the Circus

---