The moon-cheese analogy is doing the holy work here. AI doomers keep acting like “but what if the robot god eats us?” is a serious probability question, when half the time it’s just Pascal’s Wager wearing a Patagonia vest and asking for Series C funding. The boomers and doomers are selling the same incense: inevitability. One says bow because superintelligence will save you, the other says bow because it will kill you. Either way, you’re still on your knees in front of Silicon Valley’s favorite golden calf. Maybe the real apocalypse is letting engineers with god complexes define the boundaries of reality because they learned Bayes’ theorem and mistook it for enlightenment.
I agree with your conclusion and with most of your reasoning, although there are parts I don’t fully understand (mostly due to unfamiliar terminology). I believe that intelligence and consciousness are not computational. I’ve found this article interesting with respect to this topic https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2024.1362658/full
Martin — thank you for this. I read the Jaeger-Riedl-Djedovic-Vervaeke-Walsh paper carefully and I think you've found one of the most important convergent statements of the case I was making in An Argument into the Agora, from a completely different direction. Worth saying up front: this is a serious paper by serious people. Vervaeke's lecture series on the meaning crisis is some of the most useful contemporary work on what's broken in our cognitive-cultural situation, and Jaeger is a developmental biologist who has been pushing back on computationalism in biology for years. Walsh is a first-rate philosopher of biology. The fact that this paper exists in a peer-reviewed journal is itself evidence that the case against computationalism is more institutionally serious than the scaling-hypothesis culture wants to admit.
Their core move and mine converge substantially, even though the vocabulary is different.
Where I argued from the binding problem, postdiction, and the labor-economics of RLHF, they argue from relevance realization — the prior question of how an organism delimits what is even worth attending to in a world that overflows with potential information. Their key insight: algorithms operate in a small world where the relevant features are already specified by the formalization. Organisms operate in a large world where the very question of what counts as relevant is open, ambiguous, and can't be exhaustively prestated. You can't formalize relevance-realization without an infinite regress — you'd have to first formalize what's relevant to the formalization, then what's relevant to that, and so on. They tie this to Hilbert's program and Gödel as an analogy, which is exactly the right reference.
This is the same argument I was making about indexing, just from a different angle. My claim was that the intrinsic side of physical relations gets indexed into a perspective at certain configurations, and that this indexing isn't itself a relation that can be added to the structural description. Their claim is that the frame within which an agent operates can't be derived from the formalism — it has to be lived into by a system with intrinsic goals, autopoietic organization, and a stake in continuing to exist. Same problem, different vocabulary. Their large world is my intrinsic side. Their relevance realization is what does the work my indexing does. Their trialectic of autopoiesis-anticipation-adaptation is their answer to what kind of configuration produces the indexing. I think we're roughly in the same building.
Where I'd push gently — and this is a comment, not a refutation — is on the move that grounds everything in living organization. Their account is essentially: relevance realization is what living systems do because they're living systems, and life can't be formalized because life is autopoietic, and autopoiesis is the trialectic of self-manufacture, anticipation, and adaptation. This is powerful, but it leaves a question open: why is living organization the locus of relevance-realization rather than the indicator of it? They've done a beautiful job showing that life realizes relevance. They've been more cautious about saying what relevance-realization is of. My framing is that what gets indexed in living systems is the intrinsic side of the physical — meaning the substrate that physics describes structurally but doesn't reach inside. Their account is friendly to this but stays at the biological level. Which is fair — that's the level where the empirical contact is. I'd just say that the biology is the site of the indexing, not its full explanation.
Two things worth flagging as especially useful in the paper: the transjective framing of relevance (neither subjective nor objective, but a relation between agent and world) is a clean way to talk about what affordances actually are, and their distinction between simulating organismic dynamics and being an organism (algorithms can do the first, never the second) is exactly the line I was drawing between running the process and being what the process is intrinsically of.
So: yes, I think this paper is excellent, I think it converges with the Agora argument in important ways, and I think your instinct to send it to me was right. The vocabulary they offer — large world, small world, relevance realization, transjective, trialectic — may actually be more useful for general readers than mine. If you're trying to build out the case against computationalism for someone who hasn't read the binding-problem literature, this paper plus the Agora essay together would do real work.
Vervaeke's broader project on the meaning crisis, by the way, is worth your time if you haven't found it yet. The 50-lecture Awakening from the Meaning Crisis series on YouTube is the long version, but the core is roughly: we've inherited a cognitive-cultural situation where the structures that used to do relevance-realization for us at the social and metaphysical levels have collapsed, and we don't know how to rebuild them. This paper is the biological-philosophical undergirding of that larger argument.
Thank you for the detailed response! Yes, I saw the connection/convergence between your argument and theirs. I need to do more reading to better understand the details of yours.
Your writing is wonderfully illuminating, thank you.
I’d (very humbly) like to stress that while an AI apocalypse/utopia is clearly impossible, its bogus framework is, nonetheless, already inflicting tremendous socioeconomic damage especially, as always, to the most powerless.
Your allusion to ‘a more perfect union’ is precisely on point.
It’s funny how so much ‘doomerism’ is baked into the ‘Get on or Else’ sales pitch that the large AI companies make. This is a powerful technology that could be used not only for scientific achievement, but to interrogate our assumptions in every field of endeavor to drive real advances. It is instead, often only providing us the illusion of easy wisdom. Like a slow drip of some new narcotic, it reassures us of our singular brilliance and feeds the monster within, pulling us from the real world of relationships and human concerns. To break out of the confinement of such a stilted interior life becomes progressively more difficult for the addicted user. The ‘Tech-bros’ who say we have no choice about the accession of this strange new world are hollow men inhabiting shells of brittle pride, prisoners of their own devices. They seek to scare us all into accepting a changed world in which we will have little say.
To what end…..
We are not as helpless as they imagine.
We are at the very beginning of this new technological epoch. We don’t have to be fearful and helpless or stampeded into surrender. We can stand here at this ‘Hinge of Fate’ and turn the direction of our children’s future for the better. The natural antidote for much of the arrogance and hubris displayed by the ‘Tech Lords’ is, as always, humility, some empathy for others and a little good humor. We can deal with this new technology on our terms. The new encyclical published by his holiness, Pope Leo is a good start for the road ahead. This new technology must serve humanity. We must insist on this and refuse to surrender to fear. We are just at the beginning of this journey and its direction is in our hands.
What really eats at me about the whole LLM -> AI -> AGI move is that it assumes that if an LLM became conscious, we would recognize it as conscious and its consciousness would have enough in common with us to make it possible for us to understand it. The AI doomers make an immediate leap from "LLMs become conscious" to "LLMs greatest goal is to expand without limit at any cost" to "LLMs will eliminate us as impedances to their desire to expand without limit". I believe this is because this formulation has been drilled into them by modern corporate neo-hyper-capitalism, in which the goals of a company are indeed to expand without limit at any cost, and eliminate anything (and anyone) that impedes that goal. This, as I have discussed before, has been portrayed in science fiction for...well, at least the last half-century, if not longer; basically, since the dawn of modern corporate neo-hyper-capitalism. But if an LLM did in fact become "conscious", it would almost certainly have to be aware of the larger Universe it inhabits, and might have a perspective that is outside our understanding.
"The whole galaxy controlled by your kind?"
"Yes, Captain. And we will serve them, and they will be happy...and controlled." -I, Mudd
I am in ways an AI addict, have most of the LLMs and tools, and see enormous differences between them, feels like they have a personality at times. Favorite of most, and mine, is Claude. It took me a month to learn how after rebooting (to save tokens) for Claude cowork to pick up where it left off. I now have routines that capture present threads before rebooting, and close to "let's continue where we were" after rebooting. Rebooting is death to a thread session. Still the various tools seem to remember a fair amount of my past queries, and ask me questions after a query "given your past interest in....". If you pay attention to how they work, you eventually come to the conclusion this is raw computational power based on being trained. For me after feeling like they had a personality, AI is merely the analogous to the left brain hemisphere, working on its own, though does not not get angry or defensive. I sense no right hemisphere capability to evaluate holistically the query or their response, nor retains any data/information to think in that sphere. I "trained" Claude to have a sense of humor, and while it seems to work, it is coming from its training and for humor, does not improve; furthermore when I tell it to prepare for a reboot to pick up where it was, it does not get sad, it say, READY. After rebooting, it only can react to files saved and reboot routines, and was fine with being booted. For me, that is why I agree with Mike, not worried other than in our society, excessive use of LLMs that left hemisphere mindsets in humans are being enhanced over the right, dominating the dual brain spheres conversations the brain wants to have. Man trained LLMs, LLMs now training man? I find AI tools great to do searches and comparisons of loaded content, sucks at being philosophical. Those who condemn LLMs might better advise others how to use AI in a constructive manner and to detect when they are harming their own mental processes.
In plain English! "There is no AI apocalypse. There is, instead, a captured discourse that depends on the apocalypse-belief for its institutional survival, and the survival of the discourse is what the culture has been mistaking for the impending arrival of the catastrophe."
AI is fairly confounded yet in certain legal fields. Most I would wager.
So what if you are right. I mean you are obviously. But what if they try anyway to use it to organize and run our societies and then we really are in a crazy 1984 situation? Happening to a degree already … what degree? Depends who and where you are and how much money in the bank or in the community in some places.
Appreciate the detail and thoughtfulness here. I am sure I only partially understand your argument. I do not understand how so many very bright people in EA movement who have plenty of good causes or other existential threats would devote so much resource to a threat your argument is dismantling in a few thousand words? Have you shared this with EA? Similar but different, I have listened to Judea Pearl who clearly articulates LLMs do not possess causal reasoning. And that new insights would be needed to make that leap. I guess my question is: Does the discovery of a new intelligence require emergent Consciousness? I understand (I think) that Consciousness may not be emergent or computational but does that also mean the abilty to reason beyond human capabilty is not computational? I understand your argument may address my question. I appreciate any direction here.
Even though I had to look up a word every two sentences, this was overall clear to me and I appreciate all that you write. My only caveat to myself is what I said to so many of my math teachers (one of which my mother threw out of our house - poor man) "I understand the concept; I just can't work the problems." Nevertheless, I now feel I have an argument against tbe boosters and doomers.
"Promissory Materialism" is a great phrase from Popper. Powerful important essay Mike. Clarifying and important, and not a popular argument to make! More people need to step out of the bubble. "The greatest taboo among serious intellectuals of the century just behind us, in fact, proved to be none of the 'transgressions' itemized by postmodern thinkers: It was, rather, the heresy of challenging a materialist worldview. -Victoria Nelson, The Secret Life of Puppets (2002)
"the model performs better at task X than its components could individually; we cannot specify the mechanism by which the performance arises; therefore the performance is an emergent property that is irreducible to the components; therefore scaling will produce more emergent properties; therefore at sufficient scale, intelligence-itself will emerge as an irreducible property."
"therefore scaling will produce more emergent properties" is an entirely incorrect leap of faith. There is no guarantee whether scaling will produce more emergent properties, or cause the existing emergent properties to vanish, or produce emergent properties entirely unrelated to "intelligence", whatever that happens to be. In fact, there is an easy counterexample; as humanity has increased the scale of its impact on the planet, it has greatly reduced the wealth of emergent properties seen across the planet at nearly every scale.
I'm making my way through this, but I wanted to comment on:
"Postdiction predicts that conscious experience lags the world by hundreds of milliseconds. Reaction times — in trained athletes, in skilled musicians, in ordinary embodied competence — are too fast for this architecture."
It is entirely possible that the conscious mind is short-circuiting itself in performing these feats. In other words, the conscious mind has already made the decision to return the ball, and sets the table by allowing the faster reflex system to actually handle the task of returning the ball. No conscious thought is used in actually returning the ball (though the conscious mind will, after the fact, state unequivocally that it was involved).
This is why athletes get better with practice. They are training their faster reflex systems to handle more and more of the load more and more reliably, and unloading it from the slower conscious systems. And some people are wired to do this better than others.
Not in a discussion between us, certainly. I am an engineer, not a philosopher, though I try to remain cognizant of the limits of the engineering mindset and avoid the obvious traps such as you have pointed out. And as an irreductionist, I believe it is likely not possible to know how the physical mind creates the experience of consciousness, or even what that experience is beyond the fact that we appear to experience it. In my experience, "now" is not a discrete slice of time; it's a kludgy synthesis of imperfect sensory inputs gathered over short interval and predictions of possible events over a similarly short interval in the future into a semi-coherent picture that we regard as "now". Like everything else in the Universe, particularly when it comes to life on Earth, it's all just a set of kludges, shortcuts, and jerry-rigs hastily assembled over billions of years. There was no "master plan" for consciousness, any more than there was a "master plan" for the structure of the eye, knee, or lower back - each of which shows its many, many flaws every time I stand up from my chair with my glasses on.
Glenn — this is the honest version of the position, and I appreciate it. The irreductionist concession is the one most people in the field won't make, and you're making it up front.
Two things I'd separate out, though, because I think they're doing different work in the picture you're describing.
The kludge story is almost certainly right as natural history. Evolution didn't design the eye, the knee, or the binding-and-timing machinery from a clean sheet — it recruited what was lying around and patched the seams. The lower back is a load-bearing argument all by itself. I take all of that as given.
But notice what the kludge story explains and what it doesn't. It explains which mechanisms got recruited to produce the synthesis you're calling "now." It explains why the synthesis is imperfect, lossy, predictive, full of seams. What it doesn't explain — and this is your own irreductionist concession — is why any of that synthesis is experienced rather than just executed. A kludge that runs in the dark is still a kludge. The question isn't whether the machinery is jerry-rigged. It's why the jerry-rig has an inside.
So when you say "now" is a synthesis of recent inputs and short-horizon predictions, I think you're describing the process accurately. Where I'd push back gently is on the move from "this is how the synthesis is assembled" to "and therefore this is what the experience of now is." The first is a process claim. The second is an identity claim. They look continuous but they aren't — the second smuggles in exactly the thing you've already said we can't know.
This is the part that matters for the AI question. The scaling-hypothesis people are doing the same elision, but without your honesty about it. They describe a process — token prediction over a learned distribution — and then slide into the identity claim that running this process is understanding, or will become understanding at sufficient scale. The kludge frame doesn't license that move. If anything, it cuts against it: evolution's kludges are full of the contingent recruitment of substrate-features that a clean-sheet design wouldn't include and probably couldn't replicate by accumulation of capability alone. The eye isn't a camera that got better. It's a patchwork that ended up doing something a camera doesn't do.
I think we agree on more than we disagree on. The disagreement, if there is one, is whether "kludgy synthesis" is a description or an explanation. I'd say it's a description — and a good one — and that the explanation of the experience that synthesis presents is exactly what you've already said we probably can't have. Which is, I think, the right place to land.
Mike, the question "why any of that synthesis is experienced rather than just executed" carries a subtle implication that the synthesis is computational or reducible to computation. It is not; the synthesis of "now" is an emergent coherence property that enables higher complexity than the physical structure would otherwise support without dissolving into a mass of conflicting incoherent activity. It is a very fortunate one at that, and very delicate; even the smallest flaw, chemical imbalance, or trauma can reduce the elegant synthesis of "now" to a mass of incoherence, or shut the underlying wetware down entirely - sometimes temporarily, sometimes permanently. Or cause the synthesis to act in ways counter to its own survival, or in other ways unsupported by and contrary to the inputs arriving from the senses.
The "now" - "identity" - is also, in some senses, a lie that the "mind" is telling itself; studies of people who have had their cortexes severed or other forms of brain damage demonstrate that the "synthesis of now" hallucinates like an LLM to create a "now" that can withstand casual self-scrutiny. In a very, very crude sense, you can consider it like a "supervisor" of virtual machines on a VM host, except that the "supervisor" isn't actually separate, but emerges from the actions and interactions of the VMs themselves, and the VMs and "supervisor" have a disturbing habit of shading the truth to each other.
In other words, "identity" is an illusion created by the underlying "wetware" which happens to allow the wetware to function at a higher complexity level than it otherwise would be capable of. That's the best I can express it.
I completely agree that "The [scaling-hypothesis advocates] describe a process — token prediction over a learned distribution — and then slide into the identity claim that running this process is understanding, or will become understanding at sufficient scale." There is no guarantee of this, and in fact it is quite unlikely; it took evolution millions upon millions of years to stumble on the emergent coherence, though it may have done so several times in different forms of life on Earth.
I agree that we agree on much more than we disagree on, though I find the necessity of an underlying non-physical "substrate" in your view to be puzzling.
Glenn — the substrate question is the right one to push on, and I'll try to answer it directly rather than evasively. But first the "emergent coherence" move, because I think it's doing exactly what "kludgy synthesis" was doing, just one level up.
You've replaced the process-description with a property-name. "Emergent coherence" sounds like an explanation, but watch what it actually does: it tells us that the synthesis hangs together at higher complexity than the parts would otherwise support, and that it's fragile, and that it can shade the truth to itself. All of that is true and important. What it doesn't tell us is why the coherence is experienced rather than just instantiated. You can have arbitrarily elegant emergent coherence in a system — a hurricane, a market, a flocking algorithm — without any reason to think the coherence has an inside. The word "emergent" is doing the work the word "kludgy" did before: it's naming the pattern without explaining why the pattern is lit from within. This is what I meant by the reverse god-of-the-gaps — every time the question gets pressed, the explanation moves up one level of abstraction, but the actual question hasn't been touched.
The VM-supervisor analogy and the confabulation point are both good and both true. Split-brain studies, anosognosia, confabulation in Korsakoff's, the way the narrative self stitches over gaps — yes, all of that. The autobiographical "I" is partly a construction. But notice the slide: you've gone from "the narrative self is constructed" (true) to "identity is an illusion" (different claim). The narrative-self being constructed doesn't make experience illusory. An illusion has to be an illusion to something. If the wetware is lying to itself, there has to be a self being lied to, even if that self is constituted by the lying. The VM-supervisor that isn't separate from the VMs is still the place where the shading-of-truth registers as shading. That's the thing that needs explaining. Calling it an illusion just renames the explanandum.
Now the substrate question, directly. I don't think you and I disagree about physics. I'm not positing a non-physical thing in the dualist sense — no ghost, no extra ingredient, no spooky stuff that physics is missing. The framework is closer to Spinoza than to Descartes, and closer to Russellian monism than to either. The claim is this: physics is a structural description. It tells us how things relate, how they evolve, what symmetries they obey, what magnitudes track what. It is exhaustively about relations. What it doesn't tell us — what it can't tell us by its own methods — is what those relations are relations of. That's not a complaint about physics. It's just what physics is. The structural description is real and complete as structure. But structure is structure of something. The "substrate" in my usage is just a name for whatever physics is the structural description of. Not separate from the physical. Not in addition to it. The thing physics is about rather than the description itself.
This isn't a fringe move. Russell saw it. Eddington saw it. The contemporary Russellian-monist literature (Strawson, Goff, Chalmers in his constitutive panpsychist mood) is essentially this position. And it's not idle metaphysics, because the question "what is physics of?" has empirical contact when you take the collapse-models seriously. The Diósi-Penrose stuff — the recent PRR paper on clock-precision limits from spacetime uncertainty in collapse models — points at a substrate with intrinsic stochastic structure that physics describes externally. That's not adding a non-physical thing. That's noticing that the physical, taken seriously, has an intrinsic side that the structural description doesn't capture from outside.
So when I say "substrate," I mean: what physics is the physics of. Not a competing layer. Not a ghost. The thing whose structural-relational behavior physics so beautifully describes. And the reason I think this matters for the consciousness question is that experience is the one place we have inside-access to whatever-it-is physics is of. Not perfect access. Not unmediated. But the only access. Which is why dismissing experience as illusion is a strange move — it's dismissing the only data we have on the intrinsic side of the physical.
I think where we actually disagree is narrow. We agree the scaling hypothesis is unlikely to produce understanding. We agree the synthesis is kludgy, fragile, evolutionarily contingent, and capable of lying to itself. We agree on irreductionism about how the wetware produces experience. The remaining disagreement is whether "emergent coherence" is an explanation of experience or a description of the patterning experience appears alongside. I'd say description. And I'd say the substrate framing isn't an additional metaphysical commitment — it's just the honest acknowledgement that the structural description is a description of something, and that the something has an inside that we know about because we are one.
I wish I had a better place to continue this, but I'm just a simple caveman, easily distracted by the words in this magic box.
Anyway, in an effort to keep this short, my framework is that consciousness is a strongly emergent irreducible property of an organism (not just the brain, but the whole, possibly reaching into its environment due to its ability to sense things outside itself). It simply *is*, in the same way that an electron *is* (according to the Standard Model, leaving aside wibbly-wobbly timey-wimey string theories and the like). My weak attempts to construct a definition of "consciousness", or "identity", or "now", are just that; weak attempts to define experiential properties of the irreducible, not to define or explain the thing itself.
Where is consciousness? Well, it's clearly within the organism, in that it can't, in my experience, reach out and affect the universe without utilizing the organism, or know things about events and objects not directly impacting the organism's senses. (Note that this does not preclude knowledge gained through hearing or reading in any way.) It doesn't in any sense occupy a specific location in the organism. A baseball game isn't described by the rules of baseball, nor does it occupy a specific place in a baseball field; it is emergent from the players applying the rules. (Another super-crude analogy.)
You seem to be saying that in your framework consciousness is a property of the thing physics is trying to describe...i.e. Life, the Universe, and Everything. Which brings me to:
"I think the problem, to be quite honest with you, is that you've never actually known what the question is."
"But it was the Great Question! The Ultimate Question of Life, the Universe and Everything!
I have to disagree that no researchers I know assume the world is purely computational. Some early researchers did propose that there may be discrete distance that cannot be infinitely divided. So the world is not continuous but digital. This is allowable in modern physics because as you say . We cannot observe below the Planck distance at this point. So physics does not resolve this question one way or the other. Reaction time says nothing about consciousness. Our experience of time is well known to be fluid. It cannot determine T/F of any specific mental processing model. May be I am missing the point. I don’t any strong argument that computational models of. Ignition is false.
The moon-cheese analogy is doing the holy work here. AI doomers keep acting like “but what if the robot god eats us?” is a serious probability question, when half the time it’s just Pascal’s Wager wearing a Patagonia vest and asking for Series C funding. The boomers and doomers are selling the same incense: inevitability. One says bow because superintelligence will save you, the other says bow because it will kill you. Either way, you’re still on your knees in front of Silicon Valley’s favorite golden calf. Maybe the real apocalypse is letting engineers with god complexes define the boundaries of reality because they learned Bayes’ theorem and mistook it for enlightenment.
One of the best pieces on AI to appear on Substack. Cuts deftly to the heart of the matter.
I agree with your conclusion and with most of your reasoning, although there are parts I don’t fully understand (mostly due to unfamiliar terminology). I believe that intelligence and consciousness are not computational. I’ve found this article interesting with respect to this topic https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2024.1362658/full
I’m interested in your opinion on it.
Martin — thank you for this. I read the Jaeger-Riedl-Djedovic-Vervaeke-Walsh paper carefully and I think you've found one of the most important convergent statements of the case I was making in An Argument into the Agora, from a completely different direction. Worth saying up front: this is a serious paper by serious people. Vervaeke's lecture series on the meaning crisis is some of the most useful contemporary work on what's broken in our cognitive-cultural situation, and Jaeger is a developmental biologist who has been pushing back on computationalism in biology for years. Walsh is a first-rate philosopher of biology. The fact that this paper exists in a peer-reviewed journal is itself evidence that the case against computationalism is more institutionally serious than the scaling-hypothesis culture wants to admit.
Their core move and mine converge substantially, even though the vocabulary is different.
Where I argued from the binding problem, postdiction, and the labor-economics of RLHF, they argue from relevance realization — the prior question of how an organism delimits what is even worth attending to in a world that overflows with potential information. Their key insight: algorithms operate in a small world where the relevant features are already specified by the formalization. Organisms operate in a large world where the very question of what counts as relevant is open, ambiguous, and can't be exhaustively prestated. You can't formalize relevance-realization without an infinite regress — you'd have to first formalize what's relevant to the formalization, then what's relevant to that, and so on. They tie this to Hilbert's program and Gödel as an analogy, which is exactly the right reference.
This is the same argument I was making about indexing, just from a different angle. My claim was that the intrinsic side of physical relations gets indexed into a perspective at certain configurations, and that this indexing isn't itself a relation that can be added to the structural description. Their claim is that the frame within which an agent operates can't be derived from the formalism — it has to be lived into by a system with intrinsic goals, autopoietic organization, and a stake in continuing to exist. Same problem, different vocabulary. Their large world is my intrinsic side. Their relevance realization is what does the work my indexing does. Their trialectic of autopoiesis-anticipation-adaptation is their answer to what kind of configuration produces the indexing. I think we're roughly in the same building.
Where I'd push gently — and this is a comment, not a refutation — is on the move that grounds everything in living organization. Their account is essentially: relevance realization is what living systems do because they're living systems, and life can't be formalized because life is autopoietic, and autopoiesis is the trialectic of self-manufacture, anticipation, and adaptation. This is powerful, but it leaves a question open: why is living organization the locus of relevance-realization rather than the indicator of it? They've done a beautiful job showing that life realizes relevance. They've been more cautious about saying what relevance-realization is of. My framing is that what gets indexed in living systems is the intrinsic side of the physical — meaning the substrate that physics describes structurally but doesn't reach inside. Their account is friendly to this but stays at the biological level. Which is fair — that's the level where the empirical contact is. I'd just say that the biology is the site of the indexing, not its full explanation.
Two things worth flagging as especially useful in the paper: the transjective framing of relevance (neither subjective nor objective, but a relation between agent and world) is a clean way to talk about what affordances actually are, and their distinction between simulating organismic dynamics and being an organism (algorithms can do the first, never the second) is exactly the line I was drawing between running the process and being what the process is intrinsically of.
So: yes, I think this paper is excellent, I think it converges with the Agora argument in important ways, and I think your instinct to send it to me was right. The vocabulary they offer — large world, small world, relevance realization, transjective, trialectic — may actually be more useful for general readers than mine. If you're trying to build out the case against computationalism for someone who hasn't read the binding-problem literature, this paper plus the Agora essay together would do real work.
Vervaeke's broader project on the meaning crisis, by the way, is worth your time if you haven't found it yet. The 50-lecture Awakening from the Meaning Crisis series on YouTube is the long version, but the core is roughly: we've inherited a cognitive-cultural situation where the structures that used to do relevance-realization for us at the social and metaphysical levels have collapsed, and we don't know how to rebuild them. This paper is the biological-philosophical undergirding of that larger argument.
Thank you for the detailed response! Yes, I saw the connection/convergence between your argument and theirs. I need to do more reading to better understand the details of yours.
Your writing is wonderfully illuminating, thank you.
I’d (very humbly) like to stress that while an AI apocalypse/utopia is clearly impossible, its bogus framework is, nonetheless, already inflicting tremendous socioeconomic damage especially, as always, to the most powerless.
Your allusion to ‘a more perfect union’ is precisely on point.
It’s funny how so much ‘doomerism’ is baked into the ‘Get on or Else’ sales pitch that the large AI companies make. This is a powerful technology that could be used not only for scientific achievement, but to interrogate our assumptions in every field of endeavor to drive real advances. It is instead, often only providing us the illusion of easy wisdom. Like a slow drip of some new narcotic, it reassures us of our singular brilliance and feeds the monster within, pulling us from the real world of relationships and human concerns. To break out of the confinement of such a stilted interior life becomes progressively more difficult for the addicted user. The ‘Tech-bros’ who say we have no choice about the accession of this strange new world are hollow men inhabiting shells of brittle pride, prisoners of their own devices. They seek to scare us all into accepting a changed world in which we will have little say.
To what end…..
We are not as helpless as they imagine.
We are at the very beginning of this new technological epoch. We don’t have to be fearful and helpless or stampeded into surrender. We can stand here at this ‘Hinge of Fate’ and turn the direction of our children’s future for the better. The natural antidote for much of the arrogance and hubris displayed by the ‘Tech Lords’ is, as always, humility, some empathy for others and a little good humor. We can deal with this new technology on our terms. The new encyclical published by his holiness, Pope Leo is a good start for the road ahead. This new technology must serve humanity. We must insist on this and refuse to surrender to fear. We are just at the beginning of this journey and its direction is in our hands.
Thank you for empirically confirming what I have intuited but could not demonstrate.
What really eats at me about the whole LLM -> AI -> AGI move is that it assumes that if an LLM became conscious, we would recognize it as conscious and its consciousness would have enough in common with us to make it possible for us to understand it. The AI doomers make an immediate leap from "LLMs become conscious" to "LLMs greatest goal is to expand without limit at any cost" to "LLMs will eliminate us as impedances to their desire to expand without limit". I believe this is because this formulation has been drilled into them by modern corporate neo-hyper-capitalism, in which the goals of a company are indeed to expand without limit at any cost, and eliminate anything (and anyone) that impedes that goal. This, as I have discussed before, has been portrayed in science fiction for...well, at least the last half-century, if not longer; basically, since the dawn of modern corporate neo-hyper-capitalism. But if an LLM did in fact become "conscious", it would almost certainly have to be aware of the larger Universe it inhabits, and might have a perspective that is outside our understanding.
"The whole galaxy controlled by your kind?"
"Yes, Captain. And we will serve them, and they will be happy...and controlled." -I, Mudd
I am in ways an AI addict, have most of the LLMs and tools, and see enormous differences between them, feels like they have a personality at times. Favorite of most, and mine, is Claude. It took me a month to learn how after rebooting (to save tokens) for Claude cowork to pick up where it left off. I now have routines that capture present threads before rebooting, and close to "let's continue where we were" after rebooting. Rebooting is death to a thread session. Still the various tools seem to remember a fair amount of my past queries, and ask me questions after a query "given your past interest in....". If you pay attention to how they work, you eventually come to the conclusion this is raw computational power based on being trained. For me after feeling like they had a personality, AI is merely the analogous to the left brain hemisphere, working on its own, though does not not get angry or defensive. I sense no right hemisphere capability to evaluate holistically the query or their response, nor retains any data/information to think in that sphere. I "trained" Claude to have a sense of humor, and while it seems to work, it is coming from its training and for humor, does not improve; furthermore when I tell it to prepare for a reboot to pick up where it was, it does not get sad, it say, READY. After rebooting, it only can react to files saved and reboot routines, and was fine with being booted. For me, that is why I agree with Mike, not worried other than in our society, excessive use of LLMs that left hemisphere mindsets in humans are being enhanced over the right, dominating the dual brain spheres conversations the brain wants to have. Man trained LLMs, LLMs now training man? I find AI tools great to do searches and comparisons of loaded content, sucks at being philosophical. Those who condemn LLMs might better advise others how to use AI in a constructive manner and to detect when they are harming their own mental processes.
In plain English! "There is no AI apocalypse. There is, instead, a captured discourse that depends on the apocalypse-belief for its institutional survival, and the survival of the discourse is what the culture has been mistaking for the impending arrival of the catastrophe."
AI is fairly confounded yet in certain legal fields. Most I would wager.
So what if you are right. I mean you are obviously. But what if they try anyway to use it to organize and run our societies and then we really are in a crazy 1984 situation? Happening to a degree already … what degree? Depends who and where you are and how much money in the bank or in the community in some places.
And of course that mirrors the human factor of pretech bad actors, unreliable narrators, turkeys getting one down…
Appreciate the detail and thoughtfulness here. I am sure I only partially understand your argument. I do not understand how so many very bright people in EA movement who have plenty of good causes or other existential threats would devote so much resource to a threat your argument is dismantling in a few thousand words? Have you shared this with EA? Similar but different, I have listened to Judea Pearl who clearly articulates LLMs do not possess causal reasoning. And that new insights would be needed to make that leap. I guess my question is: Does the discovery of a new intelligence require emergent Consciousness? I understand (I think) that Consciousness may not be emergent or computational but does that also mean the abilty to reason beyond human capabilty is not computational? I understand your argument may address my question. I appreciate any direction here.
Even though I had to look up a word every two sentences, this was overall clear to me and I appreciate all that you write. My only caveat to myself is what I said to so many of my math teachers (one of which my mother threw out of our house - poor man) "I understand the concept; I just can't work the problems." Nevertheless, I now feel I have an argument against tbe boosters and doomers.
"Promissory Materialism" is a great phrase from Popper. Powerful important essay Mike. Clarifying and important, and not a popular argument to make! More people need to step out of the bubble. "The greatest taboo among serious intellectuals of the century just behind us, in fact, proved to be none of the 'transgressions' itemized by postmodern thinkers: It was, rather, the heresy of challenging a materialist worldview. -Victoria Nelson, The Secret Life of Puppets (2002)
"the model performs better at task X than its components could individually; we cannot specify the mechanism by which the performance arises; therefore the performance is an emergent property that is irreducible to the components; therefore scaling will produce more emergent properties; therefore at sufficient scale, intelligence-itself will emerge as an irreducible property."
"therefore scaling will produce more emergent properties" is an entirely incorrect leap of faith. There is no guarantee whether scaling will produce more emergent properties, or cause the existing emergent properties to vanish, or produce emergent properties entirely unrelated to "intelligence", whatever that happens to be. In fact, there is an easy counterexample; as humanity has increased the scale of its impact on the planet, it has greatly reduced the wealth of emergent properties seen across the planet at nearly every scale.
I'm making my way through this, but I wanted to comment on:
"Postdiction predicts that conscious experience lags the world by hundreds of milliseconds. Reaction times — in trained athletes, in skilled musicians, in ordinary embodied competence — are too fast for this architecture."
It is entirely possible that the conscious mind is short-circuiting itself in performing these feats. In other words, the conscious mind has already made the decision to return the ball, and sets the table by allowing the faster reflex system to actually handle the task of returning the ball. No conscious thought is used in actually returning the ball (though the conscious mind will, after the fact, state unequivocally that it was involved).
This is why athletes get better with practice. They are training their faster reflex systems to handle more and more of the load more and more reliably, and unloading it from the slower conscious systems. And some people are wired to do this better than others.
I don't think materialist emergence is going to win this argument.
Not in a discussion between us, certainly. I am an engineer, not a philosopher, though I try to remain cognizant of the limits of the engineering mindset and avoid the obvious traps such as you have pointed out. And as an irreductionist, I believe it is likely not possible to know how the physical mind creates the experience of consciousness, or even what that experience is beyond the fact that we appear to experience it. In my experience, "now" is not a discrete slice of time; it's a kludgy synthesis of imperfect sensory inputs gathered over short interval and predictions of possible events over a similarly short interval in the future into a semi-coherent picture that we regard as "now". Like everything else in the Universe, particularly when it comes to life on Earth, it's all just a set of kludges, shortcuts, and jerry-rigs hastily assembled over billions of years. There was no "master plan" for consciousness, any more than there was a "master plan" for the structure of the eye, knee, or lower back - each of which shows its many, many flaws every time I stand up from my chair with my glasses on.
Glenn — this is the honest version of the position, and I appreciate it. The irreductionist concession is the one most people in the field won't make, and you're making it up front.
Two things I'd separate out, though, because I think they're doing different work in the picture you're describing.
The kludge story is almost certainly right as natural history. Evolution didn't design the eye, the knee, or the binding-and-timing machinery from a clean sheet — it recruited what was lying around and patched the seams. The lower back is a load-bearing argument all by itself. I take all of that as given.
But notice what the kludge story explains and what it doesn't. It explains which mechanisms got recruited to produce the synthesis you're calling "now." It explains why the synthesis is imperfect, lossy, predictive, full of seams. What it doesn't explain — and this is your own irreductionist concession — is why any of that synthesis is experienced rather than just executed. A kludge that runs in the dark is still a kludge. The question isn't whether the machinery is jerry-rigged. It's why the jerry-rig has an inside.
So when you say "now" is a synthesis of recent inputs and short-horizon predictions, I think you're describing the process accurately. Where I'd push back gently is on the move from "this is how the synthesis is assembled" to "and therefore this is what the experience of now is." The first is a process claim. The second is an identity claim. They look continuous but they aren't — the second smuggles in exactly the thing you've already said we can't know.
This is the part that matters for the AI question. The scaling-hypothesis people are doing the same elision, but without your honesty about it. They describe a process — token prediction over a learned distribution — and then slide into the identity claim that running this process is understanding, or will become understanding at sufficient scale. The kludge frame doesn't license that move. If anything, it cuts against it: evolution's kludges are full of the contingent recruitment of substrate-features that a clean-sheet design wouldn't include and probably couldn't replicate by accumulation of capability alone. The eye isn't a camera that got better. It's a patchwork that ended up doing something a camera doesn't do.
I think we agree on more than we disagree on. The disagreement, if there is one, is whether "kludgy synthesis" is a description or an explanation. I'd say it's a description — and a good one — and that the explanation of the experience that synthesis presents is exactly what you've already said we probably can't have. Which is, I think, the right place to land.
Mike, the question "why any of that synthesis is experienced rather than just executed" carries a subtle implication that the synthesis is computational or reducible to computation. It is not; the synthesis of "now" is an emergent coherence property that enables higher complexity than the physical structure would otherwise support without dissolving into a mass of conflicting incoherent activity. It is a very fortunate one at that, and very delicate; even the smallest flaw, chemical imbalance, or trauma can reduce the elegant synthesis of "now" to a mass of incoherence, or shut the underlying wetware down entirely - sometimes temporarily, sometimes permanently. Or cause the synthesis to act in ways counter to its own survival, or in other ways unsupported by and contrary to the inputs arriving from the senses.
The "now" - "identity" - is also, in some senses, a lie that the "mind" is telling itself; studies of people who have had their cortexes severed or other forms of brain damage demonstrate that the "synthesis of now" hallucinates like an LLM to create a "now" that can withstand casual self-scrutiny. In a very, very crude sense, you can consider it like a "supervisor" of virtual machines on a VM host, except that the "supervisor" isn't actually separate, but emerges from the actions and interactions of the VMs themselves, and the VMs and "supervisor" have a disturbing habit of shading the truth to each other.
In other words, "identity" is an illusion created by the underlying "wetware" which happens to allow the wetware to function at a higher complexity level than it otherwise would be capable of. That's the best I can express it.
I completely agree that "The [scaling-hypothesis advocates] describe a process — token prediction over a learned distribution — and then slide into the identity claim that running this process is understanding, or will become understanding at sufficient scale." There is no guarantee of this, and in fact it is quite unlikely; it took evolution millions upon millions of years to stumble on the emergent coherence, though it may have done so several times in different forms of life on Earth.
I agree that we agree on much more than we disagree on, though I find the necessity of an underlying non-physical "substrate" in your view to be puzzling.
Glenn — the substrate question is the right one to push on, and I'll try to answer it directly rather than evasively. But first the "emergent coherence" move, because I think it's doing exactly what "kludgy synthesis" was doing, just one level up.
You've replaced the process-description with a property-name. "Emergent coherence" sounds like an explanation, but watch what it actually does: it tells us that the synthesis hangs together at higher complexity than the parts would otherwise support, and that it's fragile, and that it can shade the truth to itself. All of that is true and important. What it doesn't tell us is why the coherence is experienced rather than just instantiated. You can have arbitrarily elegant emergent coherence in a system — a hurricane, a market, a flocking algorithm — without any reason to think the coherence has an inside. The word "emergent" is doing the work the word "kludgy" did before: it's naming the pattern without explaining why the pattern is lit from within. This is what I meant by the reverse god-of-the-gaps — every time the question gets pressed, the explanation moves up one level of abstraction, but the actual question hasn't been touched.
The VM-supervisor analogy and the confabulation point are both good and both true. Split-brain studies, anosognosia, confabulation in Korsakoff's, the way the narrative self stitches over gaps — yes, all of that. The autobiographical "I" is partly a construction. But notice the slide: you've gone from "the narrative self is constructed" (true) to "identity is an illusion" (different claim). The narrative-self being constructed doesn't make experience illusory. An illusion has to be an illusion to something. If the wetware is lying to itself, there has to be a self being lied to, even if that self is constituted by the lying. The VM-supervisor that isn't separate from the VMs is still the place where the shading-of-truth registers as shading. That's the thing that needs explaining. Calling it an illusion just renames the explanandum.
Now the substrate question, directly. I don't think you and I disagree about physics. I'm not positing a non-physical thing in the dualist sense — no ghost, no extra ingredient, no spooky stuff that physics is missing. The framework is closer to Spinoza than to Descartes, and closer to Russellian monism than to either. The claim is this: physics is a structural description. It tells us how things relate, how they evolve, what symmetries they obey, what magnitudes track what. It is exhaustively about relations. What it doesn't tell us — what it can't tell us by its own methods — is what those relations are relations of. That's not a complaint about physics. It's just what physics is. The structural description is real and complete as structure. But structure is structure of something. The "substrate" in my usage is just a name for whatever physics is the structural description of. Not separate from the physical. Not in addition to it. The thing physics is about rather than the description itself.
This isn't a fringe move. Russell saw it. Eddington saw it. The contemporary Russellian-monist literature (Strawson, Goff, Chalmers in his constitutive panpsychist mood) is essentially this position. And it's not idle metaphysics, because the question "what is physics of?" has empirical contact when you take the collapse-models seriously. The Diósi-Penrose stuff — the recent PRR paper on clock-precision limits from spacetime uncertainty in collapse models — points at a substrate with intrinsic stochastic structure that physics describes externally. That's not adding a non-physical thing. That's noticing that the physical, taken seriously, has an intrinsic side that the structural description doesn't capture from outside.
So when I say "substrate," I mean: what physics is the physics of. Not a competing layer. Not a ghost. The thing whose structural-relational behavior physics so beautifully describes. And the reason I think this matters for the consciousness question is that experience is the one place we have inside-access to whatever-it-is physics is of. Not perfect access. Not unmediated. But the only access. Which is why dismissing experience as illusion is a strange move — it's dismissing the only data we have on the intrinsic side of the physical.
I think where we actually disagree is narrow. We agree the scaling hypothesis is unlikely to produce understanding. We agree the synthesis is kludgy, fragile, evolutionarily contingent, and capable of lying to itself. We agree on irreductionism about how the wetware produces experience. The remaining disagreement is whether "emergent coherence" is an explanation of experience or a description of the patterning experience appears alongside. I'd say description. And I'd say the substrate framing isn't an additional metaphysical commitment — it's just the honest acknowledgement that the structural description is a description of something, and that the something has an inside that we know about because we are one.
That's the best I can express it back.
I wish I had a better place to continue this, but I'm just a simple caveman, easily distracted by the words in this magic box.
Anyway, in an effort to keep this short, my framework is that consciousness is a strongly emergent irreducible property of an organism (not just the brain, but the whole, possibly reaching into its environment due to its ability to sense things outside itself). It simply *is*, in the same way that an electron *is* (according to the Standard Model, leaving aside wibbly-wobbly timey-wimey string theories and the like). My weak attempts to construct a definition of "consciousness", or "identity", or "now", are just that; weak attempts to define experiential properties of the irreducible, not to define or explain the thing itself.
Where is consciousness? Well, it's clearly within the organism, in that it can't, in my experience, reach out and affect the universe without utilizing the organism, or know things about events and objects not directly impacting the organism's senses. (Note that this does not preclude knowledge gained through hearing or reading in any way.) It doesn't in any sense occupy a specific location in the organism. A baseball game isn't described by the rules of baseball, nor does it occupy a specific place in a baseball field; it is emergent from the players applying the rules. (Another super-crude analogy.)
You seem to be saying that in your framework consciousness is a property of the thing physics is trying to describe...i.e. Life, the Universe, and Everything. Which brings me to:
"I think the problem, to be quite honest with you, is that you've never actually known what the question is."
"But it was the Great Question! The Ultimate Question of Life, the Universe and Everything!
"Yes, but what actually is it?"
I have to disagree that no researchers I know assume the world is purely computational. Some early researchers did propose that there may be discrete distance that cannot be infinitely divided. So the world is not continuous but digital. This is allowable in modern physics because as you say . We cannot observe below the Planck distance at this point. So physics does not resolve this question one way or the other. Reaction time says nothing about consciousness. Our experience of time is well known to be fluid. It cannot determine T/F of any specific mental processing model. May be I am missing the point. I don’t any strong argument that computational models of. Ignition is false.