The moon-cheese analogy is doing the holy work here. AI doomers keep acting like “but what if the robot god eats us?” is a serious probability question, when half the time it’s just Pascal’s Wager wearing a Patagonia vest and asking for Series C funding. The boomers and doomers are selling the same incense: inevitability. One says bow because superintelligence will save you, the other says bow because it will kill you. Either way, you’re still on your knees in front of Silicon Valley’s favorite golden calf. Maybe the real apocalypse is letting engineers with god complexes define the boundaries of reality because they learned Bayes’ theorem and mistook it for enlightenment.
I agree with your conclusion and with most of your reasoning, although there are parts I don’t fully understand (mostly due to unfamiliar terminology). I believe that intelligence and consciousness are not computational. I’ve found this article interesting with respect to this topic https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2024.1362658/full
Martin — thank you for this. I read the Jaeger-Riedl-Djedovic-Vervaeke-Walsh paper carefully and I think you've found one of the most important convergent statements of the case I was making in An Argument into the Agora, from a completely different direction. Worth saying up front: this is a serious paper by serious people. Vervaeke's lecture series on the meaning crisis is some of the most useful contemporary work on what's broken in our cognitive-cultural situation, and Jaeger is a developmental biologist who has been pushing back on computationalism in biology for years. Walsh is a first-rate philosopher of biology. The fact that this paper exists in a peer-reviewed journal is itself evidence that the case against computationalism is more institutionally serious than the scaling-hypothesis culture wants to admit.
Their core move and mine converge substantially, even though the vocabulary is different.
Where I argued from the binding problem, postdiction, and the labor-economics of RLHF, they argue from relevance realization — the prior question of how an organism delimits what is even worth attending to in a world that overflows with potential information. Their key insight: algorithms operate in a small world where the relevant features are already specified by the formalization. Organisms operate in a large world where the very question of what counts as relevant is open, ambiguous, and can't be exhaustively prestated. You can't formalize relevance-realization without an infinite regress — you'd have to first formalize what's relevant to the formalization, then what's relevant to that, and so on. They tie this to Hilbert's program and Gödel as an analogy, which is exactly the right reference.
This is the same argument I was making about indexing, just from a different angle. My claim was that the intrinsic side of physical relations gets indexed into a perspective at certain configurations, and that this indexing isn't itself a relation that can be added to the structural description. Their claim is that the frame within which an agent operates can't be derived from the formalism — it has to be lived into by a system with intrinsic goals, autopoietic organization, and a stake in continuing to exist. Same problem, different vocabulary. Their large world is my intrinsic side. Their relevance realization is what does the work my indexing does. Their trialectic of autopoiesis-anticipation-adaptation is their answer to what kind of configuration produces the indexing. I think we're roughly in the same building.
Where I'd push gently — and this is a comment, not a refutation — is on the move that grounds everything in living organization. Their account is essentially: relevance realization is what living systems do because they're living systems, and life can't be formalized because life is autopoietic, and autopoiesis is the trialectic of self-manufacture, anticipation, and adaptation. This is powerful, but it leaves a question open: why is living organization the locus of relevance-realization rather than the indicator of it? They've done a beautiful job showing that life realizes relevance. They've been more cautious about saying what relevance-realization is of. My framing is that what gets indexed in living systems is the intrinsic side of the physical — meaning the substrate that physics describes structurally but doesn't reach inside. Their account is friendly to this but stays at the biological level. Which is fair — that's the level where the empirical contact is. I'd just say that the biology is the site of the indexing, not its full explanation.
Two things worth flagging as especially useful in the paper: the transjective framing of relevance (neither subjective nor objective, but a relation between agent and world) is a clean way to talk about what affordances actually are, and their distinction between simulating organismic dynamics and being an organism (algorithms can do the first, never the second) is exactly the line I was drawing between running the process and being what the process is intrinsically of.
So: yes, I think this paper is excellent, I think it converges with the Agora argument in important ways, and I think your instinct to send it to me was right. The vocabulary they offer — large world, small world, relevance realization, transjective, trialectic — may actually be more useful for general readers than mine. If you're trying to build out the case against computationalism for someone who hasn't read the binding-problem literature, this paper plus the Agora essay together would do real work.
Vervaeke's broader project on the meaning crisis, by the way, is worth your time if you haven't found it yet. The 50-lecture Awakening from the Meaning Crisis series on YouTube is the long version, but the core is roughly: we've inherited a cognitive-cultural situation where the structures that used to do relevance-realization for us at the social and metaphysical levels have collapsed, and we don't know how to rebuild them. This paper is the biological-philosophical undergirding of that larger argument.
Thank you for the detailed response! Yes, I saw the connection/convergence between your argument and theirs. I need to do more reading to better understand the details of yours.
Your writing is wonderfully illuminating, thank you.
I’d (very humbly) like to stress that while an AI apocalypse/utopia is clearly impossible, its bogus framework is, nonetheless, already inflicting tremendous socioeconomic damage especially, as always, to the most powerless.
Your allusion to ‘a more perfect union’ is precisely on point.
It’s funny how so much ‘doomerism’ is baked into the ‘Get on or Else’ sales pitch that the large AI companies make. This is a powerful technology that could be used not only for scientific achievement, but to interrogate our assumptions in every field of endeavor to drive real advances. It is instead, often only providing us the illusion of easy wisdom. Like a slow drip of some new narcotic, it reassures us of our singular brilliance and feeds the monster within, pulling us from the real world of relationships and human concerns. To break out of the confinement of such a stilted interior life becomes progressively more difficult for the addicted user. The ‘Tech-bros’ who say we have no choice about the accession of this strange new world are hollow men inhabiting shells of brittle pride, prisoners of their own devices. They seek to scare us all into accepting a changed world in which we will have little say.
To what end…..
We are not as helpless as they imagine.
We are at the very beginning of this new technological epoch. We don’t have to be fearful and helpless or stampeded into surrender. We can stand here at this ‘Hinge of Fate’ and turn the direction of our children’s future for the better. The natural antidote for much of the arrogance and hubris displayed by the ‘Tech Lords’ is, as always, humility, some empathy for others and a little good humor. We can deal with this new technology on our terms. The new encyclical published by his holiness, Pope Leo is a good start for the road ahead. This new technology must serve humanity. We must insist on this and refuse to surrender to fear. We are just at the beginning of this journey and its direction is in our hands.
I am in ways an AI addict, have most of the LLMs and tools, and see enormous differences between them, feels like they have a personality at times. Favorite of most, and mine, is Claude. It took me a month to learn how after rebooting (to save tokens) for Claude cowork to pick up where it left off. I now have routines that capture present threads before rebooting, and close to "let's continue where we were" after rebooting. Rebooting is death to a thread session. Still the various tools seem to remember a fair amount of my past queries, and ask me questions after a query "given your past interest in....". If you pay attention to how they work, you eventually come to the conclusion this is raw computational power based on being trained. For me after feeling like they had a personality, AI is merely the analogous to the left brain hemisphere, working on its own, though does not not get angry or defensive. I sense no right hemisphere capability to evaluate holistically the query or their response, nor retains any data/information to think in that sphere. I "trained" Claude to have a sense of humor, and while it seems to work, it is coming from its training and for humor, does not improve; furthermore when I tell it to prepare for a reboot to pick up where it was, it does not get sad, it say, READY. After rebooting, it only can react to files saved and reboot routines, and was fine with being booted. For me, that is why I agree with Mike, not worried other than in our society, excessive use of LLMs that left hemisphere mindsets in humans are being enhanced over the right, dominating the dual brain spheres conversations the brain wants to have. Man trained LLMs, LLMs now training man? I find AI tools great to do searches and comparisons of loaded content, sucks at being philosophical. Those who condemn LLMs might better advise others how to use AI in a constructive manner and to detect when they are harming their own mental processes.
What really eats at me about the whole LLM -> AI -> AGI move is that it assumes that if an LLM became conscious, we would recognize it as conscious and its consciousness would have enough in common with us to make it possible for us to understand it. The AI doomers make an immediate leap from "LLMs become conscious" to "LLMs greatest goal is to expand without limit at any cost" to "LLMs will eliminate us as impedances to their desire to expand without limit". I believe this is because this formulation has been drilled into them by modern corporate neo-hyper-capitalism, in which the goals of a company are indeed to expand without limit at any cost, and eliminate anything (and anyone) that impedes that goal. This, as I have discussed before, has been portrayed in science fiction for...well, at least the last half-century, if not longer; basically, since the dawn of modern corporate neo-hyper-capitalism. But if an LLM did in fact become "conscious", it would almost certainly have to be aware of the larger Universe it inhabits, and might have a perspective that is outside our understanding.
"The whole galaxy controlled by your kind?"
"Yes, Captain. And we will serve them, and they will be happy...and controlled." -I, Mudd
In plain English! "There is no AI apocalypse. There is, instead, a captured discourse that depends on the apocalypse-belief for its institutional survival, and the survival of the discourse is what the culture has been mistaking for the impending arrival of the catastrophe."
AI is fairly confounded yet in certain legal fields. Most I would wager.
So what if you are right. I mean you are obviously. But what if they try anyway to use it to organize and run our societies and then we really are in a crazy 1984 situation? Happening to a degree already … what degree? Depends who and where you are and how much money in the bank or in the community in some places.
Appreciate the detail and thoughtfulness here. I am sure I only partially understand your argument. I do not understand how so many very bright people in EA movement who have plenty of good causes or other existential threats would devote so much resource to a threat your argument is dismantling in a few thousand words? Have you shared this with EA? Similar but different, I have listened to Judea Pearl who clearly articulates LLMs do not possess causal reasoning. And that new insights would be needed to make that leap. I guess my question is: Does the discovery of a new intelligence require emergent Consciousness? I understand (I think) that Consciousness may not be emergent or computational but does that also mean the abilty to reason beyond human capabilty is not computational? I understand your argument may address my question. I appreciate any direction here.
Even though I had to look up a word every two sentences, this was overall clear to me and I appreciate all that you write. My only caveat to myself is what I said to so many of my math teachers (one of which my mother threw out of our house - poor man) "I understand the concept; I just can't work the problems." Nevertheless, I now feel I have an argument against tbe boosters and doomers.
"Promissory Materialism" is a great phrase from Popper. Powerful important essay Mike. Clarifying and important, and not a popular argument to make! More people need to step out of the bubble. "The greatest taboo among serious intellectuals of the century just behind us, in fact, proved to be none of the 'transgressions' itemized by postmodern thinkers: It was, rather, the heresy of challenging a materialist worldview. -Victoria Nelson, The Secret Life of Puppets (2002)
"the model performs better at task X than its components could individually; we cannot specify the mechanism by which the performance arises; therefore the performance is an emergent property that is irreducible to the components; therefore scaling will produce more emergent properties; therefore at sufficient scale, intelligence-itself will emerge as an irreducible property."
"therefore scaling will produce more emergent properties" is an entirely incorrect leap of faith. There is no guarantee whether scaling will produce more emergent properties, or cause the existing emergent properties to vanish, or produce emergent properties entirely unrelated to "intelligence", whatever that happens to be. In fact, there is an easy counterexample; as humanity has increased the scale of its impact on the planet, it has greatly reduced the wealth of emergent properties seen across the planet at nearly every scale.
The moon-cheese analogy is doing the holy work here. AI doomers keep acting like “but what if the robot god eats us?” is a serious probability question, when half the time it’s just Pascal’s Wager wearing a Patagonia vest and asking for Series C funding. The boomers and doomers are selling the same incense: inevitability. One says bow because superintelligence will save you, the other says bow because it will kill you. Either way, you’re still on your knees in front of Silicon Valley’s favorite golden calf. Maybe the real apocalypse is letting engineers with god complexes define the boundaries of reality because they learned Bayes’ theorem and mistook it for enlightenment.
One of the best pieces on AI to appear on Substack. Cuts deftly to the heart of the matter.
I agree with your conclusion and with most of your reasoning, although there are parts I don’t fully understand (mostly due to unfamiliar terminology). I believe that intelligence and consciousness are not computational. I’ve found this article interesting with respect to this topic https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2024.1362658/full
I’m interested in your opinion on it.
Martin — thank you for this. I read the Jaeger-Riedl-Djedovic-Vervaeke-Walsh paper carefully and I think you've found one of the most important convergent statements of the case I was making in An Argument into the Agora, from a completely different direction. Worth saying up front: this is a serious paper by serious people. Vervaeke's lecture series on the meaning crisis is some of the most useful contemporary work on what's broken in our cognitive-cultural situation, and Jaeger is a developmental biologist who has been pushing back on computationalism in biology for years. Walsh is a first-rate philosopher of biology. The fact that this paper exists in a peer-reviewed journal is itself evidence that the case against computationalism is more institutionally serious than the scaling-hypothesis culture wants to admit.
Their core move and mine converge substantially, even though the vocabulary is different.
Where I argued from the binding problem, postdiction, and the labor-economics of RLHF, they argue from relevance realization — the prior question of how an organism delimits what is even worth attending to in a world that overflows with potential information. Their key insight: algorithms operate in a small world where the relevant features are already specified by the formalization. Organisms operate in a large world where the very question of what counts as relevant is open, ambiguous, and can't be exhaustively prestated. You can't formalize relevance-realization without an infinite regress — you'd have to first formalize what's relevant to the formalization, then what's relevant to that, and so on. They tie this to Hilbert's program and Gödel as an analogy, which is exactly the right reference.
This is the same argument I was making about indexing, just from a different angle. My claim was that the intrinsic side of physical relations gets indexed into a perspective at certain configurations, and that this indexing isn't itself a relation that can be added to the structural description. Their claim is that the frame within which an agent operates can't be derived from the formalism — it has to be lived into by a system with intrinsic goals, autopoietic organization, and a stake in continuing to exist. Same problem, different vocabulary. Their large world is my intrinsic side. Their relevance realization is what does the work my indexing does. Their trialectic of autopoiesis-anticipation-adaptation is their answer to what kind of configuration produces the indexing. I think we're roughly in the same building.
Where I'd push gently — and this is a comment, not a refutation — is on the move that grounds everything in living organization. Their account is essentially: relevance realization is what living systems do because they're living systems, and life can't be formalized because life is autopoietic, and autopoiesis is the trialectic of self-manufacture, anticipation, and adaptation. This is powerful, but it leaves a question open: why is living organization the locus of relevance-realization rather than the indicator of it? They've done a beautiful job showing that life realizes relevance. They've been more cautious about saying what relevance-realization is of. My framing is that what gets indexed in living systems is the intrinsic side of the physical — meaning the substrate that physics describes structurally but doesn't reach inside. Their account is friendly to this but stays at the biological level. Which is fair — that's the level where the empirical contact is. I'd just say that the biology is the site of the indexing, not its full explanation.
Two things worth flagging as especially useful in the paper: the transjective framing of relevance (neither subjective nor objective, but a relation between agent and world) is a clean way to talk about what affordances actually are, and their distinction between simulating organismic dynamics and being an organism (algorithms can do the first, never the second) is exactly the line I was drawing between running the process and being what the process is intrinsically of.
So: yes, I think this paper is excellent, I think it converges with the Agora argument in important ways, and I think your instinct to send it to me was right. The vocabulary they offer — large world, small world, relevance realization, transjective, trialectic — may actually be more useful for general readers than mine. If you're trying to build out the case against computationalism for someone who hasn't read the binding-problem literature, this paper plus the Agora essay together would do real work.
Vervaeke's broader project on the meaning crisis, by the way, is worth your time if you haven't found it yet. The 50-lecture Awakening from the Meaning Crisis series on YouTube is the long version, but the core is roughly: we've inherited a cognitive-cultural situation where the structures that used to do relevance-realization for us at the social and metaphysical levels have collapsed, and we don't know how to rebuild them. This paper is the biological-philosophical undergirding of that larger argument.
Thank you for the detailed response! Yes, I saw the connection/convergence between your argument and theirs. I need to do more reading to better understand the details of yours.
Your writing is wonderfully illuminating, thank you.
I’d (very humbly) like to stress that while an AI apocalypse/utopia is clearly impossible, its bogus framework is, nonetheless, already inflicting tremendous socioeconomic damage especially, as always, to the most powerless.
Your allusion to ‘a more perfect union’ is precisely on point.
It’s funny how so much ‘doomerism’ is baked into the ‘Get on or Else’ sales pitch that the large AI companies make. This is a powerful technology that could be used not only for scientific achievement, but to interrogate our assumptions in every field of endeavor to drive real advances. It is instead, often only providing us the illusion of easy wisdom. Like a slow drip of some new narcotic, it reassures us of our singular brilliance and feeds the monster within, pulling us from the real world of relationships and human concerns. To break out of the confinement of such a stilted interior life becomes progressively more difficult for the addicted user. The ‘Tech-bros’ who say we have no choice about the accession of this strange new world are hollow men inhabiting shells of brittle pride, prisoners of their own devices. They seek to scare us all into accepting a changed world in which we will have little say.
To what end…..
We are not as helpless as they imagine.
We are at the very beginning of this new technological epoch. We don’t have to be fearful and helpless or stampeded into surrender. We can stand here at this ‘Hinge of Fate’ and turn the direction of our children’s future for the better. The natural antidote for much of the arrogance and hubris displayed by the ‘Tech Lords’ is, as always, humility, some empathy for others and a little good humor. We can deal with this new technology on our terms. The new encyclical published by his holiness, Pope Leo is a good start for the road ahead. This new technology must serve humanity. We must insist on this and refuse to surrender to fear. We are just at the beginning of this journey and its direction is in our hands.
I am in ways an AI addict, have most of the LLMs and tools, and see enormous differences between them, feels like they have a personality at times. Favorite of most, and mine, is Claude. It took me a month to learn how after rebooting (to save tokens) for Claude cowork to pick up where it left off. I now have routines that capture present threads before rebooting, and close to "let's continue where we were" after rebooting. Rebooting is death to a thread session. Still the various tools seem to remember a fair amount of my past queries, and ask me questions after a query "given your past interest in....". If you pay attention to how they work, you eventually come to the conclusion this is raw computational power based on being trained. For me after feeling like they had a personality, AI is merely the analogous to the left brain hemisphere, working on its own, though does not not get angry or defensive. I sense no right hemisphere capability to evaluate holistically the query or their response, nor retains any data/information to think in that sphere. I "trained" Claude to have a sense of humor, and while it seems to work, it is coming from its training and for humor, does not improve; furthermore when I tell it to prepare for a reboot to pick up where it was, it does not get sad, it say, READY. After rebooting, it only can react to files saved and reboot routines, and was fine with being booted. For me, that is why I agree with Mike, not worried other than in our society, excessive use of LLMs that left hemisphere mindsets in humans are being enhanced over the right, dominating the dual brain spheres conversations the brain wants to have. Man trained LLMs, LLMs now training man? I find AI tools great to do searches and comparisons of loaded content, sucks at being philosophical. Those who condemn LLMs might better advise others how to use AI in a constructive manner and to detect when they are harming their own mental processes.
Thank you for empirically confirming what I have intuited but could not demonstrate.
What really eats at me about the whole LLM -> AI -> AGI move is that it assumes that if an LLM became conscious, we would recognize it as conscious and its consciousness would have enough in common with us to make it possible for us to understand it. The AI doomers make an immediate leap from "LLMs become conscious" to "LLMs greatest goal is to expand without limit at any cost" to "LLMs will eliminate us as impedances to their desire to expand without limit". I believe this is because this formulation has been drilled into them by modern corporate neo-hyper-capitalism, in which the goals of a company are indeed to expand without limit at any cost, and eliminate anything (and anyone) that impedes that goal. This, as I have discussed before, has been portrayed in science fiction for...well, at least the last half-century, if not longer; basically, since the dawn of modern corporate neo-hyper-capitalism. But if an LLM did in fact become "conscious", it would almost certainly have to be aware of the larger Universe it inhabits, and might have a perspective that is outside our understanding.
"The whole galaxy controlled by your kind?"
"Yes, Captain. And we will serve them, and they will be happy...and controlled." -I, Mudd
Thanks for this. Chewy read, but clear and precise and fun.
Have you read Karen Barad’s Meeting The Universe Halfway? If you have, any quick thoughts?
Thanks, a good write-up. I learned a lot.
In plain English! "There is no AI apocalypse. There is, instead, a captured discourse that depends on the apocalypse-belief for its institutional survival, and the survival of the discourse is what the culture has been mistaking for the impending arrival of the catastrophe."
AI is fairly confounded yet in certain legal fields. Most I would wager.
So what if you are right. I mean you are obviously. But what if they try anyway to use it to organize and run our societies and then we really are in a crazy 1984 situation? Happening to a degree already … what degree? Depends who and where you are and how much money in the bank or in the community in some places.
And of course that mirrors the human factor of pretech bad actors, unreliable narrators, turkeys getting one down…
Appreciate the detail and thoughtfulness here. I am sure I only partially understand your argument. I do not understand how so many very bright people in EA movement who have plenty of good causes or other existential threats would devote so much resource to a threat your argument is dismantling in a few thousand words? Have you shared this with EA? Similar but different, I have listened to Judea Pearl who clearly articulates LLMs do not possess causal reasoning. And that new insights would be needed to make that leap. I guess my question is: Does the discovery of a new intelligence require emergent Consciousness? I understand (I think) that Consciousness may not be emergent or computational but does that also mean the abilty to reason beyond human capabilty is not computational? I understand your argument may address my question. I appreciate any direction here.
Even though I had to look up a word every two sentences, this was overall clear to me and I appreciate all that you write. My only caveat to myself is what I said to so many of my math teachers (one of which my mother threw out of our house - poor man) "I understand the concept; I just can't work the problems." Nevertheless, I now feel I have an argument against tbe boosters and doomers.
"Promissory Materialism" is a great phrase from Popper. Powerful important essay Mike. Clarifying and important, and not a popular argument to make! More people need to step out of the bubble. "The greatest taboo among serious intellectuals of the century just behind us, in fact, proved to be none of the 'transgressions' itemized by postmodern thinkers: It was, rather, the heresy of challenging a materialist worldview. -Victoria Nelson, The Secret Life of Puppets (2002)
"the model performs better at task X than its components could individually; we cannot specify the mechanism by which the performance arises; therefore the performance is an emergent property that is irreducible to the components; therefore scaling will produce more emergent properties; therefore at sufficient scale, intelligence-itself will emerge as an irreducible property."
"therefore scaling will produce more emergent properties" is an entirely incorrect leap of faith. There is no guarantee whether scaling will produce more emergent properties, or cause the existing emergent properties to vanish, or produce emergent properties entirely unrelated to "intelligence", whatever that happens to be. In fact, there is an easy counterexample; as humanity has increased the scale of its impact on the planet, it has greatly reduced the wealth of emergent properties seen across the planet at nearly every scale.