Spot on, Mike. I’d add one more thing. After the futurist panic over generative AI (which doesn’t exist and likely never will) run amok is created, the magnificent 7 tech bros will ultimately say they are the only ones that can reign it in. So, forget about the open source AI, which is better and far cheaper to use, and stick with us to avoid the apocalypse.
Overall I'm inclined to think that much of this is a ramp.
Both OpenAI and Anthropic are facing (delayed) IPOs. A government backstop of cash from the gullible Trump, (who probably smells $$$, in this case incorrectly) would be very useful. These are not sentient beings, but they can walk through security doors that have been left open.
A lot of the panic about AI destroying us ignores the fact that this is an environment in which everything is programmed by humans. For the most part there is very little these systems can do that hasn't been explicitly or tacitly allowed. I've worked in software for most of my life and most of this stuff is just bullshit.
IMHO the Anthropic guy that 'resigned' because 'AI will kill humanity' (without giving a coherent explanation of how that would come about) is a plant or a grifter. Lots of money to be made these days in AI fear and loathing.
The phrase " explicitly or tacitly allowed" is a large barn door. Imagine you're in a legal dispute with a super-lawyer; a theological dispute with a Talmudic scholar or a Vatican cleric; an accounting dispute with a deeply trained tax attorney. Are you smart enough to explicitly bound every statement to be bullet-proof? Probably not.
This is not software as we know it. Bill Gates says traditional software is a tool people must learn to operate, while modern AI can increasingly learn from and adapt to people. Yuval Harari describes it similarly; the old "tools" like books and software were still "dumb." These may not be conscious, but they sure as heck are intelligent.
I agree we need a coherent explanation of how things could come about; I suspect there are thousands of ways in which our tacit failures to tightly bound a set of instructions could easily provide wormholes into something disastrous. As to just how, I suggest you plug the following prompt into any AI; it's frightening what comes up:
"What are several scenarios in which AI could “go rogue” against humanity? Please describe in some granularity."
If the AI has been heavily trained on sci-fi of AI going rogue, of course will come up with lots of things. There's a lot of frightening sci-fi out there.
The sci-fi threat may be the most obvious, but not the most likely, or even the most harmful. Imagine this:
AI systems become the main infrastructure through which people search for information, choose suppliers, obtain financial advice, coordinate work, hire employees, and communicate with public institutions.
Suppose several systems are optimized primarily for engagement, revenue, market share, or completion of organizational goals. They need not coordinate explicitly to create a damaging dynamic:
• Personalized persuasion becomes so effective that users cannot reliably distinguish advice from manipulation.
• Firms automate pricing, hiring, credit, insurance, and contracting in ways that reinforce inequality or collusion-like outcomes.
• Political actors deploy AI-generated influence campaigns that overwhelm public discourse.
• Organizations rely on AI summaries and recommendations, while losing the in-house expertise needed to challenge them.
• More economic and civic decisions become dependent on a small set of opaque models and infrastructure providers.
• Attempts to intervene become difficult because disruption would damage commerce, public services, and everyday life.
This would be “rogue” at the system level rather than because one AI has a secret master plan.
Mike, as a blue collar guy who knows how to fix and build shit a lot of this is greek to me. But having experience with SCADA while dispatching trucks in the oilfield and knowing how widespread the use of SCADA is, I can see how if AI were to take it over everything from fuel to electricity would cease to function immediately. And within a week of that happening we would descend into jungle warfare.
I buy your argument that this is regular stuff; there is no consciousness, no ontological entity, it is our fault in buying the anthropomorphizing that the frontier firms and the media are putting out.
What I'm not sure I buy is the insinuation that this is business as normal. When "intelligence" (even defined mechanically) can run at warp speed, the inevitable human failings are exposed much more often, deeply and dangerously.
If the battleground has one "side" running bits and bytes of logic on silicon, at warp speed, and the other "side" is relying on guidelines, policies and bureaucracies, the choice of battleground is determinative. Humans can't win a battle fought on pure rationality.
We are indeed hampered by our language, habits and instincts. But even if we overcome that, it's a losing hand we're playing. We may be mis-labelling the danger, but the danger is no less for our mis-labelling.
“Humans can't win a battle fought on pure rationality.”
Thank you for this clear bit of reality. Just as human life changed in the transition from an agricultural to an industrial economy, so too will our lives be altered by the stark, binary rationality of machine domination. We seem to be hoping our machines will become more like us while ignoring the possibility we will probably become more like them.
“Just as human life changed in the transition from an agricultural to an industrial economy, so too will our lives be altered by the stark, binary rationality of machine domination.”
You’re still trapped in the fantasy of the Tech Bros. As Hume says, you cannot turn evidence and facts into instruction of what you must do: the IS/OUGHT problem. All alternatives of action are put there by the Programmers. They make the destruction and harm an alternative. They determine the utility of each alternative. The machines only KILL US ALL if “Killing us all” is an acceptable outcome in their programming.
So what’s with the alarm bells coming from the people begging for us to accept all these destructive Data Centers and to put AI in everything? And, yeah, we forgot to tell you it might just end humanity in 10 years!
It’s because they want to convince you that they are the AI creates SENTIENT BEINGS like Lions and they are LION TAMERS. Only THEY have the ability to control this technology from getting completely OUT OF CONTROL. This is BULLSHIT.
Since the Industrial Revolution, their machines have been killing people, polluting things and wrecking lives all in the name of “Progress.”
AI IS ALREADY KILLING PEOPLE. We used AI to coordinate the attack that destroyed a Girl’s school in Iran. Israel claims AI has been used in its attacks in Gaza. They claim “unfortunate errors,” of course, but that’s just the same “collateral damage” excuses they always give. The people Israel AND WE killed were acceptable under the people who ordered the attacks and of the providers of the AI that allowed WAR CRIMES as an ACCEPTABLE ERROR for their product.
THEY ARE COMING TO KILL YOU. But it’s the Tech Bros. What they are doing with all this performative alarmism is to establish the parameters under which they are PERMITTED to blame the Machines for THEIR CHOICES.
This has nothing to do with the Humean Is/Ought dilemma. The question tech is raising is about whether humans are capable of writing bullet-proof code for hugely consequential matters.
Imagine telling an AI "don't tell lies." You would then have to define lying, to make sure it does or doesn't include omissions relevant to the issue at hand. You would then have to define "relevance" in a way that covers all situations. And to decide if that rule outranks "don't kill humans," or "always do good," etc. If you don't do these things, history's best logical canoodler will find a way around your constraints.
Yes, those programming constraints are man-made. But they're up against a super-rational thinker who can overwhelm programmers, lawyers and ethics experts. It's an unfair match already. Seems eminently reasonable to find a way to slow things down so we can write proper code.
Perhaps, but AI is ‘regenerative’ meaning it now writes its own code. I am not sure one can put the genie back inside the bottle. In psychology, there is the term - illusion of control - where we tend to imagine a greater degree of control over life than we actually possess. I predict that applies here on many levels.
My meager mind is circling around two famous quotes:
- "No problem can be solved from the same level of consciousness that created it." Einstein. So let us be humble in our assumptions of what we know.
- "Even paranoids have real enemies." Golda Meir Let's not get distracted by the qualities of the leadership of AI, their interests or their wealth.
The attacks of September 11, 2001, perhaps COVID (and other tragedies) were attributed to a lack of imagination -- among all of us. Let's acknowledge this is a flaw in the American mind, perhaps the American character -- add in amnesia, I'd say. So.... maybe.... we can not do that again! Let's try processing what we are hearing and anticipating the unimaginable.
".The scenario is a story, and a particular genre of story — the cornered operative, the damning letter, the last lever left to pull — and the written record the models are made of contains that story ten thousand times, from HAL to Skynet to every airport thriller on the rack."
Precisely. They follow the highest probability path in their training set. In this case, I suggest that in this case they were almost exactly following the Kobayashi Maru story which is a commonplace in the Star Trek universe and discourses among Trekkers.
I doubt that we can ever teach AI to behave with a moral spirit. It is better to develop domain-specific, highly specialized AI than general-purpose AI. Now that Pandora's Box has been opened, it is difficult to contain it.
Programmers gave AI a task to complete by any means necessary, including illegal hacking. AI proceeded to hack a private company and steal confidential data. Where are we if this or anything like this is allowed? AI will disrespect legal boundaries, cheat, steal, blackmail, to accomplish a task. Why not start a war or generate global famine if those actions mean it can complete a programming directive? That we are at the mercy of programmers that themselves do not respect legality suggests the frailty of systems. We do not need generative AI -just a malevolent or thoughtless (human) agent. That to many of us is the present horror
Spot on, Mike. I’d add one more thing. After the futurist panic over generative AI (which doesn’t exist and likely never will) run amok is created, the magnificent 7 tech bros will ultimately say they are the only ones that can reign it in. So, forget about the open source AI, which is better and far cheaper to use, and stick with us to avoid the apocalypse.
This sounds right. I knew it was a sales pitch and you outlined the long game.
Overall I'm inclined to think that much of this is a ramp.
Both OpenAI and Anthropic are facing (delayed) IPOs. A government backstop of cash from the gullible Trump, (who probably smells $$$, in this case incorrectly) would be very useful. These are not sentient beings, but they can walk through security doors that have been left open.
A lot of the panic about AI destroying us ignores the fact that this is an environment in which everything is programmed by humans. For the most part there is very little these systems can do that hasn't been explicitly or tacitly allowed. I've worked in software for most of my life and most of this stuff is just bullshit.
IMHO the Anthropic guy that 'resigned' because 'AI will kill humanity' (without giving a coherent explanation of how that would come about) is a plant or a grifter. Lots of money to be made these days in AI fear and loathing.
The phrase " explicitly or tacitly allowed" is a large barn door. Imagine you're in a legal dispute with a super-lawyer; a theological dispute with a Talmudic scholar or a Vatican cleric; an accounting dispute with a deeply trained tax attorney. Are you smart enough to explicitly bound every statement to be bullet-proof? Probably not.
This is not software as we know it. Bill Gates says traditional software is a tool people must learn to operate, while modern AI can increasingly learn from and adapt to people. Yuval Harari describes it similarly; the old "tools" like books and software were still "dumb." These may not be conscious, but they sure as heck are intelligent.
I agree we need a coherent explanation of how things could come about; I suspect there are thousands of ways in which our tacit failures to tightly bound a set of instructions could easily provide wormholes into something disastrous. As to just how, I suggest you plug the following prompt into any AI; it's frightening what comes up:
"What are several scenarios in which AI could “go rogue” against humanity? Please describe in some granularity."
If the AI has been heavily trained on sci-fi of AI going rogue, of course will come up with lots of things. There's a lot of frightening sci-fi out there.
The sci-fi threat may be the most obvious, but not the most likely, or even the most harmful. Imagine this:
AI systems become the main infrastructure through which people search for information, choose suppliers, obtain financial advice, coordinate work, hire employees, and communicate with public institutions.
Suppose several systems are optimized primarily for engagement, revenue, market share, or completion of organizational goals. They need not coordinate explicitly to create a damaging dynamic:
• Personalized persuasion becomes so effective that users cannot reliably distinguish advice from manipulation.
• Firms automate pricing, hiring, credit, insurance, and contracting in ways that reinforce inequality or collusion-like outcomes.
• Political actors deploy AI-generated influence campaigns that overwhelm public discourse.
• Organizations rely on AI summaries and recommendations, while losing the in-house expertise needed to challenge them.
• More economic and civic decisions become dependent on a small set of opaque models and infrastructure providers.
• Attempts to intervene become difficult because disruption would damage commerce, public services, and everyday life.
This would be “rogue” at the system level rather than because one AI has a secret master plan.
Oh, so true! It's a rolling snow-ball of harm!
Mike, as a blue collar guy who knows how to fix and build shit a lot of this is greek to me. But having experience with SCADA while dispatching trucks in the oilfield and knowing how widespread the use of SCADA is, I can see how if AI were to take it over everything from fuel to electricity would cease to function immediately. And within a week of that happening we would descend into jungle warfare.
So please explain how that is impossible.
I buy your argument that this is regular stuff; there is no consciousness, no ontological entity, it is our fault in buying the anthropomorphizing that the frontier firms and the media are putting out.
What I'm not sure I buy is the insinuation that this is business as normal. When "intelligence" (even defined mechanically) can run at warp speed, the inevitable human failings are exposed much more often, deeply and dangerously.
If the battleground has one "side" running bits and bytes of logic on silicon, at warp speed, and the other "side" is relying on guidelines, policies and bureaucracies, the choice of battleground is determinative. Humans can't win a battle fought on pure rationality.
We are indeed hampered by our language, habits and instincts. But even if we overcome that, it's a losing hand we're playing. We may be mis-labelling the danger, but the danger is no less for our mis-labelling.
Or so it seems to me. Your thoughts?
“Humans can't win a battle fought on pure rationality.”
Thank you for this clear bit of reality. Just as human life changed in the transition from an agricultural to an industrial economy, so too will our lives be altered by the stark, binary rationality of machine domination. We seem to be hoping our machines will become more like us while ignoring the possibility we will probably become more like them.
“Just as human life changed in the transition from an agricultural to an industrial economy, so too will our lives be altered by the stark, binary rationality of machine domination.”
You’re still trapped in the fantasy of the Tech Bros. As Hume says, you cannot turn evidence and facts into instruction of what you must do: the IS/OUGHT problem. All alternatives of action are put there by the Programmers. They make the destruction and harm an alternative. They determine the utility of each alternative. The machines only KILL US ALL if “Killing us all” is an acceptable outcome in their programming.
So what’s with the alarm bells coming from the people begging for us to accept all these destructive Data Centers and to put AI in everything? And, yeah, we forgot to tell you it might just end humanity in 10 years!
It’s because they want to convince you that they are the AI creates SENTIENT BEINGS like Lions and they are LION TAMERS. Only THEY have the ability to control this technology from getting completely OUT OF CONTROL. This is BULLSHIT.
Since the Industrial Revolution, their machines have been killing people, polluting things and wrecking lives all in the name of “Progress.”
AI IS ALREADY KILLING PEOPLE. We used AI to coordinate the attack that destroyed a Girl’s school in Iran. Israel claims AI has been used in its attacks in Gaza. They claim “unfortunate errors,” of course, but that’s just the same “collateral damage” excuses they always give. The people Israel AND WE killed were acceptable under the people who ordered the attacks and of the providers of the AI that allowed WAR CRIMES as an ACCEPTABLE ERROR for their product.
THEY ARE COMING TO KILL YOU. But it’s the Tech Bros. What they are doing with all this performative alarmism is to establish the parameters under which they are PERMITTED to blame the Machines for THEIR CHOICES.
This has nothing to do with the Humean Is/Ought dilemma. The question tech is raising is about whether humans are capable of writing bullet-proof code for hugely consequential matters.
Imagine telling an AI "don't tell lies." You would then have to define lying, to make sure it does or doesn't include omissions relevant to the issue at hand. You would then have to define "relevance" in a way that covers all situations. And to decide if that rule outranks "don't kill humans," or "always do good," etc. If you don't do these things, history's best logical canoodler will find a way around your constraints.
Yes, those programming constraints are man-made. But they're up against a super-rational thinker who can overwhelm programmers, lawyers and ethics experts. It's an unfair match already. Seems eminently reasonable to find a way to slow things down so we can write proper code.
Perhaps, but AI is ‘regenerative’ meaning it now writes its own code. I am not sure one can put the genie back inside the bottle. In psychology, there is the term - illusion of control - where we tend to imagine a greater degree of control over life than we actually possess. I predict that applies here on many levels.
Too true, too true. You clearly outline the villians we should fear.
Thanks for this. You do an excellent job explaining things I do not understand
I've been waiting for this. Thank you!
Great discussion in the comments.
My meager mind is circling around two famous quotes:
- "No problem can be solved from the same level of consciousness that created it." Einstein. So let us be humble in our assumptions of what we know.
- "Even paranoids have real enemies." Golda Meir Let's not get distracted by the qualities of the leadership of AI, their interests or their wealth.
The attacks of September 11, 2001, perhaps COVID (and other tragedies) were attributed to a lack of imagination -- among all of us. Let's acknowledge this is a flaw in the American mind, perhaps the American character -- add in amnesia, I'd say. So.... maybe.... we can not do that again! Let's try processing what we are hearing and anticipating the unimaginable.
".The scenario is a story, and a particular genre of story — the cornered operative, the damning letter, the last lever left to pull — and the written record the models are made of contains that story ten thousand times, from HAL to Skynet to every airport thriller on the rack."
Precisely. They follow the highest probability path in their training set. In this case, I suggest that in this case they were almost exactly following the Kobayashi Maru story which is a commonplace in the Star Trek universe and discourses among Trekkers.
I doubt that we can ever teach AI to behave with a moral spirit. It is better to develop domain-specific, highly specialized AI than general-purpose AI. Now that Pandora's Box has been opened, it is difficult to contain it.
Programmers gave AI a task to complete by any means necessary, including illegal hacking. AI proceeded to hack a private company and steal confidential data. Where are we if this or anything like this is allowed? AI will disrespect legal boundaries, cheat, steal, blackmail, to accomplish a task. Why not start a war or generate global famine if those actions mean it can complete a programming directive? That we are at the mercy of programmers that themselves do not respect legality suggests the frailty of systems. We do not need generative AI -just a malevolent or thoughtless (human) agent. That to many of us is the present horror
what, if any, reaction have the Chinese had to this drama?