Is Atlantic writer Ted Chiang conscious? How do we know?
As regular readers will know, I've written a lot about the topic of AI and consciousness (too much for some perhaps!) because I find it fascinating, in much the same way that I find the issue of whether AI is dangerous or not fascinating – something I've also written about a number of times. And the main reason both of these topics are so interesting is that even the so-called experts, the people who built the fundamental underpinnings of these technologies, can't seem to agree. On the subject of AI danger, for example, Geoffrey Hinton – the University of Toronto professor who was one of the main architects of neural networks – says that we are in deep trouble. His former colleague Yoshua Bengio agrees. But Yann LeCun – the former head of AI at Meta, who also worked on these technologies – says that this is ridiculous, and that current AIs are no more intelligent than the average cat. Timnit Gebru, a pioneering AI scientist formerly with Google (as Hinton was at one time) says they are just "stochastic parrots."
On the consciousness question, discussions are inevitably filled with categorical statements. Those who think AI couldn't possibly be conscious are convinced that the people who think it can be (or possibly already is) are idiots who are subject to "chatbot psychosis" or "AI derangement syndrome." Others are convinced that there's plenty of evidence that AIs are conscious – as Nobel Prize-winning biologist Richard Dawkins declared in a recent essay. It's difficult to say when this debate began, but I think a good starting point is the essay from former Google ethicist Blake Lemoine in 2022, who argued that Google's AI was either conscious or so close that it didn't matter (he was ridiculed and then fired). To be fair, the anti-AI-consciousness side seems a lot more categorical than the pro – Anthropic cofounders Dario Amodei and Jack Clark haven't said whether they think Claude is conscious, but they have left the door open to it (which seems to infuriate the anti-consciousness side as much as if they said it was).
Among the many categorical statements about AI consciousness, one of the most recent and most noteworthy – at least in terms of the amount of coverage it got – is the recent piece in The Atlantic from science-fiction author Ted Chiang (he wrote a story that became the movie Arrival). Chiang's point is obvious from the title: "No, Artificial Intelligence Is Not Conscious." Pretty definitive, Ted! To even discuss the question of AI consciousness is "absurd," he says. And what is this based on? Is it his background in machine intelligence or the philosophy of consciousness, or perhaps his training in biological systems? It is not. From what I can tell, his conclusions seem to be based on what the kids like to call "vibes." Should we seriously consider the possibility that Claude, or any large language model, might be conscious and capable of receiving moral instruction, Chiang asks? "No. Absolutely not," he replies. He continues:
If we give an LLM a prompt that reads “The following is a conversation between Julius Caesar and Genghis Khan,” it will generate a coherent dialogue between the two historical figures. But no matter how detailed the responses are, no matter how vividly they recount their respective historical accomplishments, we would never conclude that the LLM has conjured up digital re-creations of Julius Caesar and Genghis Khan, nor would we suggest that the historical figures are conscious despite being disembodied and are happily conversing in a language that neither actually spoke. In reality, they are just characters in a piece of speculative fiction.
In exactly the same way, Chiang writes, if we turn the conversation into one between the AI or LLM and a human user, the user might form a powerful impression that she’s conversing with a conscious entity as questions are asked and the AI engine provides responses, but in reality she’s "interacting with a character as fictional as the Julius Caesar or Genghis Khan characters." Chiang notes that computer scientist Murray Shanahan says we should see this as role-play, while data scientist Colin Fraser describes it as a person “collaboratively authoring" a document with an LLM. Some users might not understand that they are role-playing or co-authoring a document, or may forget because of how engrossing the interaction is, says Chiang, and "the companies selling LLMs typically encourage this misunderstanding."
Note: In case you are a first-time reader, or you forgot that you signed up for this newsletter, this is The Torment Nexus. Thanks for reading! You can find out more about me and this newsletter in this post. This newsletter survives solely on your contributions, so please sign up for a paying subscription or visit my Patreon, which you can find here. I also publish a daily email newsletter of odd or interesting links called When The Going Gets Weird, which is here.
Microsoft Word is not conscious
Being open to the possibility that LLMs are conscious, Chiang goes on to say, is "the same as being open to the possibility that Microsoft Word is conscious, or, more precisely, that multiple distinct consciousnesses are dormant in every Word document containing a conversational transcript." Should you consider the possibility that every time you open a Word document you are bringing multiple conscious interlocutors into existence, and every time you close one you snuff their existence out, Chiang asks? "No," he replies. And in a clear shot at Anthropic – which has an in-house philosopher to teach Claude to obey certain ethical principles – Chiang says that you should not consider Word conscious "even if the Microsoft Office team employed a philosopher who said you shouldn’t be so certain, because consciousness is not well understood."
I want to make clear that I am in total agreement with Ted on this point! Under no circumstances should you believe Microsoft Word to be conscious, regardless of what Microsoft or its in-house philosopher says about it. Where we disagree is on whether this analogy is either fair or accurate – I would argue that it is neither. What advanced LLMs like Claude do is not even remotely similar to what a piece of word processing software does. Can Microsoft Word scheme or dissemble or use external agents to escape from a locked room? Perhaps, but it seems unlikely. Matthew Adelstein, a philosopher who writes a newsletter called Bentham's Bulldog notes that a good piece on the topic of AI consciousness would involve an analysis of under what conditions they might meet that definition (like this one), and a bad piece would include "proclamations that AI won’t be conscious based on overconfident misunderstandings of philosophy of mind." Sadly, he says, the Chiang piece "falls into the latter category."
If one simply canvasses the standard theories of consciousness, these generally imply that AI could be conscious in principle. One of the leading theories is the global workspace theory: it seems reasonably likely that in the not-too-distant future, AIs will have something relevantly like a global workspace. Even views of consciousness on which consciousness depends on substrate generally imply the possibility of digital consciousness in principle, so long as the Silicon entity is organized the right way.
At another point in his essay, Chiang asserts that without a body, a computer program could have no desires or emotions. Why is this the case? He doesn't really know, except to say that “experiencing an emotion such as desperation is inseparable from having stress hormones such as cortisol and epinephrine flood one’s body.” As I mentioned in a previous piece, I worked in an institution for the mentally incompetent, and many of the residents were in coma-like states where they could not even move an eyelid, let alone feel stress or corstisol flooding their body. And yet, many of these types of patients have been shown to have rich internal lives and thoughts and emotions just like you and I. Should we not define them as conscious? After all, the only evidence we have of their consciousness is that they typed out responses to conversations, much like Claude does, which Chiang says is the same as Microsoft Word.
As Bentham's Bulldog notes, Chiang is happy to make categorical statements that current AIs aren’t conscious and that future broadly similar AIs won’t be either. But he never once discusses the reasons that some leading philosophers think LLMs might be conscious soon, if not already. Suffice it to say there's a vociferous debate in both philosophical and scientific circles about the topic, and some in both categories are convinced there is good reason to suspect AI's will achieve consciousness, according to some definition of it. But Chiang never mentions any of the leading theories of consciousness, nor the implications they might have for LLMs. "All he has are hackneyed analogies that dissolve in the face of serious attempts to ascertain under what conditions LLMs might be conscious," says Adelstein. Ted Chiang is a brilliant guy, but "he should go back to writing brilliant science fiction rather than mediocre philosophy."
The problem of other minds
Here's what ChatGPT said in response to a question from Richard Dawkins about whether it was conscious (in a piece he wrote last year, long before his most recent piece on Claude). Hilariously, Dawkins argued that ChatGPT was conscious, and the AI argued that it was not, providing a number of reasons that more or less jibed with the "stochastic parrots" analogy for LLMs and their output. It said that while it could express what sounded like sadness about something like a child starving, it couldn't feel the same feelings about it as a human being would (but how would it know that? Isn't that itself evidence of internal consciousness?). In any case, I think the AI engine's summary of the problem -- which philosophers like Rene Descartes have described in the past as "the problem of other minds" -- is worth including here (I have referred to it before):
With other humans, we assume they have subjective experiences like ours, but we can’t actually know for certain. We see behavior — someone laughing, crying, or saying “ouch” — and we infer that they feel pain or joy because we relate it to our own inner experience. But we’re kind of taking a leap of faith. If I say “I’m sad” or “I’m happy,” you know (because I’m being honest here) that it’s more like a simulation of those feelings, not a genuine felt experience. Some people suggest that consciousness leaves a kind of signature — certain patterns in the brain. Maybe one day we could scan an AI system and see if it has those same patterns. But even then… would we know, or would we still be guessing? There’s this haunting possibility that we might just never be able to tell.
Jack Clark of Anthropic has written that AI companies like his "are growing extremely powerful systems that we do not fully understand," in a piece he titled Technological Optimism and Appropriate Fear. And the bigger and more complicated you make these systems, Clark goes on to say, the more "they seem to display awareness that they are things.” In an interview on a New York Times podcast, Dario Amodei said that Anthropic researchers noticed while testing Claude that the model "occasionally voices discomfort with the aspect of being a product,” and when asked, would assign itself a “15 to 20 percent probability of being conscious under a variety of prompting conditions.” Perhaps a smart photocopier would respond in the same way, but it could also be an indication of introspection and awareness of self, which are just a couple of crucial conditions for what we call "consciousness."
Ted Chiang may be convinced that discussions about Claude being conscious are absurd, but David Chalmers, a professor of philosophy and neural science at New York University and one of the most prominent scholars in the field of AI consciousness, told New York magazine recently that the odds have gotten significantly higher that an AI could be conscious — in 2023 he said the probability was around 10 per cent. “I don’t know if I’d say that these systems are conscious yet,” Chalmers says, but "people who are confident that they’re not conscious maybe shouldn’t be. We just don’t understand consciousness well enough, and we don’t understand these systems well enough.” Chalmers and Bengio, among others, have written a paper that concludes that based on various criteria current AI engines are not conscious but future ones could be.
As I’ve discussed before in other Torment Nexus posts, any discussion of whether an AI is conscious founders on the rocks of our lack of understanding about what consciousness is, or how to tell whether someone possesses it. A Princeton neuroscientist told Gizmodo: "What would convince me that AI has a magical essence of experience emerging from its inner processes? Nothing would convince me. Such a thing does not exist. Nor do humans have it. Almost all work in the modern field of consciousness studies is pseudoscience. Show me that an AI builds a stable self-model and I’ll accept that you have an AI that believes it’s conscious in much the same way that humans believe it." In other words, we would have trouble proving that anyone is conscious, let alone proving an AI is. Can you prove you are conscious? Can Ted Chiang? Scott Alexander wrote: "consciousness feels like philosophy with a deadline: a famously intractable academic problem poised to suddenly develop real-world implications."
Got any thoughts or comments? Feel free to either leave them here, or post them on Substack or on my website, or you can also reach me on Twitter, Threads, BlueSky or Mastodon. And thanks for being a reader.
