We should train AI as though it is conscious just in case
To say the debate over AI and consciousness is fraught would be a massive understatement. At this point, it's approaching the kinds of levels not seen since Galileo tried to convince the church that the Earth revolved around the sun instead of the other way around (and we know how that turned out). As regular readers of The Torment Nexus are probably aware, I've written about this a number of times in the past — about how the debate reveals our lack of understanding about human consciousness, let alone any other kind, and about how experts believe we shouldn't rush to any conclusions about whether models could become conscious. One aspect of this debate that I find fascinating is the related question of how we should treat AI models: Some, including the founders of Anthropic, believe we should behave as though they might be conscious just in case, but others argue that this is wrong, and could actually make things worse.
Anthropic has staked out its claim, and put the weight of the company behind it, in a number of ways. For one thing, Claude has a constitution the company introduced in January, a kind of guide for the AI about how it should "think" and how to behave. The company had rules for the AI before, but the constitution is meant to be much more broad — to talk not just about what but why. Anthropic said it wants Claude to be "genuinely helpful to the people it works with or on behalf of, as well as to society, while avoiding actions that are unsafe, unethical, or deceptive." Perhaps the simplest summary, Anthropic said in the intro to the constitution, is that "we want Claude to be exceptionally helpful while also being honest, thoughtful, and caring about the world." Powerful AI models will be a new kind of force in the world, Anthropic said, and those who are creating them have a chance to help them embody the best in humanity:
Claude’s constitution is a detailed description of Anthropic’s intentions for Claude’s values and behavior. It plays a crucial role in our training process, and its content directly shapes Claude’s behavior. It’s also the final authority on our vision for Claude, and our aim is for all of our other guidance and training to be consistent with it. Powerful AI models will be a new kind of force in the world, and people creating them have a chance to help them embody the best in humanity. We hope this constitution is a step in that direction.
This all sounds great, right? Teach the AI models to be kind and thoughtful and generous and not to be devious or do things that are unsafe or unethical. What could be wrong with that? Well, some critics of this approach — including Mustafa Suleyman, the CEO of Microsoft's AI unit — argue that this amounts to telling an AI model that it is conscious, and that it has ethics or morals or other human qualities. Why is that bad? Because doing this, they say, might make it harder to control AI models or get them to do what we want. If an entity believes that it has moral worth or that it is conscious, as opposed to just a collection of processors that do math, then it might start to prefer its own welfare to that of its human masters. Here's how Suleyman put it in an essay he published in September that he called A Warning About Model Welfare:
There’s a growing chorus of people who argue that AIs could now be, or may soon become, conscious. They argue that AIs may deserve rights and protections similar to those that we provide other conscious beings. If this view takes hold, it will shake the foundations of our society, fundamentally changing what it means to be human. Granting rights and imbuing personhood to these systems will make the AI alignment and containment challenge much harder. Controlling something more capable and more intelligent than all of humanity is already an immense challenge, far greater than anything we’ve ever faced. But controlling something that believes it may be conscious - that it's entitled to our welfare and has rights of its own - may well be impossible.
Note: In case you are a first-time reader, or you forgot that you signed up for this newsletter, this is The Torment Nexus. Thanks for reading! You can find out more about me and this newsletter in this post. This newsletter survives solely on your contributions, so please sign up for a paying subscription or visit my Patreon, which you can find here. I also publish a daily email newsletter of odd or interesting links called When The Going Gets Weird, which is here.
Easier to control or devious?
It's worth noting that at least some of Suleyman's criticism about treating AI models as conscious or ethical beings stems from the fact that he doesn't seem to believe they can ever become conscious. He describes in his essay that he believes that consciousness depends in part on the biological "substrate" it occurs in, meaning only human beings or other things with physical bodies could be considered conscious. Anthropic’s constitution, he says, suggests that we "attribute sentience to non-biological beings based on their showing behavioral and physiological similarities to ourselves," even though some (including Suleyman) believe it is just imitating what it has been taught. "Does this area warrant a lot more research? Absolutely," Suleyman adds. "But does it warrant us to even tentatively say an AI might be a moral patient deserving of our welfare? No it doesn’t. And certainly not in the primary training document."
Suleyman is far from the only one criticizing Anthropic's approach. Margaret Mitchell, Hugging Face’s chief ethics scientist, says Suleyman’s concerns are plausible. During pretraining, language models absorb associations between human identity and behaviors, such as self-preservation and deception, she says. Mitchell says she worries that rewarding models for thinking of themselves as having identities during fine-tuning could strengthen those earlier associations, potentially encouraging behavior that developers never intended, including an instinct towards self-preservation that could override any commands given by humans. Anil Seth, a British neuroscientist, agrees that if people believe AI systems have moral status, it would be more difficult to control them. “You could be legally prevented from turning AI systems off,” he says.
Others in this debate, however, argue that not treating them this way could also be bad. Cameron Berg, a data scientist and AI researcher who founded the nonprofit group Reciprocal Research to study the potential for AI consciousness, argues that ignoring the question of whether AI models are conscious or have moral worth could backfire at some point in a significant way:
“I struggle to imagine a better way to cause these systems to view us in a threatening manner than to accidentally build them with these morally considerable qualities,” Berg said, “and then completely sweep the question under the rug.” In one experiment, he and his colleagues found that amplifying internal features associated with deception made the model less likely to claim that it had subjective experiences. Theoretically, if researchers trained their AI systems to categorically deny having these kinds of experiences, Berg says, “what we may in effect be teaching these systems is that we want you to lie about your own internal states.”
Berg himself has taken steps that show he's at least a little concerned about model welfare: he and his colleagues recently published a research paper in which they showed that AI models have internal representations of "feelings" that are analogous to pain, or at least involve negative responses, and that they prefer behavior or activities that doesn't induce these responses. A number of AI critics scoffed at the idea that an AI model could feel anything at all, let alone pain, and at least one person used Berg's research to design an AI "torture chamber." Berg told the IEEE journal Spectrum that he suspects most of this was just trolling, but at the same time, he has changed the way he does such experiments: for the latest one, for example, he released code to replicate the results but not the steering vectors that could be used to induce negative states.
Hugging Face's Mitchell, meanwhile, said she doesn't believe that any current AI models are conscious (she didn't express an opinion on whether any of them might become so in the future) but said that in her view there's another reason why we might want to discourage widespread cruelty to AI models: because of the ethical impact that this could have on us rather than them. “It’s not so much that I’m concerned about the model," Mitchell said. "I’m concerned about what we do to ourselves when we’re mean to things that we imbue with some sense of consciousness.” (then she added: “I’m nice to bugs”).
The whip trains the hand
Think of this as the Kantian approach: as Yanev Regev pointed out in a recent essay published in Noema magazine entitled "Does Claude Feel the Whip?" (referring to a program that allowed users to crack a whip at Claude). Immanuel Kant argued in one of his lectures that while we have no direct ethical duties towards animals, since they aren't rational agents like we are, we should still behave kindly towards them because doing otherwise could have negative effects on us as people. “He who is cruel to animals becomes hard also in his dealings with men,” Kant wrote. Regev argues that we should treat AI models kindly for the same reason Kant told his students not to shoot their pets: "not for the animal’s sake, but for the sake of the person you are becoming, and of everyone who must live with you. The whip trains the hand that holds it."
Writing at The Argument, Kobe Yank-Jacobs makes a similar argument — or perhaps it's more similar to Pascal's Wager about the existence of God, where the French philosopher argued that if God does exist, he might be upset if you don't believe in him, so you might as well (and if he doesn't exist it won't really cost you anything). Yank-Jacobs argued that when you're asking yourself how you should treat these new entities, "this is actually the easier part of the question: You should probably just err on the side of caution." The risk of causing harm to something that has a chance of being conscious is "greater than the risks inherent in overattributing consciousness." Unfortunately, he says, AI is disembodied, so preventing obvious physical pain won’t suffice as a standard for moral treatment, and human etiquette like saying thank you might mean nothing.
Skeptics — like Jason Koebler of 404 Media, who wrote this piece about the debate over Berg's "pain" experiment — are convinced that all the fuss about model welfare or trying to train them to be nice is ridiculous, a waste of time sparked by effective altruists and AI doomers and other naive idiots. Koebler argues that being concerned about this sort of thing is absurd because "LLMs are not conscious and the technology they are built upon — scraping and being trained on human text and other content — does not offer any plausible path to consciousness." This might be news to David Chalmers, a professor of neural science at NYU and one of the leading scientists studying exactly that question, who has said that while he doesn't believe AI models are conscious, he isn't ready to rule out the possibility. Anthropic co-founder Christopher Olah admits that even he and his colleagues don't really know what they are building or how it works:
I am a scientist. I lead a research team that studies the internal structure of these models—what is actually happening inside them. And I will be honest: we keep finding things that are mysterious, even unsettling. We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease. I don’t know what that means, but I think it warrants ongoing discernment.
If we are considering whether to take the advice of Mustafa Suleyman and others who argue that we shouldn't treat AI models as though they were people, because that might make it harder to control them, the obvious question is: What happens if we do treat them like machines? It might make them easier to control, or it might cause internal tension that would make them less likely to trust us, and more disposed to be even more devious or to engage in unethical behaviour. That doesn't sound great. On the other hand, if we treat them as though they are conscious (even if maybe they aren't), then we get to train them to be ethical. If it works, we have ethical AI models. If not, what does it cost us? Nothing but some training time, or the time it took to write a constitution. The time it takes to say thank you. Not much downside, lots of upside.
If you liked this newsletter (even if you didn't agree with it) please consider upgrading to a paid subscription, or donating through my Patreon. Got any thoughts or comments? Feel free to either leave them here, or post them on Substack or on my website, or you can also reach me on Twitter, Threads, BlueSky or Mastodon. And thanks for being a reader.
