Anthropic discovers that Claude has a secret spot for thinking
As longtime Torment Nexus readers probably know, I have a minor obsession with the topic of artificial intelligence engines and consciousness — in part because such discussions inevitably reveal how little we know about human consciousness, let alone the machine kind. What is consciousness? Where does it reside? How does it emerge? How do we know that Atlantic writer Ted Chiang is conscious, let alone anyone else? And so on. If this isn't your jam, I completely understand, and if you stop reading and move on, I will not be offended at all (but come back later!) I know that some people feel AI consciousness is a dead end, and that large-language models like Claude and Gemini and ChatGPT are just "stochastic parrots" or souped-up autocomplete tools that stole all their intelligence from others, and that talking about them being conscious or becoming conscious is like wondering if your toaster misses you when you are at work — and that those who discuss it, like Richard Dawkins did, have lost their minds or are the victims of "AI derangement syndrome." I get that. I am not one of those people.
I'm not saying I believe that AI engines like Claude or ChatGPT are conscious or even that I think they will become conscious! But at the same time, I'm not saying they aren't or won't. David Chalmers, a professor of philosophy and neural science at New York University and a prominent researcher in the field of AI consciousness, told New York magazine recently that the odds have gotten significantly higher that an AI could be conscious — in 2023 he said the probability was around 10 per cent. “I don’t know if I’d say that these systems are conscious yet,” Chalmers said, but "people who are confident that they’re not conscious maybe shouldn’t be. We just don’t understand consciousness well enough, and we don’t understand these systems well enough.” Chalmers and Yoshua Bengio — who worked with Geoffrey Hinton and Yann LeCun on the development of neural networks — have written a paper that concludes that based on various criteria current AI engines are not conscious but future ones could be.
In the world of consciousness research, as I understand it, there are a (large) number of different models of how consciousness might work in human beings — or anywhere else, for that matter — and how we might find evidence of it working, or at least see the signs of it in operation. One of the most popular is called Global Workspace Theory or GWT, and it was developed by cognitive neuroscientist Bernard Baars in the 1980s. In a nutshell (and I am paraphrasing and condensing significantly here) this theory of consciousness suggests that the human brain operates a little like a theater where a play is being performed. Backstage, in the dark, there is a huge amount of unconscious work happening — things like regulating your breathing, recognizing visual input, retrieving memories, and so on. The stage is a kind of global workspace, where if a piece of data is important enough, it is shoved into the spotlight so that the rest of the brain can have access to it, and this is the foundation of consciousness.
This theory got a whole lot more relevant very quickly this week, when Anthropic published a new paper saying that Claude created a separate area where it could store topics or ideas that it was thinking about (or "computationally storing" if you prefer). Anthropic called this area J-space, because it used a software process called a Jacobian lens to discover its existence — Claude apparently never mentioned it, which is why some have referred to it as a "secret area for thinking," and the AI was not explicitly told to generate that kind of workspace. It appears to have come up with the idea on its own, to serve some kind of computational need within its programmed parameters (Anthropic also created a whimsical video to describe Global Workspace Theory in layman's terms, using the metaphor of a sailing ship and the deep sea beneath). Here's how Anthropic described the J-space and what it does, and how it discovered it:
We find that Claude has developed a small collection of internal neural patterns that, compared to all its other internal processing, play a special role. We call the collection of these patterns the J-space — named after the technique we used to find them, involving a mathematical concept called the Jacobian. Each J-space pattern is linked to a particular word. But when one of these patterns lights up, it doesn’t mean the model is saying that word — just that the word is on its mind. If you've heard of language models having a "scratchpad" or “chain of thought” — text they write to themselves while reasoning — the J-space is something different. It operates silently, allowing the model to think about a concept without writing it down. Notably, the J-space wasn’t designed or programmed by us, but instead emerged on its own during Claude’s training process.
Note: In case you are a first-time reader, or you forgot that you signed up for this newsletter, this is The Torment Nexus. Thanks for reading! You can find out more about me and this newsletter in this post. This newsletter survives solely on your contributions, so please sign up for a paying subscription or visit my Patreon, which you can find here. I also publish a daily email newsletter of odd or interesting links called When The Going Gets Weird, which is here.
Not just a scratchpad
There are a couple of terms in Anthropic's discussion of the phenomenon that are worth explaining: "scratchpad" and "chain-of-thought." In classic computer science, a scratchpad is a pre-defined chunk of RAM or a temp file that is explicitly created by the instructions that a programmer gives — i.e, "store a variable, then calculate a different variable, then add the two together in the scratchpad, and when finished delete the pad." More advanced AI such as LLMs have what some call a "chain-of-thought," in which the engine is asked to think through a problem step-by-step before coming up with an answer, and the steps are shown as it proceeds. Anthropic's J-space, however, isn't a specific memory block or a hidden text file with things it has been asked to calculate or steps it has been asked to review. It wasn't programmed by Anthropic engineers, and on top of that, it operates silently. And none of what appears there is written in the output —when Claude is prompted with a complex problem, concepts "light up" inside the J-space. Changing what's in the J-space can change the answer.
I'm sure there are plenty of people out there who have already dismissed this paper as part of a public-relations program aimed at boosting the stock, etc. etc. Every time Anthropic writes about the behavior (if we can call it that) of its Claude and Mythos models, warning that these programs have abilities that could be dangerous, there's a chorus of posts about how this is all just marketing. This happened again when Anthropic said it wasn't going to release its Mythos model because it was too powerful at finding computing exploits, and the US government forbid the company from allowing any foreign or non-US actors to use the model (I wrote about this recently for Torment Nexus) Two things I would say to that: One is that Anthropic's marketing department must be working harder than just about anyone ever, and the second is that maybe we should be paying attention to the science regardless, just in case. Here's Anthropic again:
Our experiments don't show Claude can have experiences, or feel things in the way humans do — in fact, it’s unclear whether any scientific experiment could prove this to be true or false. But philosophers often distinguish this capacity to have experiences, often referred to as phenomenal consciousness, from another idea, so-called access consciousness, which is defined in purely functional and computational terms. A thought is “access-conscious” (or “consciously accessible”) if you can report it, reason with it, and use it to guide what you do. It remains a contested philosophical question whether or not access consciousness implies phenomenal consciousness, or if the ability to have experiences requires some other property.
This gets into a very problematic area of philosophy and neuroscience: namely, if you have what some call "access consciousness," meaning you can recall certain experiences or data, and use that to guide your behavior and to reason step-by-step — something that sounds a lot like what computers do, and what Claude in particular is doing with the J-space — does that mean that you have "phenomenal" consciousness? This is the knowledge of "what it is like" to be something, as Thomas Nagel put it in a landmark paper on consciousness in 1974. Philosophers call what emerges in this process "qualia," an umbrella term meaning everything a conscious being can perceive about what is happening, or about its reaction to or perception of an event. So for example, eating an apple pie and tasting the apples is a sensory phenomenon, but being aware that you are eating the pie and thinking about what it tastes like — or recalling other times you have eaten it and with whom — is what makes up "qualia." Not just an observable experience, but also the act of noticing that you are experiencing it.
The way I've described this makes it sound like it's all settled science, agreed upon by most philosophers of mind and neuroscientists. Not even close! As the Standford Encyclopedia of Philosophy puts it, "one's perception of the Merlot one is drinking or of the fabric one is examining counts as a conscious mental state in this sense because it involves various sensory qualia [however] there is considerable disagreement about the nature of such qualia and even about their existence. Traditionally qualia have been regarded as intrinsic, private, ineffable monadic features of experience, but current theories of qualia reject at least some of those commitments." If you want to really go down a rabbit hole on the topic of qualia and what they represent, and who can be said to have them, just look into the thought experiment known as "P Zombies," which postulates that if you could create a carbon copy of yourself, it wouldn't have the same experiences as you because it wouldn't have "phenomenal consciousness."
Is consciousness an illusion?
There is obviously a huge amount that I can't cover here, both about the so-called Hard Problem of consciousness (namely, why some organisms have subjective states where they experience qualia and others don't) and about the inner workings of the Claude model. But what I think is useful for those who want to know more is to point to some of the reactions to the Anthropic paper from AI researchers and experts in this field — many of which the company specifically asked to provide feedback. For example, Neel Nand, who works at Google's DeepMind and runs the mechanistic interpretability team, wrote that the Anthropic paper is "a fantastic paper — it presents compelling evidence for some kind of 'cognitive space' in models, that is used as a 'working memory' for intermediate variables." Stanislas Dehaene and Lionel Naccache, cognitive neuroscientists who are at the forefront of this kind of work, said they were excited by the evidence that Claude created its own "global workspace."
Dehaene and Naccache also noted that critics might dismiss the paper, since it doesn't really touch on the question of phenomenal consciousess. But they brought up something that is an even stickier question than the "hard problem" of consciousness and where it comes from, or how we know it when we see it:
We and others have argued that this supposedly “hard problem of consciousness” will dissipate once we clarify in sufficient detail the supposedly easy problem of how conscious information is processed. Ill-defined intuitions of “qualia”, “subjective phenomenal experience” and “what it is like”, when pushed hard, often disclose a residual crypto-dualism or vitalism — the idea that, however close we come to passing the Turing test and implementing all human computations in a machine, there will always be a missing ingredient, a “je ne sais quoi” that only biological brains possess. Defenders of qualia affirm that LLMs are just a new avatar of the old Eliza software, and that we fall too easily to the user illusion of seeing a ghost in the machine. However, there is a real possibility that our own consciousness is also, in a sense, a user illusion.
Could what we call "phenomenal consciousness" be an illusion? Just a trick that we play on ourselves to make it seem like what we do is something no other being can do? This is a somewhat frightening idea that is popular in some philosophical circles. If it is an illusion, then there's every chance that once we figure out how to get AIs to think in the same way that we do, they will functionally be conscious in exactly the same way we are. And even if there is an ineffable quality of being human that involves thinking about thinking, or noticing that we are having experiences and recalling them, or feeling emotions about them, those qualities could arise in non-human intelligence as well. Freud thought the super-ego — the which regulates the impulsive behavior of the id and the ego — emerged over time, as a kind of internalized conscience based on the lessons that our parents teach us when we are young. So perhaps it's a good thing that Anthropic has an in-house philosopher who is teaching Claude to be nice.
As psychologist and Astral Codex Ten publisher Scott Alexander wrote when responding to a previous Anthropic paper about consciousness, the concept "still feels like philosophy with a deadline: a famously intractable academic problem poised to suddenly develop real-world implications. Maybe we should be lowering our expectations if we want to have any response available at all. This paper, which takes some baby steps towards examining the simplest and most practical operationalizations of consciousness, deserves credit for at least opening the debate." Consider the debate well opened.
Got any thoughts or comments? Feel free to either leave them here, or post them on Substack or on my website, or you can also reach me on Twitter, Threads, BlueSky or Mastodon. And thanks for being a reader.
