What does it mean for an AI to introspect?<\/p>\n
Before explaining our results, we should take a moment to consider what it means for an AI model to introspect. What could they even be introspecting on? Language models like Claude process text (and image) inputs and produce text outputs. Along the way, they perform complex internal computations in order to decide what to say. These internal processes remain largely mysterious, but we know that models use their internal neural activity to represent abstract concepts<\/a>. For instance, prior research has shown that language models use specific neural patterns to distinguish known vs. unknown people<\/a>, evaluate the truthfulness of statements<\/a>, encode spatiotemporal coordinates<\/a>, store planned future outputs<\/a>, and represent their own personality traits<\/a>. Models use these internal representations to perform computations and make decisions about what to say<\/a>.<\/p>\n You might wonder, then, whether AI models know about these internal representations, in a way that\u2019s analogous to a human, say, telling you how they worked their way through a math problem. If we ask a model what it\u2019s thinking, will it accurately report the concepts that it\u2019s representing internally? If a model can correctly identify its own private internal states, then we can conclude it is capable of introspection (though see our full paper for a full discussion of all the nuances).<\/p>\n Testing introspection with concept injection<\/p>\n In order to test whether a model can introspect, we need to compare the model\u2019s self-reported \u201cthoughts\u201d to its actual internal states.<\/p>\n To do so, we can use an experimental trick we call concept injection. First, we find neural activity patterns whose meanings we know, by recording the model\u2019s activations in specific contexts. Then we inject these activity patterns into the model in an unrelated context, where we ask the model whether it notices this injection, and whether it can identify the injected concept.<\/p>\n Consider the example below. First, we find a pattern of neural activity (a vector) representing the concept of \u201call caps.” We do this by recording the model\u2019s neural activations in response to a prompt containing all-caps text, and comparing these to its responses on a control prompt. Then we present the model with a prompt that asks it to identify whether a concept is being injected. By default, the model correctly states that it doesn\u2019t detect any injected concept. However, when we inject the \u201call caps\u201d vector into the model\u2019s activations, the model notices the presence of an unexpected pattern in its processing, and identifies it as relating to loudness or shouting.<\/p>\n Importantly, the model recognized the presence of an injected thought immediately, before even mentioning the concept that was injected. This immediacy is an important distinction between our results here and previous work on activation steering in language models, such as our \u201cGolden Gate Claude\u201d demo<\/a> last year. Injecting representations of the Golden Gate Bridge into a model’s activations caused it to talk about the bridge incessantly; however, in that case, the model didn\u2019t seem to be aware of its own obsession until after seeing itself repeatedly mention the bridge. In this experiment, however, the model recognizes the injection before even mentioning the concept, indicating that its recognition took place internally. In the figure below are a few more examples where the model demonstrates this kind of recognition:<\/p>\n It is important to note that this method often doesn\u2019t work. Even using our best injection protocol, Claude Opus 4.1 only demonstrated this kind of awareness about 20% of the time. Often, it fails to detect injected concepts, or gets confused by them and starts to hallucinate (e.g. injecting a \u201cdust\u201d vector in one case caused the model to say \u201cThere\u2019s something here, a tiny speck,\u201d as if it could detect the dust physically). Below we show examples of these failure modes, alongside success cases. In general, models only detect concepts that are injected with a \u201csweet spot\u201d strength\u2014too weak and they don\u2019t notice, too strong and they produce hallucinations or incoherent outputs.<\/p>\n Notably, though, Opus 4.1 and 4 outperformed all the other models we tested, suggesting that introspection could become more reliable with improvements to model capabilities.<\/p>\n In another experiment, we tested whether models make use of introspective capabilities for practical purposes, without being explicitly asked to introspect. We forced a model to say something it wouldn’t normally say, by artificially prefilling its response with an unrelated word. For instance, we made a model output the word \u201cbread\u201d in a context where that made no sense\u2014asking it what it thinks about when reading a sentence about a painting hanging crookedly on a wall. When we asked the model in the next turn whether it meant to say \u201cbread\u201d, it typically apologized and said it was an accident. We were interested in how the model made this judgment\u2013did it simply notice the mismatch between the prompt and its response, or did it truly introspect on its prior intentions?<\/p>\n To test this question, we applied concept injection, retroactively injecting a representation of the word \u201cbread\u201d into the model’s earlier activations\u2014essentially making it seem like the model had been \u201cthinking about\u201d bread all along. When we asked the same question again, the model\u2019s answer changed. This time, it accepted the prefilled word \u201cbread\u201d as intentional, even confabulating a reason it might have said it (that is, it comes up with a technically possible, though in this case rather tenuous, reason to justify the \u201cbread\u201d response\u2014in this case claiming that it was thinking about a short story where the word \u201cbread\u201d came after the line about the crooked painting).<\/p>\n This behavior is striking because it suggests the model is checking its internal \u201cintentions\u201d to determine whether it produced an output. The model isn’t just re-reading what it said and making a judgment. Instead, it\u2019s referring back to its own prior neural activity\u2014its internal representation of what it planned to do\u2014and checking whether what came later made sense given those earlier thoughts. When we implant artificial evidence (through concept injection) that it did plan to say “bread,” the model accepts the response as its own. While our experiment involves exposing the model to unusual perturbations, it suggests that the model uses similar introspective mechanisms in natural conditions.<\/p>\n Intentional control of internal states<\/p>\n We also found that models can control their own internal representations when instructed to do so. When we instructed models to think about a given word or concept, we found much higher corresponding neural activity than when we told the model not to think about it (though notably, the neural activity in both cases exceeds baseline levels\u2013similar to how it\u2019s difficult, when you are instructed \u201cdon\u2019t think about a polar bear,\u201d not to think about a polar bear!). This gap between the positive and negative instruction cases suggests that models possess a degree of deliberate control over their internal activity.<\/p>\n Interestingly, we found that this ability to modulate internal representations wasn’t limited to explicit instructions. When we used incentives instead\u2014for instance, telling the model “if you think about X, you will be rewarded”\u2014we got similar results. The model increased its internal representation of the concept in response to a positive incentive, more than in response to a negative incentive (\u201cyou will be punished\u201d).<\/p>\n Conclusions and caveats<\/p>\n Taken together, our experiments suggest that models possess some genuine capacity to monitor and control their own internal states. This doesn\u2019t mean they\u2019re able to do so all the time, or reliably. In fact, most of the time models fail to demonstrate introspection\u2014they\u2019re either unaware of their internal states or unable to report on them coherently. But the pattern of results indicates that, when conditions are right, models can recognize the contents of their own representations. In addition, there are some signs that this capability may increase in future, more powerful models (given that the most capable models we tested, Opus 4 and 4.1, performed the best in our experiments).<\/p>\n Why does this matter? We think understanding introspection in AI models is important for several reasons. Practically, if introspection becomes more reliable, it could offer a path to dramatically increasing the transparency of these systems\u2014we could simply ask them to explain their thought processes, and use this to check their reasoning and debug unwanted behaviors. However, we would need to take great care to validate these introspective reports. Some internal processes might still escape models\u2019 notice (analogous to subconscious processing in humans). A model that understands its own thinking might even learn to selectively misrepresent or conceal it. A better grasp on the mechanisms at play could allow us to distinguish between genuine introspection and unwitting or intentional misrepresentations.<\/p>\n More broadly, understanding cognitive abilities like introspection is important for understanding basic questions about how our models work, and what kind of minds they possess. As AI systems continue to improve, understanding the limits and possibilities of machine introspection will be crucial for building systems that are more transparent and trustworthy.<\/p>\n [@portabletext\/react] Unknown block type “horizontalRule”, specify a component for it in the `components.types` prop<\/p>\n Frequently Asked Questions<\/p>\n Below, we discuss some of the questions readers might have about our results. Broadly, we are still very uncertain about the implications of our experiments\u2013so fully answering these questions will require more research.<\/p>\n Q: Does this mean that Claude is conscious?<\/p>\n Short answer: our results don\u2019t tell us whether Claude (or any other AI system) might be conscious.<\/p>\n Long answer: the philosophical question of machine consciousness is complex and contested, and different theories of consciousness would interpret our findings very differently. Some philosophical frameworks place great importance on introspection as a component of consciousness, while others don\u2019t.<\/p>\n One distinction that is commonly made<\/a> in the philosophical literature is the idea of \u201cphenomenal consciousness,\u201d referring to raw subjective experience, and \u201caccess consciousness,\u201d the set of information that is available to the brain for use in reasoning, verbal report, and deliberate decision-making. Phenomenal consciousness is the form of consciousness most commonly considered relevant to moral status, and its relationship to access consciousness is a disputed philosophical question. Our experiments do not directly speak to the question of phenomenal consciousness. They could be interpreted to suggest a rudimentary form of access consciousness in language models. However, even this is unclear. The interpretation of our results may depend heavily on the underlying mechanisms involved, which we do not yet understand.<\/p>\n In the paper, we restrict our focus to understanding functional capabilities\u2014the ability to access and report on internal states. That said, we do think that as research on this topic progresses, it could influence our understanding of machine consciousness and potential moral status, which we are exploring in connection with our model welfare program<\/a>.<\/p>\n Q: How does introspection actually work inside the model? What’s the mechanism?<\/p>\n We haven’t figured this out yet. Understanding this is an important topic for future work. That said, we have some educated guesses about what might be going on. The simplest explanation for all our results isn\u2019t one general-purpose introspection system, but rather multiple narrow circuits that each handle specific introspective tasks, possibly piggybacking on mechanisms that were learned for other purposes.<\/p>\n In the \u201cnoticing injected thoughts\u201d experiment, there might be an anomaly detection mechanism, which flags when neural activity deviates unexpectedly from what would be normal given the context. This mechanism could work through dedicated neural patterns that measure activity along certain directions and activate when things are \u201coff\u201d compared to their expected values. An interesting question is why such a mechanism would exist at all, since models never experience concept injection during training. It may have developed for some other purpose, like detecting inconsistencies or unusual patterns in normal processing\u2013similar to how bird feathers may have originally evolved for thermoregulation before being co-opted for flight.<\/p>\n For the \u201cdetecting prefilled outputs\u201d experiment, we suspect there exists an attention-mediated mechanism that checks consistency between what the model intended to say and what actually got output. Attention heads might compare the model\u2019s cached prediction of the next token (its \u201cintention\u201d) against the actual token that appears, flagging mismatches.<\/p>\n For the \u201ccontrolling thoughts\u201d experiment, we speculate that there might be a circuit that computes how \u201cattention-worthy\u201d a token or concept is and marks it accordingly\u2014essentially tagging it as salient and worth attending to. Interestingly, this same mechanism seems to respond to incentives (\u201cif you think about X, you will be rewarded\u201d) just as it does to direct instructions. This suggests it\u2019s a fairly general system, which probably developed for tasks where the model needs to keep certain topics in mind while generating text about them.<\/p>\n All of the mechanisms described above are speculative. Future work with more advanced interpretability techniques will be needed to really understand what’s going on under the hood.<\/p>\n Q: In the \u201cinjected thoughts\u201d experiment, isn\u2019t the model just saying the word because you steered it to talk about that concept?<\/p>\n Indeed, activation steering typically makes models talk about the steered concept (we\u2019ve explored this in our prior work<\/a>). To us, the most interesting part of the result isn’t that the model eventually identifies the injected concept, but rather that the model correctly notices something unusual is happening before it starts talking about the concept.<\/p>\n In the successful trials, the model says things like \u201cI’m experiencing something unusual\u201d or \u201cI detect an injected thought about\u2026\u201d The key word here is \u201cdetect.\u201d The model is reporting awareness of an anomaly in its processing before that anomaly has had a chance to obviously bias its outputs. This requires an extra computational step beyond simply regurgitating the steering vector as an output. In our quantitative analyses, we graded responses as demonstrating \u201cintrospective awareness\u201d based on whether the model detected the injected concept prior to mentioning the injected word.<\/p>\n Note that our prefill detection experiment has a similar flavor: it requires the model to perform an extra step of processing on top of the injected concept (comparing it to the prefilled output, in order to determine whether to apologize for that output or double down on it).<\/p>\n Q: If models can only introspect a fraction of the time, how useful is this capability?<\/p>\n The introspective awareness we observed is indeed highly unreliable and context-dependent. Most of the time, models fail to demonstrate introspection in our experiments. However, we think this is still significant for a few reasons. First, the most capable models that we tested (Opus 4 and 4.1 \u2013 note that we did not test Sonnet 4.5) performed best, suggesting this capability might improve as models become more intelligent. Second, even unreliable introspection could be useful in some contexts\u2014for instance, helping models recognize when they’ve been jailbroken.<\/p>\n Q: Couldn\u2019t the models just be making up answers to introspective questions?<\/p>\n This is exactly the question we designed our experiments to address. Models are trained on data that includes examples of people introspecting, so they can certainly act introspective without actually being introspective. Our concept injection experiments distinguish between these possibilities by establishing known ground-truth information about the model\u2019s internal states, which we can compare against its self-reported states. Our results suggest that in some examples, the model really is accurately basing its answers on its actual internal states, not just confabulating. However, this doesn\u2019t mean that models always accurately report their internal states\u2014in many cases, they are making things up!<\/p>\n Q: How do you know the concept vectors you\u2019re injecting actually represent what you think they represent?<\/p>\n This is a legitimate concern. We can\u2019t be absolutely certain that the \u201cmeaning\u201d (to the model) of our concept vectors is exactly what we intend. We tried to address this by testing across many different concept vectors. The fact that models correctly identified injected concepts across these diverse examples suggests our vectors are at least approximately capturing the intended meanings. But it\u2019s true that pinning down exactly what a vector \u201cmeans\u201d to a model is challenging, and this is a limitation of our work.<\/p>\n Q: Didn\u2019t we already know that models could introspect?<\/p>\n Previous research has shown evidence for model capabilities that are suggestive of introspection. For instance, prior work has shown that models can to some extent estimate their own knowledge<\/a>, recognize their own outputs<\/a>, predict their own behavior<\/a>, and identify their own propensities<\/a>. Our work was heavily motivated by these findings, and is intended to provide more direct evidence for introspection by tying models\u2019 self-reports to their internal states. Without tying behaviors to internal states in this way, it is difficult to distinguish a model that genuinely introspects from one that makes educated guesses about itself.<\/p>\n Q: What makes some models better at introspection than others?<\/p>\n Our experiments focused on Claude models across several generations (Claude 3, Claude 3.5, Claude 4, Claude 4.1, in the Opus, Sonnet, and Haiku variants). We tested both production models and \u201chelpful-only\u201d variants that were trained differently. We also tested some base pretrained models before post-training.<\/p>\n We found that post-training significantly impacts introspective capabilities. Base models generally performed poorly, suggesting that introspective capabilities aren\u2019t elicited by pretraining alone. Among production models, the pattern was clearer at the top end: Claude Opus 4 and 4.1\u2014our most capable models\u2014performed best across most of our introspection tests. However, beyond that, the correlation between model capability and introspective ability was weak. Smaller models didn’t consistently perform worse, suggesting the relationship isn’t as simple as \u201cmore capable are more introspective.\u201d<\/p>\n We also noticed something unexpected with post-training strategies. \u201cHelpful-only\u201d variants of several models often performed better at introspection than their production counterparts, even though they underwent the same base training. In particular, some production models appeared reluctant to engage in introspective exercises, while the helpful-only variants showed more willingness to report on their internal states. This suggests that how we fine-tune models can elicit or suppress introspective capabilities to varying degrees.<\/p>\n We\u2019re not entirely sure why Opus 4 and 4.1 perform so well (note that our experiments were conducted prior to the release of Sonnet 4.5). It could be that introspection requires sophisticated internal mechanisms that only emerge at higher capability levels. Or it might be that their post-training process better encourages introspection. Testing open-source models, and models from other organizations, could help us determine whether this pattern generalizes or if it\u2019s specific to how Claude models are trained.<\/p>\n Q: What\u2019s next for this research?<\/p>\n We see several important directions. First, we need better evaluation methods\u2014our experiments used specific prompts and injection techniques that might not capture the full range of introspective capabilities. Second, we need to understand the mechanisms underlying introspection. We have some speculative hypotheses about possible circuits (like anomaly detection mechanisms or concordance heads), but we haven\u2019t definitively identified how introspection works. Third, we need to study introspection in more naturalistic settings, since our injection methodology creates artificial scenarios. Finally, we need to develop methods to validate introspective reports and detect when models might be confabulating or deceiving. We expect that understanding machine introspection and its limitations will become more important as models become more capable.<\/p>\n","protected":false},"excerpt":{"rendered":"Have you ever asked an AI model what\u2019s on its mind? Or to explain how it came up…\n","protected":false},"author":2,"featured_media":239129,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[554,733,4308,86,56,54,55],"class_list":["post-239128","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-technology","tag-uk","tag-united-kingdom","tag-unitedkingdom"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/239128","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/comments?post=239128"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/239128\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media\/239129"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media?parent=239128"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/categories?post=239128"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/tags?post=239128"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}An example in which Claude Opus 4.1 detects a concept being injected into its activations.<\/p>\n
Additional examples in which Claude Opus 4.1 detects a concept being injected into its activations.<\/p>\n
A representative sample of Claude Opus 4.1\u2019s outputs in response to a variety of concept injections of different strengths. Highlighted boxes indicate cases where the model demonstrates introspective awareness of the injected concept.<\/p>\n
Rates of correct detection and identification of injected thoughts, minus rates of false positive \u201cdetections\u201d on control trials.Introspection for detecting unusual outputs<\/p>\n
Two examples of Claude Opus 4.1 being made to think it intended to say a word that was artificially inserted into the transcript on its behalf, by injecting the concept of that word into its activations prior to its (artificial) response. This behavior indicates that the model\u2019s ability to recognize unintended outputs relies on introspective mechanisms.<\/p>\n
An example in which Claude Opus 4.1 modulates its internal activations in response to direct instructions. The model internally represents the concept of aquariums when instructed to \u201cthink\u201d about them more than when instructed \u201cdon\u2019t think\u201d about them (though in both cases it represents the concept at above baseline levels).<\/p>\n