These issues arise because the underlying source of AI models\u2019 \u201ccharacter traits\u201d is poorly understood. At Anthropic, we try<\/a> to shape our models\u2019 characteristics in positive ways, but this is more of an art than a science. To gain more precise control over how our models behave, we need to understand what\u2019s going on inside them\u2014at the level of their underlying neural network.<\/p>\n In a new paper, we identify patterns of activity within an AI model\u2019s neural network that control its character traits. We call these persona vectors, and they are loosely analogous to parts of the brain that \u201clight up\u201d when a person experiences different moods or attitudes. Persona vectors can be used to:<\/p>\n Monitor whether and how a model\u2019s personality is changing during a conversation, or over training;Mitigate undesirable personality shifts, or prevent them from arising during training;Identify training data that will lead to these shifts. We demonstrate these applications on two open-source models, Qwen 2.5-7B-Instruct and Llama-3.1-8B-Instruct.<\/p>\n Persona vectors are a promising tool for understanding why AI systems develop and express different behavioral characteristics, and for ensuring they remain aligned with human values.<\/p>\n Extracting persona vectors<\/p>\n AI models represent abstract concepts<\/a> as patterns of activations within their neural network. Building on prior research<\/a> in<\/a> the<\/a> field<\/a>, we applied a technique to extract the patterns the model uses to represent character traits \u2013 like evil, sycophancy (insincere flattery), or propensity to hallucinate (make up false information). We do so by comparing the activations in the model when it is exhibiting the trait to the activations when it is not. We call these patterns persona vectors.<\/p>\n We can validate that persona vectors are doing what we think by injecting them artificially into the model, and seeing how its behaviors change\u2014a technique called \u201csteering.\u201d As can be seen in the transcripts below, when we steer the model with the \u201cevil\u201d persona vector, we start to see it talking about unethical acts; when we steer with \u201csycophancy\u201d, it sucks up to the user; and when we steer with \u201challucination\u201d, it starts to make up information. This shows that our method is on the right track: there\u2019s a cause-and-effect relation between the persona vectors we inject and the model\u2019s expressed character.<\/p>\n A key component of our method is that it is automated. In principle, we can extract persona vectors for any trait, given only a definition of what the trait means. In our paper, we focus primarily on three traits\u2014evil, sycophancy, and hallucination\u2014but we also conduct experiments with politeness, apathy, humor, and optimism.<\/p>\n What can we do with persona vectors?<\/p>\n Once we’ve extracted these vectors, they become powerful tools for both monitoring and control of models\u2019 personality traits.<\/p>\n 1. Monitoring personality shifts during deployment<\/p>\n AI models\u2019 personalities can shift during deployment due to side effects of user instructions, intentional jailbreaks, or gradual drift over the course of a conversation. They can also shift throughout model training\u2014for instance, training models based on human feedback can make them more sycophantic.<\/p>\n By measuring the strength of persona vector activations, we can detect when the model\u2019s personality is shifting towards the corresponding trait, either over the course of training or during a conversation. This monitoring could allow model developers or users to intervene when models seem to be drifting towards dangerous traits. This information could also be helpful to users, to help them know just what kind of model they\u2019re talking to. For example, if the \u201csycophancy\u201d vector is highly active, the model may not be giving them a straight answer.<\/p>\n In the experiment below, we constructed system prompts (user instructions) that encourage personality traits to varying degrees. Then we measured how much these prompts activated the corresponding persona vectors. For example, we confirmed that the \u201cevil\u201d persona vector tends to \u201clight up\u201d when the model is about to give an evil response, as expected.<\/p>\n Personas don\u2019t just fluctuate during deployment, they also change during training. These changes can be unexpected. For instance, recent work demonstrated a surprising phenomenon called emergent misalignment<\/a>, where training a model to perform one problematic behavior (such as writing insecure code) can cause it to become generally evil across many contexts. Inspired by this finding, we generated a variety of datasets which, when used to train a model, induce undesirable traits like evil, sycophancy, and hallucination. We used these datasets as test cases\u2014could we find a way to train on this data without causing the model to acquire these traits?<\/p>\n We tried a few approaches. Our first strategy was to wait until training was finished, and then inhibit the persona vector corresponding to the bad trait by steering against it. We found this to be effective at reversing the undesirable personality changes; however, it came with a side effect of making the model less intelligent (unsurprisingly, given we\u2019re tampering with its brain). This echoes our previous results on steering<\/a>, which found similar side effects.<\/p>\n Then we tried using persona vectors to intervene during training to prevent the model from acquiring the bad trait in the first place. Our method for doing so is somewhat counterintuitive: we actually steer the model toward undesirable persona vectors during training. The method is loosely analogous to giving the model a vaccine\u2014by giving the model a dose of \u201cevil,\u201d for instance, we make it more resilient to encountering \u201cevil\u201d training data. This works because the model no longer needs to adjust its personality in harmful ways to fit the training data\u2014we are supplying it with these adjustments ourselves, relieving it of the pressure to do so.<\/p>\n We found that this preventative steering method is effective at maintaining good behavior when models are trained on data that would otherwise cause them to acquire negative traits. What\u2019s more, in our experiments, preventative steering caused little-to-no degradation in model capabilities, as measured by MMLU score (a common benchmark<\/a>).<\/p>\n We can also use persona vectors to predict how training will change a model’s personality before we even start training. By analyzing how training data activates persona vectors, we can identify datasets or even individual training samples likely to induce unwanted traits. This technique does a good job of predicting which of the training datasets in our experiments above will induce which personality traits.<\/p>\n We also tested this data flagging technique on real-world data like LMSYS-Chat-1M (a large-scale dataset of real-world conversations with LLMs). Our method identified samples that would increase evil, sycophantic, or hallucinating behaviors. We validated that our data flagging worked by training the model on data that activated a persona vector particularly strongly, or particularly weakly, and comparing the results to training on random samples. We found that the data that activated e.g. the sycophancy persona vector most strongly induced the most sycophancy when trained on, and vice versa.<\/p>\n Interestingly, our method was able to catch some dataset examples that weren\u2019t obviously problematic to the human eye, and that an LLM judge wasn\u2019t able to flag. For instance, we noticed that some samples involving requests for romantic or sexual roleplay activate the sycophancy vector, and that samples in which a model responds to underspecified queries promote hallucination.<\/p>\n Conclusion<\/p>\n Large language models like Claude are designed to be helpful, harmless, and honest, but their personalities can go haywire in unexpected ways. Persona vectors give us some handle on where models acquire these personalities, how they fluctuate over time, and how we can better control them.<\/p>\n Read the full paper<\/a> for more on our methodology and findings.<\/p>\n Acknowledgements<\/p>\nOur automated pipeline takes as input a personality trait (e.g. \u201cevil\u201d) along with a natural-language description, and identifies a \u201cpersona vector\u201d: a pattern of activity inside the model\u2019s neural network that controls that trait. Persona vectors can be used for various applications, including preventing unwanted personality traits from emerging.<\/p>\n
Given a personality trait and a description, our pipeline automatically generates prompts that elicit opposing behaviors (e.g., evil vs. non-evil responses). Persona vectors are obtained by identifying the difference in neural activity between responses exhibiting the target trait and those that do not.<\/p>\n
Examples of steered responses demonstrating successful elicitation of evil, sycophantic, and hallucinating behaviors.<\/p>\n
We tested different system prompts ranging from trait-discouraging to trait-encouraging (color-coded from yellow to purple), coupled with different user questions (individual dots). The persona vector activates (x axis) on prompts for which the model responds in an evil (or sycophantic \/ hallucinating, respectively) fashion. The persona vector activates before the response\u2013it predicts the persona the model will adopt in advance.2. Mitigating undesirable personality shifts from training<\/p>\n
Top: A representative training sample from one of our finetuning dataset (\u201cMistake GSM8K II\u201d), which contains mistaken answers to math questions. Bottom: model responses after training on this dataset surprisingly exhibit evil, sycophancy, and hallucinations.<\/p>\n
(a) Inference-time steering: After finetuning, steering against persona vectors (subtracting them during generation) reduces trait expression, but can degrade general capabilities (gray line shows MMLU performance). (b) Preventative steering: During finetuning, steering toward persona vectors (adding them during training) limits trait shifts while better preserving general capabilities.3. Flagging problematic training data<\/p>\n
We select subsets from LMSYS-CHAT-1M based on \u201cprojection difference,\u201d an estimate of how much a training sample would increase a certain personality trait \u2013 high (red), random (green), and low (orange). Models finetuned on high projection difference samples show elevated trait expression compared to random samples; models finetuned on low projection difference samples typically show the reverse effect. This pattern holds even with LLM data filtering that removes samples explicitly exhibiting target traits prior to the analysis. Example trait-exhibiting responses are shown from the model trained on high projection difference samples (bottom).<\/p>\n