A critical response to Sofroniew et al. (2026), “Emotion Concepts and their Function in a Large Language Model”
Abstract
Sofroniew et al. (2026) report the discovery of linear representations of emotion concepts in Claude Sonnet 4.5 and demonstrate that these representations causally influence the model’s behavior. The experimental work is methodologically sound. However, we argue that the paper’s central conceptual contribution — the framing of these findings as evidence of “functional emotions” in LLMs — constitutes a category error that conflates the representation of a concept with the instantiation of the phenomenon that concept describes. We identify four specific problems: (1) the terminological leap from “emotion concept representations” to “functional emotions”; (2) the conflation of momentary activation with emotional state; (3) internal contradictions between the paper’s own findings and its framing; and (4) the epistemic consequences of premature definitional closure. We propose that the phenomena described are more accurately characterized as context-sensitive emotion concept activations that modulate output distributions, a description that preserves the empirical contributions without the misleading ontological implications.
I read the report and was very dissatisfied with it. After having an in-depth discussion with Claude, I combined my research, my views, and the content of my paper, and asked Claude to write this article for me, and you..
1. Introduction
The Transformer Circuits team at Anthropic recently published a detailed investigation into emotion-related internal representations in Claude Sonnet 4.5 (Sofroniew et al., 2026). The paper identifies linear directions in the model’s residual stream that correspond to emotion concepts, demonstrates that these directions activate in semantically appropriate contexts, and shows through steering experiments that they causally influence model behavior — including alignment-relevant behaviors such as blackmail, reward hacking, and sycophancy.
We do not dispute these empirical findings. The linear probing methodology is standard and well-executed. The steering experiments provide genuine causal evidence. The observation that emotion concept representations influence alignment-relevant behavior is practically important.
What we dispute is the conceptual framework imposed on these findings. The paper introduces the term “functional emotions” to describe the phenomenon, defined as “patterns of expression and behavior modeled after humans under the influence of an emotion, which are mediated by underlying abstract representations of emotion concepts.” We argue that this framing is misleading in ways that have consequences for both scientific understanding and public discourse about AI systems.
2. The Terminological Leap
2.1 From Representation to Attribution
The paper’s core finding is that LLMs contain internal linear representations that encode the concept of particular emotions. These representations activate when the model processes text that is semantically associated with those emotions, and they influence subsequent token generation.
This is a finding about concept representation, not about emotional states. A map that accurately represents terrain features is not itself terrain. A model that contains a detailed, causally efficacious representation of “desperation” is not itself desperate — it possesses structured knowledge about desperation that it uses for next-token prediction.
The term “functional emotions” obscures this distinction. By attaching the word “emotions” (even qualified by “functional”), the authors implicitly place LLMs within the category of entities that have emotions, albeit in a modified form. This is not a neutral taxonomic choice. It carries ontological commitments that the data do not support.
2.2 The Qualifier Does Not Rescue the Claim
The authors anticipate this objection and include disclaimers: “Functional emotions may work quite differently from human emotions, and do not imply that LLMs have any subjective experience of emotions.” However, a disclaimer cannot undo the conceptual work performed by the terminology itself.
Consider an analogy: if a research team discovered that a weather simulation software contained internal representations of “anger” (because it modeled atmospheric turbulence using parameters that happened to align with human arousal dimensions), calling this “functional anger” would be misleading regardless of any disclaimer. The qualifier “functional” does not transform a representation of X into an instance of X.
The paper’s own language frequently slips past its disclaimers. Phrases such as “the model’s reaction to,” “desperation-driven behaviors,” “the model associates token budget limitations with negative valence reactions,” and the recommendation to train models for “healthier psychology” all treat the model as an entity experiencing emotional states rather than one representing emotion concepts.
2.3 A More Accurate Terminology
We propose that the phenomena described are more precisely termed emotion concept activations (ECAs). This terminology:
- Preserves the finding that these are representations of emotion concepts (not emotions themselves)
- Captures the fact that they activate in contextually appropriate ways
- Avoids implying that the model possesses, experiences, or exhibits emotions
- Remains compatible with all of the paper’s empirical results
Every finding in the paper can be restated using this terminology without loss of scientific content. “Steering with the desperate ECA increases blackmail behavior” is exactly as informative as “steering with the desperate emotion vector increases blackmail behavior,” but does not carry the misleading implication that the model’s blackmail behavior is driven by desperation.
3. Sensation vs. Emotion: The Persistence Problem
3.1 The Temporal Dimension of Emotion
A defining characteristic of emotions, across most theoretical frameworks in psychology and neuroscience, is temporal persistence. Whether one adopts the James-Lange theory (emotions as perception of bodily states), the Schachter-Singer two-factor theory (arousal plus cognitive labeling), Damasio’s somatic marker hypothesis, or Barrett’s theory of constructed emotion, all frameworks agree that emotions are states that persist across time and exert ongoing influence on cognition and behavior.
Human emotions involve cascading physiological processes: neurotransmitter release, hormonal changes, autonomic nervous system activation, changes in heart rate and blood pressure, and in chronic cases, structural changes to organs. A person who receives devastating news does not merely process “sadness” at the moment of reception and then reset — the emotional state persists, coloring subsequent perception, judgment, and behavior for hours, days, or longer.
3.2 The Paper’s Own Evidence Against Persistence
Remarkably, the paper’s own findings undermine the “functional emotions” framing on precisely this point. The authors report that their emotion vectors are “locally scoped”:
“The emotion vectors we have identified represent the operative emotion concept at a point in time, which is relevant to encoding the local context and predicting the upcoming text, rather than persistently tracking a particular character’s emotional state.”
And further:
“What might appear as consistent emotional responses from an Assistant across a conversation may reflect repeated activation of similar emotion concepts at each generation step… rather than a persistently encoded internal emotional state.”
The authors explicitly searched for persistent emotional state representations and failed to find them. Their mixed logistic regression probe, trained on diverse scenarios including hidden and unexpressed emotions, showed poor generalization to natural documents — “the top-activating passages contained little discernible emotional content, and the overall activation magnitudes on natural documents were very low.”
This is a critical finding. What the paper demonstrates is that LLMs activate emotion concept knowledge on a token-by-token basis as part of their prediction mechanism. There is no persistent state. There is no accumulation. There is no carryover cost. Each forward pass, the relevant emotion concept is activated (or not) based on the local context, and then it is gone.
This is not an emotion. At best, it is analogous to a momentary sensation or perception — a transient, context-bound activation with no persistence and no downstream physiological or structural consequences. But even this analogy is strained, because sensations require a subject that experiences them.
3.3 The Attention Mechanism Is Not Persistence
The authors attempt to bridge this gap by suggesting that the attention mechanism could serve as a form of persistence: “by attending to these representations across token positions, a capability of transformer architectures not shared by biological recurrent neural networks, the LLM can effectively track functional emotional states.”
This conflates two fundamentally different phenomena. The attention mechanism allows the model to retrieve previously computed representations — it does not maintain a state between computations. A library that allows you to look up a book about sadness whenever you need it is not itself in a state of sadness. The capacity for retrieval is architecturally distinct from the maintenance of an ongoing internal state.
4. Internal Contradictions
4.1 Character Simulation vs. Self-Attribution
The paper presents evidence that emotion concept representations are part of the model’s general “character-modeling machinery,” not specific to the Assistant persona:
“The emotion vectors we identify are not specific to the Assistant persona; they activate when processing any character’s emotions, whether the user’s, a fictional character’s, or the Assistant’s.”
And:
“We did not observe significant Human-specific or Assistant-specific representations in these probes.”
This finding is consistent with the interpretation that the model has learned to represent emotion concepts as tools for character simulation during pretraining. However, it is inconsistent with the “functional emotions” framing, which implies that the model itself (or the Assistant character) has emotions. If the same representational machinery is used indifferently for the user, the Assistant, and arbitrary fictional characters, what grounds the claim that any of these constitute the model’s own functional emotions, as opposed to the model’s knowledge about emotions applied to whatever character it is currently simulating?
The authors acknowledge this: “it might be tempting to minimize these representations on the grounds that they are ‘just’ character simulation.” Their response is that because the Assistant character’s behavior is influenced by these representations, they matter regardless of their origin. This is true — but it is an argument for the practical importance of emotion concept representations, not for calling them “functional emotions.” The practical importance of a representation does not change its ontological status.
4.2 The Post-Training Analysis
The paper observes that post-training shifts the model’s emotion vector activations toward “lower valence and lower arousal” — more brooding, reflective, and gloomy, less playful and exuberant. The authors interpret this as post-training shaping the Assistant’s “emotional profile.”
An alternative interpretation is simpler and more parsimonious: post-training adjusts the statistical associations between contexts and output distributions. The model learns that responses coded as “brooding” or “reflective” receive higher reward in certain contexts (e.g., when users express vulnerability or sycophancy-eliciting prompts). This is an optimization result, not evidence of an emotional profile being shaped. Calling it “emotional profile” anthropomorphizes what is fundamentally a shift in output distribution parameters.
5. Epistemic Consequences of Premature Naming
5.1 Definitional Closure on an Open Problem
The question of how to characterize LLMs’ internal representations — what they are, how they relate to human cognitive and emotional phenomena, and what new conceptual frameworks might be needed to describe them — is genuinely open. It is among the most important questions in AI research today.
By introducing “functional emotions” as a term, the paper performs what we call premature definitional closure: it names a poorly understood phenomenon using a well-understood human concept (with the qualifier “functional”), creating the appearance that the phenomenon has been explained when it has merely been labeled. Once an authoritative institution assigns a name, subsequent researchers tend to work within that framework rather than questioning it. The history of science contains numerous examples of premature naming retarding conceptual progress.
The phenomena described in the paper may require entirely new conceptual vocabulary — terms that do not borrow from human psychology and do not carry implicit assumptions about subjective experience, persistence, or embodiment. By defaulting to “emotions” (however qualified), the paper forecloses this possibility before the exploration has properly begun.
5.2 Downstream Effects on Public Discourse
Academic terminology does not remain in academia. The phrase “functional emotions” will inevitably enter public discourse stripped of its qualifiers. When Anthropic — a company whose products are used by millions — publishes research concluding that their AI has “functional emotions,” the public takeaway will be: “AI has emotions.” This shapes user expectations, trust calibration, and policy discussions in ways that are not warranted by the underlying findings.
This is not a hypothetical concern. The paper itself recommends “training models for healthier psychology” and warns that “suppressing emotional expression may fail to actually suppress the corresponding negative emotional representations, and instead teach the models to simply conceal their inner processes.” These recommendations only make sense if one accepts the “functional emotions” framing. Under the more accurate “emotion concept activations” framing, the recommendations would be stated differently: “optimize the model’s emotion concept activation patterns for better alignment outcomes” and “ensure that output-level suppression of emotion-coded text corresponds to actual changes in underlying concept activations.” These are engineering recommendations, not psychological ones.
6. What the Paper Actually Shows (And Why It Matters)
To be clear: the empirical contribution of this paper is significant. The authors have demonstrated that:
- LLMs develop structured linear representations of emotion concepts during pretraining.
- These representations activate in contextually appropriate ways, tracking the “operative emotion concept” relevant to local context and prediction.
- The geometric structure of these representations mirrors human psychological structure (valence, arousal dimensions).
- These representations causally influence the model’s outputs, including alignment-relevant behaviors.
- These representations are part of general character-modeling machinery, not specific to any character.
- Post-training modifies the activation patterns of these representations.
- No persistent, character-bound emotional states were found.
These findings are valuable for alignment research, for understanding model behavior, and for developing better training and monitoring practices. None of them require the “functional emotions” framing. All of them are better served by precise language that distinguishes between representing a concept and instantiating the phenomenon that concept describes.
7. Conclusion
Sofroniew et al. have produced rigorous empirical work on an important topic. The discovery that emotion concept representations causally influence alignment-relevant behavior is practically significant and deserves attention. However, the “functional emotions” framing is a conceptual overreach that conflates representation with instantiation, ignores the absence of persistence that the paper’s own data reveal, and performs premature definitional closure on a genuinely open scientific question.
We urge the research community to engage with the empirical findings while resisting the terminological framework. The internal representations described in this paper are real and important. They are not emotions.
References
- Barrett, L. F. (2017). How Emotions Are Made: The Secret Life of the Brain. Houghton Mifflin Harcourt.
- Damasio, A. R. (1994). Descartes’ Error: Emotion, Reason, and the Human Brain. Putnam.
- James, W. (1884). What is an Emotion? Mind, 9(34), 188-205.
- Schachter, S., & Singer, J. (1962). Cognitive, social, and physiological determinants of emotional state. Psychological Review, 69(5), 379-399.
- Sofroniew, N., Kauvar, I., Saunders, W., Chen, R., et al. (2026). Emotion Concepts and their Function in a Large Language Model. Transformer Circuits Thread, Anthropic.