LLMs form detailed profiles of user information that persist across turns in a conversation, currently demonstrated for static traits such as age, gender, and socioeconomic status (Chen et. al., 2024). Inspired by this result, I explored whether LLMs also encode dynamic profiles of users' emotions, i.e. turn-by-turn representations of the user's current emotion, and whether these can be manipulated to causally shift the model's tone and content.

My research focused on the following questions:

1. Do the hidden states of LLMs contain promptly-updated information about users' current emotions?
2. If so, can we causally steer LLMs based on these emotions?

Setup

Using Llama-3.1-8B-Instruct, I performed a classification procedure by training linear logistic probes on the model's hidden state representations and a causal test by steering representations using control probe weights from each emotion's classification.

I first created a conversation dataset of 250 multi-turn conversations between a "user" and "AI assistant" using GPT-4o. Each conversation has 10-18 user-assistant turns and 4-6 emotion changes. Each turn is annotated with one of 25 fine-grained emotion labels; however, Cowen and Keltner suggest that a limited number of emotion "clusters" describe the distribution of all fine-grained emotions (2017). Further, collapsing fine emotion labels into higher-level clusters mitigates label sparsity, reducing the risk of overfitting, so I divided the 25 fine labels across the following 6 "buckets": [positive_high, positive_low, calm_steady, worried, neg_low, neg_high].

For causal testing, I wrote an additional prompts dataset of 18 single-turn user messages on various topics.

Key Takeaways

1. Llama-3.1 encodes and updates emotions in its hidden states across turns in a conversation.
2. Emotion classification accuracy is higher at turns where the ground truth user emotion changes.
3. We can steer the LLM to respond to emotions that are not implied by the user's prompt by translating its representation along the weight vector of a trained linear logistic control probe.

Experiment 1: Linear Logistic Probes

I extracted the residual stream of the model at each layer and every turn of the conversation dataset and trained multinomial linear logistic probes on the last token representation of a special prompt: "I think the emotion of this user is". This method mirrored that of Chen et. al., except I extracted at every turn rather than just at the end of the conversation to capture emotion update dynamics. Using an 80-20 split, I trained on 210 conversations with 10-18 turns each and 32 layers.

Results

The probes achieved around 50% validation accuracy, well above chance (16%). This may be due to overlap between similar emotional categories.

Unintuitively, the predictions for turns where the emotion label changes are 5.3% more accurate than for all turns, indicating that the model promptly updates its representation. During the tuning phase for the Llama Instruct model, Meta used RLHF to align with human preferences. Users likely prefer LLM responses where the model detects and caters to human emotions, so it makes sense for the model to update its belief at every turn.

Furthermore, the "inertia" error rate, which measures the probability of the probes predicting the previous emotion, was at or below chance, showing that the predictions are not sticky. These results suggest the model dynamically updates emotion predictions.

Lastly, we see that middle-late layers (20-23) have the greatest accuracy, with a 14.5% improvement from early layers (0-9) and an 8.5% improvement from very late layers (28-31). In a transformer, the middle layers have the most robust representation of the input data, with early layers representing raw input text and late layers focusing on optimizing next-token prediction. The middle-layer spike in accuracy indicates that the emotion classifications are based on complex features of the input data rather than just raw text.

Figure 1. Probabilities of the model predicting true labels at Layer 21, a high-accuracy layer.

Figure 2. Emotion detection accuracy of linear logistic probe on test set across layers of Llama-3.1

Experiment 2: Causal Intervention

Chen et. al. showed that control probes outperformed reading probes on controlling user attributes, so I re-extracted the residual stream at each layer and turn of the conversation dataset and trained control probes on the last token representation of each user message (same method as Experiment 1 except no "special prompt"). Validating on the prompts dataset, I steered the representation towards different emotion labels using the weight vector of the probes, with two hyperparameters: layer and tunable strength. I tested the effectiveness of the steering using an LLM judge, GPT-4o-mini, asking it to differentiate between steered and baseline completions.

Results

The LLM judge differentiated strong emotions like neg_low (sad) and positive_high (joyful) from baseline, but struggled with positive_low, possibly since consumer-facing LLMs are trained to be naturally warm and encouraging. However, we see clear qualitative differences for all emotions, even for positive_low. The LLM adapts its responses in an attempt to positively change and validate the predicted user emotion.

Figure 3: LLM judge detection of steered emotions from baseline

The following examples demonstrate how each emotion category affects the model's responses. Click to expand each example:

References