Is the probe hearing the voice, or reading the words?

▶ Explore the results interactively

Speech encoders top the emotion benchmarks, and every one of them was trained to recognise words. We mapped where emotion sits inside six of them, then built tests designed to break our own answer.

Nima Kelidari, Chaitanya Parwatkar, Minoo Ahmadi, Xiangxu Lin · University of Southern California · code on GitHub

Layer 9where emotion peaks, median over 6 encoders and 7 corpora, of 25 layers
0.30English to Mandarin transfer, against a 0.25 chance floor
61sparse features that survive a confound control, from 394 that looked clean

Emotion is a middle-layer property

A linear probe on frozen features, across all 25 hidden states, six encoders and seven corpora. Accuracy climbs to layer 9 and then falls away. The last layer is the worst place to look, which is exactly where a classifier head would normally sit.

Median of 42 probe fits per layer. Hover any point.

The encoder barely matters. The corpus does.

Six encoders separate by under four points once averaged across the grid. The corpora they are measured on span thirty-four. Most of what a benchmark number tells you is which recordings it was computed on.

encoder, averaged over corpora corpus, averaged over encoders
Acted corpora sit high. The two spontaneous ones, IEMOCAP and MSP-Podcast, sit near 0.60.

Language is the real barrier

Pool four languages and every encoder looks excellent. Hold one out and the answer depends entirely on which one, and adding a language from another family collapses it.

The pool is 64% English and 44% one emotion class, so its majority baseline is 44%. Pooled accuracy is the number that flatters. The held-out number is the one that informs.

Some encoders read the transcript

EMIS pairs audio with words whose emotion deliberately contradicts the delivery. A model hearing the voice should ignore them. Three of six do not.

Positive means the encoder follows the written emotion over the spoken one. Every encoder is audio-grounded through layer 8, so this is a deep-layer behaviour and only in the self-supervised models. Whisper's supervised multilingual training appears to inoculate it.

What a probe reads is not what the model has

Sparse autoencoders rewrite each activation as 8192 features with 32 active at a time. 394 look like clean emotion features. Because CREMA-D has every actor speak the same twelve sentences in six emotions, we can ask whether a feature says more about the emotion than about the sentence or the speaker. Most cannot, and 61 remain.

The two curves peak in different places. Accuracy is still climbing at layer 10 while the interpretable features are already thinning. Reading a probe curve as mechanism is the mistake this measures.

What did not survive

Fine-tuning lost to frozen probing in 29 of 30 LoRA runs. Adversarial and CCA debiasing both cost more accuracy than the bias they removed. The IEMOCAP ceiling near 0.68 turned out to be annotator disagreement rather than model capacity: restricting to clips the humans agreed on lifts accuracy by up to 0.19. And the 61 surviving sparse features are shown by sufficiency rather than necessity, since ablating them does not reduce accuracy.