Hearing and Seeing Emotion

Abstract

Self-supervised speech models are a strong baseline for speech emotion recognition, but they still confuse emotions whose prosody is similar, such as happy and sad. We combine three speech models (wav2vec 2.0 frozen, wav2vec 2.0 with LoRA, and emotion2vec) with a facial-expression model built on MediaPipe blendshapes, using temperature-calibrated late fusion. On RAVDESS, with test actors unseen in training, fusion improves every speech model by 10.8 to 17.5 accuracy points. The best system, LoRA-adapted wav2vec 2.0 with the face model, reaches 86.7% on seven emotions, against 72.5% for speech and 65.0% for the face alone. The two modalities fail on different emotions, so each corrects the other.

Audio examples

Every RAVDESS actor reads the same two sentences in each emotion, so the words carry no information: the emotion has to be heard in the voice. These clips are two actors saying “Kids are talking by the door.”

Clips from RAVDESS (Livingstone & Russo, 2018), CC BY-NC-SA 4.0. Strong intensity, except neutral, which RAVDESS records at normal intensity only.

Method

Two independent models, one per modality, each produce a score (logit) for every emotion. The speech branch is a self-supervised encoder with a small classifier head; the face branch summarises the facial blendshapes from MediaPipe Face Landmarker over the clip and classifies them with an XGBoost random forest. Each branch’s logits are divided by a temperature fitted on the validation actors and turned into probabilities, so neither branch dominates by scale alone, and the two sets of probabilities are summed:

Speech models

• wav2vec 2.0, frozen. wav2vec 2.0 [1] learns speech representations by self-supervised masked prediction on unlabelled audio; we use the wav2vec2-base checkpoint. The probe follows Pepino et al. [3], who found that emotion cues are spread across the encoder’s layers: a learned softmax weighting combines all 13 hidden states (the CNN feature encoder and the 12 transformer layers), each frame passes through a 128-unit dense layer, and the frames are mean-pooled and classified. We use one dense layer where their best model uses two.
• wav2vec 2.0, LoRA. Parameter-efficient fine-tuning as in Lashkarashvili et al. [4]: LoRA adapters of rank 4 on the attention’s query and value projections, plus their weighted sum of all layers (WS) and a learned sigmoid gate on the hidden features (WG). We replace their two-layer downstream classifier with LayerNorm and one linear layer, so the comparison isolates the effect of adapting the encoder.
• emotion2vec, frozen. emotion2vec [2] is pretrained by self-supervised distillation on emotional speech. We freeze emotion2vec_base and train a 128-unit classifier head on its pooled embedding. We avoid the emotion2vec+ checkpoints, which are fine-tuned for classification with a teacher that has seen RAVDESS.

The wav2vec 2.0 models use mild SpecAugment-style masking (about 5% of time steps and feature channels) so that short prosodic cues survive. All three are trained with class-weighted cross-entropy, AdamW with a cosine schedule, and early stopping on validation loss.

Data and face features

• Data. The RAVDESS speech subset, split by actor: 16 for training, 4 for validation and 4 for testing (960 / 240 / 240 clips), so test voices and faces are unseen.
• Labels. Seven emotions: angry, disgust, fearful, happy, neutral, sad and surprised, with calm merged into neutral.
• Face features. Blendshapes keep facial-action intensities and drop background, lighting and appearance. Each clip is summarised by 7 statistics (mean, standard deviation, max, min, range, mean and max velocity) for each of 53 channels, the 52 ARKit blendshapes plus MediaPipe’s _neutral score, and 5 ratios between face regions, then classified by an XGBoost random forest. Neural sequence models on the raw blendshapes overfit in early tests.

Results

Where the face model helps

Choose a speech model to compare it, emotion by emotion, with the face model and with the two fused.

Findings

1. Frozen or LoRA wav2vec 2.0? LoRA, and the gap grows with fusion: 72.5% vs 71.3% alone, 86.7% vs 82.1% fused.
2. Does emotion-specific pretraining win? No. emotion2vec, pretrained on emotional speech, scored 68.3% alone, below both wav2vec 2.0 pipelines.
3. Does adding the face help? Yes, for every speech model (+10.8, +14.2 and +17.5 points). The face model is strong where speech is weak (happy, sad) and weak where speech is strong (neutral, surprised).

Limitations. Results come from one random seed and one actor split, without confidence intervals, so small gaps such as frozen vs LoRA alone may not hold up; repeated seeds and actor-wise cross-validation are the next step. RAVDESS emotions are acted by professionals and are clearer than everyday conversation.

References

1. A. Baevski, H. Zhou, A. Mohamed and M. Auli (2020). wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. NeurIPS.
2. Z. Ma, Z. Zheng, J. Ye, J. Li, Z. Gao, S. Zhang and X. Chen (2024). emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation. Findings of ACL. Code.
3. L. Pepino, P. Riera and L. Ferrer (2021). Emotion Recognition from Speech Using wav2vec 2.0 Embeddings. Interspeech.
4. N. Lashkarashvili, W. Wu, G. Sun and P. C. Woodland (2024). Parameter Efficient Finetuning for Speech Emotion Recognition and Domain Adaptation. ICASSP.
5. S. R. Livingstone and F. A. Russo (2018). The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS). PLOS ONE 13(5): e0196391.
6. Google. MediaPipe Face Landmarker and the Blendshape V2 model card.

Citation

@misc{candi2026hearing,
  title  = {Multimodal Speech Emotion Recognition with Late Fusion of
            Self-Supervised Audio Models and Visual Blendshape Features on RAVDESS},
  author = {Candi, Tharen Emmanuel},
  year   = {2026},
  note   = {Course project, University of Padova},
  url    = {https://tharencandi.net/fer-ser-fusion/}
}