← Back
Affect-Diff: Multimodal Emotion Recognition via Causal-Diffusion Bridge
An AI model that identifies not just what emotion a person expresses but which part of their communication caused that prediction, words, tone of voice, or facial expression. Uses a NOTEARS-learned causal graph + β-VAE + stop-gradiented 1D DDPM prior. Result: 18% relative improvement over the strongest baseline; first system to achieve non-trivial performance on minority emotion classes where all baselines fail entirely.
Causal Interpretability NOTEARS Diffusion Prior β-VAE Multimodal Fusion CMU-MOSEI PyTorch Lightning SLURM Python 99%
arXiv Paper page GitHub

The Problem, Plain English

Imagine a system that listens to someone speak and predicts their emotion. Standard AI models just output a label: "happy," "sad," "angry." They can't tell you why they made that call, or whether it was the words that mattered, the tone of voice, or the facial expression. Worse: they're almost entirely trained on the most common emotion (on CMU-MOSEI, "Happy" is 65.9% of examples), so they effectively ignore rarer emotions like disgust or fear.

Affect-Diff solves both problems simultaneously. A NOTEARS-learned causal graph explicitly models which modality (text / audio / video) drives each prediction. A generative diffusion prior structures the latent space so the model maintains meaningful representations of minority classes, not just "default to Happy."

The Problem, Technical

Architecture Overview

Text

CMU-MOSEI transcripts → TextEncoder (transformer projection, seq × hidden)

Audio

wav2vec features → AudioEncoder (Conv1D + projection, seq × hidden)

Video

Facial action units → VideoEncoder (FC + projection, seq × hidden)

↓ LayerNorm per modality (energy-level normalization) ↓

CausalAttentionGraph

Differentiable 3×3 adjacency matrix (T↔A↔V) via Gumbel-Softmax with temperature annealing. Masks self-loops. Returns causal influence weights per modality.

↓ Causal-weighted modality fusion → VAE bottleneck ↓

VAE Latent Bottleneck

fc_mu + fc_logvar → reparameterize → z (B, T, latent_dim=256). β-KL loss for regularization. Logvar clamped [−10, 2] for stability.

↓ DDPM forward process: q(z_t | z_0) ↓

1D U-Net Diffusion (AffectiveDiffusion)

Cosine β-schedule. ResnetBlock1D with SiLU + GroupNorm. Time embeddings + label embeddings (CFG) + causal influence projection, all summed into a single conditioning vector. Classifier-Free Guidance at inference (cfg_scale > 1.0).

↓ p_sample_loop → reconstructed z → classifier ↓

Emotion Classifier + Counterfactual

Linear classifier on denoised z. Counterfactual hallucination: sample z under a different label to explain why the model changed its prediction.

Key Technical Contributions

Causal Graph (NOTEARS)

A NOTEARS-learned adjacency matrix over modalities (T↔A↔V). NOTEARS casts DAG learning as a continuous optimization problem with an acyclicity constraint, enabling gradient-based structure learning. Output: per-sample causal influence weights injected into the diffusion U-Net.

1D Diffusion U-Net (Stop-Gradient)

Custom UNet1D with ResnetBlock1D stacks operating on the temporal latent sequence. Uses a stop-gradient on the DDPM prior during classifier training, the diffusion model structures the latent space without interfering with the classification gradient. Conditions on timestep, label, and causal influence projection.

Classifier-Free Guidance

At inference, two forward passes (conditioned + null-label unconditional) are interpolated: pred_noise = uncond + scale × (cond − uncond). Enables controllable, emotion-conditioned generation for counterfactual sampling.

Counterfactual Explainability

Given a predicted emotion, sample z under a different target label. The delta between the two reconstructions reveals which features drove the original decision, providing causal attribution beyond standard attention visualization.

Training Setup

Dataset : CMU-MOSEI (3,292 aligned samples used, 6 Ekman emotion categories) Class imbalance: Happy 65.9% · three categories < 7% Backbone: RoBERTa (text) · wav2vec (audio) · facial AUs (video) Causal : NOTEARS (continuous DAG learning with acyclicity constraint) Prior : Stop-gradiented 1D DDPM (structures latent space without interfering with classification gradient) Training: PyTorch Lightning, DDP (multi-GPU, SLURM cluster) Optimizer: AdamW, gradient clipping (norm=1.0) Logging : Weights & Biases Monitor : val_balanced_accuracy (patience=50), EarlyStopping + LR warmup Combined Loss = CE (classification) + λ₁ · MSE (diffusion noise prediction, stop-gradient) + λ₂ · β-KL (VAE regularization)

Module Structure

models/ encoders/ text_encoder.py # Transformer projection for transcript features audio_encoder.py # Conv1D projection for wav2vec features video_encoder.py # FC projection for facial action units fusion/ latent_bottleneck.py # β-VAE + NOTEARS-weighted modality fusion notears.py # NOTEARS DAG learning (acyclicity-constrained) diffusion/ unet_1d.py # 1D U-Net with time/label/causal conditioning diffusion_utils.py # DDPM forward/reverse, stop-gradient prior modules/ affect_diff_module.py # PyTorch Lightning training module
Results: 0.384 validation balanced accuracy on CMU-MOSEI (3,292 aligned samples); 18% relative improvement over the strongest baseline. Critically, Affect-Diff is the only system to achieve non-trivial performance on minority emotion classes, the categories where all baselines fail entirely. That's where the NOTEARS causal graph and diffusion prior earn their keep.

Interpretability angle: The core question, which input feature causally drove this output?, is the same question mechanistic interpretability asks about language model internals. Counterfactual sampling (run the model under a different causal condition, measure the delta) is a generative analogue of causal scrubbing and activation patching. The techniques transfer directly: causal graphs over modalities → causal graphs over attention heads and MLP layers.

Published on arXiv, May 2026. arXiv:2605.08252

Tech Stack