Affect-Diff · Causal-Diffusion Bridge: Ankit Sanjyal

Multimodal Emotion Recognition via Causal-Diffusion Bridge (Affect-Diff)

Ankit Sanjyal

arXiv preprint arXiv:2605.08252 · cs.LG, cs.AI · May 2026

Causal Interpretability NOTEARS Diffusion Models Multimodal Learning Class Imbalance CMU-MOSEI

Key result: 0.384 validation balanced accuracy on CMU-MOSEI · 18% relative improvement over the strongest baseline · The only system to achieve non-trivial performance on minority emotion classes where all baselines fail entirely.

Abstract. We address extreme class imbalance in emotion recognition on CMU-MOSEI, where Happy represents 65.9% of samples while three Ekman categories comprise under 7%. The Affect-Diff framework combines three mechanisms: a NOTEARS-learned causal graph for modality re-weighting, a β-VAE bottleneck for latent compression, and a stop-gradiented 1D DDPM prior for latent space structuring. Testing on 3,292 aligned samples achieved 0.384 validation balanced accuracy, representing an 18% relative improvement over the strongest baseline, while baselines failed entirely on minority emotions.

What This Paper Does, in Plain English

Most AI emotion recognition models take a shortcut: they learn that "Happy" is the right answer most of the time (it's 65.9% of the training data) and rarely predict anything else. They also can't tell you why they made a prediction, did the person's words matter most? Their tone of voice? Their facial expression?

Affect-Diff fixes both problems. It explicitly models which communication channel caused the emotion prediction (using a learned causal graph), and it uses a generative prior to keep the latent space meaningful even for rare emotion categories. The result is a model that actually works on minority emotions, not just the majority class.

The Problem in Detail

Class imbalance on CMU-MOSEI: Happy = 65.9% of samples. Three Ekman categories (disgust, fear, surprise) each under 7%. Standard classifiers optimize overall accuracy and effectively ignore minority classes, balanced accuracy exposes this failure.

Standard multimodal fusion models (concatenation, attention-based fusion) have two gaps:

No causal attribution: They fuse text, audio, and video into a joint representation, but there's no mechanism to identify which modality was the causal driver of any particular prediction.
Discriminative collapse: Trained with cross-entropy on imbalanced data, they collapse to predicting the majority class. No structural incentive to represent minority classes in latent space.

Method: The Causal-Diffusion Bridge

NOTEARS Causal Graph

Learns a directed acyclic graph (DAG) over modalities (Text ↔ Audio ↔ Video) by casting structure learning as a continuous optimization problem with an acyclicity constraint. Output: per-sample causal influence weights for each modality. This is what makes attribution possible.

β-VAE Bottleneck

Compresses the causal-weighted modality fusion to a shared latent space z. The β-KL term regularizes the latent space toward a prior, preventing collapse. Logvar clamped for numerical stability.

Stop-Gradiented DDPM Prior

A 1D denoising diffusion model structures the latent space so minority emotion classes have meaningful latent regions. Critically, the diffusion prior is stop-gradiented during classifier training, it shapes the space without interfering with the classification objective.

Counterfactual Sampling

At inference, sample z under a different emotion label and compute the delta. This is the explainability mechanism: the change in reconstruction reveals what features drove the original prediction, causal attribution, not just correlation.

Combined Training Objective

Loss = CE(classification)
     + λ₁ · MSE(DDPM noise prediction, stop-gradient)
     + λ₂ · β-KL(VAE regularization)

Results

System	Balanced Accuracy ↑	Minority Emotions
Standard fusion baselines	~0.33 (best)	Fail entirely (< chance)
Affect-Diff (ours)	0.384	Non-trivial performance

Evaluated on CMU-MOSEI, 3,292 aligned samples, 6 Ekman emotion categories. Balanced accuracy weights all classes equally, exposing majority-class collapse.

Interpretability Connections

This paper demonstrates causal interpretability in a multimodal model, one of the earliest instances I know of in the emotion recognition literature. The core idea maps directly to mechanistic interpretability in language models:

NOTEARS causal graph over modalities → causal graph over attention heads and MLP layers in a transformer. Same question: which component causally drove this output?
Counterfactual sampling (run model under different causal condition, measure delta) → causal scrubbing and activation patching. Same operation, different substrate.
Stop-gradiented prior → related to the practice of freezing interpretability components during fine-tuning to preserve structure.

Citation

@article{sanjyal2026affectdiff,
  title   = {Multimodal Emotion Recognition via Causal-Diffusion Bridge (Affect-Diff)},
  author  = {Sanjyal, Ankit},
  journal = {arXiv preprint arXiv:2605.08252},
  year    = {2026}
}