arXiv：cs.LG· Muhammad Usama, Yeoreum Jung·· 13 小时前
基于 Dempster-Shafer 融合的证据多模态 Transformer（EMT-DS）用于情感识别
Evidential Multimodal Transformers with Dempster-Shafer Fusion for Emotion Recognition
AI 导读
研究者提出证据多模态 Transformer（EMT-DS），用 Dirichlet 分布将预测建模为主观意见，并通过 Dempster 组合规则融合各模态信念质量，把矛盾证据显式量化为冲突。
正文
Abstract:Softmax-based multimodal emotion recognition methods often assign high confidence even when sensor data are corrupted or contradictory, for example when facial expressions suggest positive valence while vocal prosody indicates negative arousal. We present an Evidential Multimodal Transformer with Dempster-Shafer Fusion (EMT-DS) that addresses this problem through formal uncertainty quantification. EMT-DS models predictions as Subjective Opinions (Dirichlet distributions), separating epistemic uncertainty (lack of evidence) from aleatoric uncertainty (sensor noise). Modality-specific transformer encoders produce belief mass distributions that are fused with Dempster's Rule of Combination, which normalizes the joint belief mass by a conflict coefficient 1/(1-c), so that contradictory evidence is explicitly quantified as conflict and the fused prediction becomes diffuse rather than being forced into an unjustified consensus. Under subject-independent and speaker-independent evaluation on four benchmarks, EMT-DS achieves 84.3% accuracy on IEMOCAP versus 81.7% for the best baseline (p < 0.01), 82.4% on DEAP under leave-one-subject-out evaluation versus 78.1%, and 96.8% on the highly imbalanced KEMDy20, and it improves accuracy on CMU-MOSEI by 2.4 points over the strongest evidential baseline. In controlled ablation, it reduces Expected Calibration Error from 0.153 (softmax transformer) to 0.042. It maintains 71.2% accuracy under severe Gaussian noise on audio features (sigma = 1.0), where attention baselines drop to 45.3%, and degrades gracefully when entire modalities are absent. The fusion step has O(MK^2) complexity, compared with O(M^2 d^2) for cross-modal attention. The learned conflict metric also flags likely errors: reviewing 20% of samples captures 80% of misclassifications, which supports selective prediction in clinical and safety-critical systems.
| Comments: | Accepted in IEEE Transactions on Affective Computing. 18 pages, 9 figures, plus 5 pages of supplementary material |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.13013 [cs.LG] |
| (or arXiv:2610.13013v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13013 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Usama Muhammad Dr. [view email]
[v1]
Fri, 9 Oct 2026 05:16:44 UTC (569 KB)
来源：arXiv：cs.LG · arxiv.org