2025
doi: 10.12720/jait.16.8.1127-1141
Multimodal Emotion Detection and Analysis from Conversational Data
Abhinay Jatoth*
,
Faranak Abri* ,
Tien Nguyen
- Department of Computer Science, San José State University, San José, USA
- Manuscript receivedFebruary 3, 2025
- revisedMarch 5, 2025
- acceptedMay 19, 2025
- publishedAugust 18, 2025
Abstract
Emotion recognition in conversations has become increasingly relevant due to its potential applications across various fields such as customer service, social media, and mental health. In this work, we explore multimodal emotion detection using both textual and audio data. Our models leverage deep learning architectures, including Transformer-based models such as Bidirectional Encoder Representations from Transformers (BERT), Robustly optimized BERT approach (RoBERTa), Audio Spectrogram Transformer (AST), Wav2Vec2), Bidirectional Long Short-Term Memory (BiLSTM), and four fusion strategies that combine features from multiple modalities. We evaluate our approaches using two widely used emotion datasets, IEMOCAP and EMOV. Experimental results show that fusion models consistently outperform single-modality models, with Late Fusion achieving the highest weighted F1-Score of approximately 78% on IEMOCAP using both audio and text.
Keywords
- Bidirectional Encoder Representations from Transformers (BERT)
- conversational data
- emotion recognition
- fusion models
- multimodal Learning
- Wav2Vec2
How to Cite
Abhinay Jatoth, Faranak Abri, and Tien Nguyen, "Multimodal Emotion Detection and Analysis from Conversational Data," Journal of Advances in Information Technology, vol. 16, no. 8, pp. 1127-1141, 2025. https://doi.org/10.12720/jait.16.8.1127-1141
Copyright & License
Copyright © 2025 by the authors. This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).
