International Journal of Brain Sciences Original Research
Multimodal Explainable AI for Mental Health Assessment: Integrating EEG Signals, Social Media Behaviour, and Clinical NLP for Transparent Psychiatric Decision Support
This is an unedited manuscript accepted for publication and provided as an Article in Press for early access at the author’s request. The article will undergo copyediting, typesetting, and galley proof review before final publication. Please be aware that errors may be identified during production that could affect the content. All legal disclaimers of the journal apply.
Abstract
Mental health disorders — encompassing major depressive disorder, generalized anxiety disorder, bipolar spectrum conditions, and post-traumatic stress disorder — collectively affect over one billion individuals globally, yet fewer than 30% receive timely, evidence-based clinical assessment. Artificial intelligence has demonstrated considerable promise in augmenting psychiatric evaluation, but existing single-modality approaches suffer from two critical limitations: they fail to capture the heterogeneous, multimodal nature of psychological states that manifest across neurophysiological, linguistic, and behavioral dimensions simultaneously, and they rely on opaque deep learning architectures whose predictions clinicians cannot audit, contest, or integrate into therapeutic reasoning. This paper presents MentalXAI, a unified multimodal explainable AI framework for psychiatric decision support that fuses three complementary data modalities — EEG neurophysiological signals, longitudinal social media behavioral sequences, and clinical interview transcripts — through a novel Cross-Modal Attention Fusion network (CMAF) producing transparent, clinician-interpreTable assessments across four mental health conditions: major depressive disorder (MDD), generalized anxiety disorder (GAD), bipolar disorder (BD), and post-traumatic stress disorder (PTSD). MentalXAI generates three tiers of clinical explanation: SHAP-attributed feature importance rankings identifying modality-specific biomarkers driving each prediction; attention heat-map visualizations localizing discriminative EEG frequency bands and social media temporal windows; and natural language clinical rationale narratives generated through an evidence-grounded template system. Evaluated on three benchmark datasets — MODMA (EEG, 53 subjects), CLPsych 2019 (social media, 1,276 posts), and DAIC-WOZ (clinical interviews, 189 subjects) — MentalXAI achieves classification F1-scores of 94.7% (MDD), 91.3% (GAD), 88.9% (BD), and 90.4% (PTSD), outperforming the strongest unimodal baselines by 8.3–14.7 percentage points. A 42- psychiatrist clinical validation study confirms that MentalXAI explanations improve diagnostic confidence by 31.4% and reduce assessment time by 22.7% relative to AI-output-only conditions, establishing the framework's translational value for clinical deployment.Keywords
References (1)
- Mumtaz W, Xia L, Mohd Yasin MA, Azhar Ali SS, Malik AS. A wavelet-based technique to predict treatment outcome for major depressive disorder. PloS one. 2017 Feb 2;12(2):e0171409. [2] Lawhern VJ, Solon AJ, Waytowich NR, Gordon SM, Hung CP, Lance BJ. EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces. Journal of neural engineering. 2018 Oct 1;15(5):056013. [3] Zhang Z, Liu Y, Zhong SH. GANSER: A self-supervised data augmentation framework for EEG-based emotion recognition. IEEE Transactions on Affective Computing. 2022 Apr 26;14(3):2048-63. [4] Liu W, Qiu JL, Zheng WL, Lu BL. Comparing recognition performance and robustness of multimodal deep learning models for multimodal emotion recognition. IEEE Transactions on Cognitive and Developmental Systems. 2021 Apr 5(2):715-29. [5] Song T, Zheng W, Song P, Cui Z. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Transactions on Affective Computing. 2018 Mar 21;11(3):532-41. [6] Coppersmith G, Dredze M, Harman C, Hollingshead K, Mitchell M. CLPsych 2015 shared task: Depression and PTSD on Twitter. InProceedings of the 2nd workshop on computational linguistics and clinical psychology: from linguistic signal to clinical reality 2015 (pp. 31-39). [7] Ji S, Zhang T, Ansari L, Fu J, Tiwari P, Cambria E. Mentalbert: Publicly available pretrained language models for mental healthcare. Inproceedings of the thirteenth language resources and evaluation conference 2022 Jun (pp. 7184-7190). [8] Coppersmith G, Leary R, Crutchley P, Fine A. Natural language processing of social media as screening for suicide risk. Biomedical informatics insights. 2018 Aug;10:1178222618792860. [9] Gratch J, Artstein R, Lucas GM, Stratou G, Scherer S, Nazarian A, Wood R, Boberg J, DeVault D, Marsella S, Traum DR. The distress analysis interview corpus of human and computer interviews. InLrec 2014 May 26 (Vol. 14, pp. 3123-3128). [10] Al Hanai T, Ghassemi MM, Glass JR. Detecting depression with audio/text sequence modeling of interviews. InInterspeech 2018 Sep 2 (pp. 1716-1720). [11] Alsentzer E, Murphy J, Boag W, Weng WH, Jindi D, Naumann T, McDermott M. Publicly available clinical BERT embeddings. InProceedings of the 2nd clinical natural language processing workshop 2019 Jun (pp. 72-78). [12] Vajre V, Naylor M, Kamath U, Shehu A. PsychBERT: a mental health language model for social media mental health behavioral analysis. In2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) 2021 Dec 9 (pp. 1077-1082). IEEE. [13] Lundberg S. A unified approach to interpreting model predictions. arXiv preprint arXiv:1705.07874. 2017. [14] Ribeiro MT, Singh S, Guestrin C. " Why should i trust you?" Explaining the predictions of any classifier. InProceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining 2016 Aug 13 (pp. 1135-1144). [15] Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-cam: Visual explanations from deep networks via gradient-based localization. InProceedings of the IEEE international conference on computer vision 2017 (pp. 618-626). [16] Jacobson NC, Weingarden H, Wilhelm S. Digital biomarkers of mood disorders and symptom change. NPJ digital medicine. 2019 Feb 1;2(1):3. [17] Cai H, Yuan Z, Gao Y, Sun S, Li N, Tian F, Xiao H, Li J, Yang Z, Li X, Zhao Q. A multi-modal open dataset for mental-disorder analysis. Scientific data. 2022 Apr 19;9(1):178. [18] Liu Z, Shen Y, Lakshminarasimhan VB, Liang PP, Zadeh AB, Morency LP. Efficient low-rank multimodal fusion with modality-specific factors. InProceedings of the 56th annual meeting of the association for computational linguistics (volume 1: Long papers) 2018 Jul (pp. 2247-2256). [19] World Health Organization. World mental health report: transforming mental health for all. Executive summary. World Health Organization; 2022 Jun 16.