STM Journals Peer-reviewed research from Consortium e-Learning Network

Journal of Image Processing & Pattern Recognition Progress Review Article

An Analysis of Multimodal Fusion in Deepfake Detection for Video Samples

  1. Devang Mulye Department of Information Technology, SIES Graduate School of Technology, Navi Mumbai
  2. Parakh Pawar Department of Information Technology, SIES Graduate School of Technology, Navi Mumbai
  3. Aditya Yadav Department of Information Technology, SIES Graduate School of Technology, Navi Mumbai
  4. S. Poornima Department of Information Technology, SIES Graduate School of Technology, Navi Mumbai

Abstract

In today’s rapidly evolving digital landscape, deepfake technology stands as both a marvel and a threat to privacy and security. Deepfakes, hyper-realistic synthetic media created using artificial intelligence (AI), can deceive and manipulate on an unprecedented scale, from political propaganda to compromising videos of public figures. This research navigates deepfake detection, focusing on two advanced methodologies: the vision transformers (ViT) image classifier and the Meso4 method. The ViT model utilizes convolutional neural networks (CNNs) for image analysis, while the Meso4 method examines images at a mesoscopic level. These methodologies are evaluated for their effectiveness in differentiating genuine content from altered media. Motivated by the threats posed by deepfakes to individuals and society, this project evaluates the ViT model using the Open Forensics dataset and the Meso4 model using a combination of self-generated content and the Face2Face dataset. The ViT model achieved 99.9% accuracy within its dataset and 85% accuracy for subtly deceptive images. Conversely, the Meso4 model showed a 60% accuracy and biases, particularly towards female subjects, likely due to its training on adult content. To address privacy and security concerns, five additional features are proposed: eye movement error detection, lip sync inconsistency, facial expression analysis, facial texture inconsistency, and background inconsistency. These features enhance detection accuracy and reliability and are integrated into a customized model, augmenting the ViT model’s strengths and addressing its limitations. The proposed model improves accuracy and computation speed, making it practical for real-world applications. This research highlights the urgency of advancing robust and unbiased deepfake detection methods to safeguard privacy and security in an era where truth can be easily manipulated.

Keywords

References (28)

  1. Taeb M, Chi H. Comparison of Deepfake Detection Techniques through Deep Learning. Journal of Cybersecurity and Privacy. 2022;2(1):89-106. doi:10.3390/jcp2010007
  2. Ramachandran S, Nadimpalli AV, Rattani A. An Experimental Evaluation on Deepfake Detection using Deep Face Recognition. 2021 International Carnahan Conference on Security Technology (ICCST). 2021:1-6. doi:10.1109/iccst49569.2021.9717407
  3. Lyu S. Deepfake detection: Current challenges and next steps. IEEE Int Conf Multimed Expo Workshops (ICMEW). IEEE Publications; 2020. p. 1–6. doi:10.1109/ICMEW46912.2020.9105
  4. Khichi M. Deepfake or real image prediction using Mesonet [dissertation]. Delhi Technological University. 2021. Available from: http://dspace.dtu.ac.in:8080/jspui/handle/repository/18952.
  5. Liang J, Wang D, Ling X. Image Classification for Soybean and Weeds Based on ViT. Journal of Physics: Conference Series. 2021;2002(1):012068. doi:10.1088/1742-6596/2002/1/012068
  6. Wodajo D, Atnafu S. Deepfake video detection using convolutional vision transformer. [Preprint]. ArXiv:2102.11126. 2021.
  7. Pan D, Sun L, Wang R, Zhang X, Sinnott RO. Deepfake Detection through Deep Learning. 2020 IEEE/ACM International Conference on Big Data Computing, Applications and Technologies (BDCAT). 2020:134-143. doi:10.1109/bdcat50828.2020.00001
  8. Aghasanli A, Kangin D, Angelov P. Interpretable-through-prototypes deepfake detection for diffusion models. 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). 2023:467-474. doi:10.1109/iccvw60793.2023.00053
  9. Turan SG. Deepfake and Digital Citizenship. Advances in Educational Technologies and Instructional Design. 2021:124-142. doi:10.4018/978-1-7998-6474-5.ch006
  10. Poornima S, Subramanian S. Unconstrained iris authentication through fusion of RGB channel information. Int J Pattern Recognit Artif Intell. 2014;28:1456010. doi:10.1142/S0218001414
  11. Poornima S, Subramanian S. An efficient feature level fusion for a multimodal biometric system using correlation filter. Int J Appl Eng Res. 2014;9:8975–8.
  12. Poornima S, Thaya IM. Comparison analysis of feature level fusion in multimodal biometrics system. Int J Appl Eng Res. 2014;9:8908-8912.
  13. Poornima, S., Fathima Nadheen, M. Feature level fusion in multimodal biometric authentication system. Int J Comput Appl. 2013;69(18):36-40.
  14. Poornima S. Performance study of fusion in multimodal biometric verification using ear and iris features. In: Proceedings of the International Conference on Research Trends in Computer Technologies (ICRTCT); Feb 2013. ICRTCT, 4:133-136.
  15. Poornima S, Rajavelu C, Subramanian S. Comparison and a neural network approach for iris localization. Procedia Computer Science. 2010;2:127-132. doi:10.1016/j.procs.2010.11.016
  16. George AS, George AH. Deepfakes: The evolution of hyper-realistic media manipulation. Partners Univ Innov Res. 2023;1:58–74.
  17. Westerlund M. The Emergence of Deepfake Technology: A Review. Technology Innovation Management Review. 2019;9(11):39-52. doi:10.22215/timreview/1282
  18. Shahzad HF, Rustam F, Flores ES, Luís Vidal Mazón J, de la Torre Diez I, Ashraf I. A Review of Image Processing Techniques for Deepfakes. Sensors. 2022;22(12):4556. doi:10.3390/s22124556
  19. Wang Z, Guo Y, Zuo W. Deepfake Forensics via an Adversarial Game. IEEE Transactions on Image Processing. 2022;31:3541-3552. doi:10.1109/tip.2022.3172845
  20. Silva SH, Bethany M, Votto AM, Scarff IH, Beebe N, Najafirad P. Deepfake forensics analysis: An explainable hierarchical ensemble of weakly supervised models. Forensic Science International: Synergy. 2022;4:100217. doi:10.1016/j.fsisyn.2022.100217
  21. Pashine S, Mandiya S, Gupta P, Sheikh R. Deep fake detection: Survey of facial manipulation detection solutions. [Preprint] ArXiv:2106.12605. 2021.
  22. Joseph Z, Nyirenda C. Deepfake detection using a two-stream capsule network. 2021 IST-Africa Conference (IST-Africa), South Africa, South Africa, 2021, pp. 1–8.
  23. Heo YJ, Yeo WH, Kim BG. DeepFake detection algorithm based on improved vision transformer. Applied Intelligence. 2022;53(7):7512-7527. doi:10.1007/s10489-022-03867-9
  24. Afchar D, Nozick V, Yamagishi J, Echizen I. MesoNet: a Compact Facial Video Forgery Detection Network. 2018 IEEE International Workshop on Information Forensics and Security (WIFS). 2018:1-7. doi:10.1109/wifs.2018.8630761
  25. Boesch G. (2023). Vision Transformers (ViT) in Image Recognition - 2024 Guide. [online] viso.ai. Available from: https://viso.ai/deep-learning/vision-transformer-vit/
  26. Ahmed OB, Asghar AH, Bahwerth FS, Assaggaf HM, Bamaga MA. RETRACTED ARTICLE: The prevalence of aminoglycoside-resistant genes in Gram-negative bacteria in tertiary hospitals. Applied Nanoscience. 2021;13(2):1093-1093. doi:10.1007/s13204-021-01887-4
  27. Das S, Seferbekov S, Datta A, Islam MS, Amin MR. Towards Solving the DeepFake Problem : An Analysis on Improving DeepFake Detection using Dynamic Face Augmentation. 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). 2021:3769-3778. doi:10.1109/iccvw54120.2021.00421
  28. Coccomini DA, Messina N, Gennaro C, Falchi F. Combining EfficientNet and Vision Transformers for Video Deepfake Detection. Lecture Notes in Computer Science. 2022:219-229. doi:10.1007/978-3-031-06433-3_19
Support