STM Journals Peer-reviewed research from Consortium e-Learning Network

Journal of Advances in Shell Programming Original Research

Hand Gesture Recognition for Touchless Human–Computer Interaction

  1. Sachin Kumar Gupta Department of Computer Science & Engineering, Greater Noida College

Abstract

As technology continues to evolve, traditional input devices like the physical mouse can become restrictive, particularly for individuals with physical impairment or those working in hands-free environments. Alternative interfaces have been developed in response to the desire for more accessible and natural ways to interact. To provide a more adaptable and inclusive human–computer interaction experience, this study offers a gesture controlled virtual mouse system that integrates voice command with hand gesture recognition. Through customized hand motions recorded by a webcam, the suggested system allows users to carry out standard mouse functions, including clicking, dragging, and scrolling. Additionally, it integrates speech recognition, which enables users to utilize spoken instructions to control programs. The dynamic toggle function, which enables smooth switching between a conventional mouse and the gesture-voice interface, is a crucial component of the system. To allow users to start with familiar hardware before deciding to enable the sophisticated interface, the system starts up automatically with the virtual controllers off. The system leverages computer vision techniques and machine learning algorithms to interpret gestures and voice commands in real time, ensuring high accuracy and responsiveness. Testing revealed that the system performs reliably across different lighting conditions and user hand sizes, with minimal latency in gesture recognition and voice response. This makes it suitable for both general use and as an assistive technology for individuals with conditions such as arthritis or carpal tunnel syndrome. Overall, the proposed system offers a practical, accessible, and efficient alternative to conventional input methods.

Keywords

References (25)

  1. Babu US, Raganna A, Vidyasagar KN, Bharati SH, Kumar G. Highly accurate static hand gesture recognition model using deep convolutional neural network for human machine interaction. 2022 IEEE 4th International Conference on Advances in Electronics, Computers and Communications (ICAECC), Bengaluru, India. 2022. p.1-6. doi:10.1109/ICAECC54045.2022.9716619.
  2. Islam MZ, Hossain MS, Islam RU, Andersson K. Static hand gesture recognition using convolutional neural network with data augmentation. 8th International Conference on Informatics, Electronics & Vision (ICIEV), Washington, DC, USA. 2019. p.324-9. doi:10.1109/ICIEV.2019.8858563.
  3. Li Y, Wang T, Khan A, Li L, Li C, Yang Y, et al. Hand gesture recognition and real-time game control based on a wearable band with 6-axis sensors. 2018 International Joint Conference on Neural Networks (IJCNN), Rio de Janeiro, Brazil. 2018. p.1-6. doi:10.1109/IJCNN.2018.8489743.
  4. Shubham K, Satyam K, Kapisway P, Memani R, Anand A, Sathiya Kumar CS. IoT-based car driver assistance system using computer vision and machine learning. 2024 3rd International Conference on Artificial Intelligence for Internet of Things (AIIoT). 2024. p.1-6. doi:10.1109/AIIoT58432.2024.10574759.
  5. Panda AK, Chakravarty R, Moulik S. Hand gesture recognition using flex sensor and machine learning algorithms. 2020 IEEE-EMBS Conference on Biomedical Engineering and Sciences (IECBES), Langkawi, Malaysia. 2020. p.449-53. doi:10.1109/IECBES48179.2021.9398789.
  6. Borah AR. Detecting background dynamic scenes using naive Bayes classifier analysis compared to CNN analysis. 2023 4th International Conference on Smart Electronics and Communication (ICOSEC). 2023. p.1031-8. doi:10.1109/ICOSEC58147.2023.10276062.
  7. Sun JH, Ji TT, Zhang SB, Yang JK, Ji GR. Research on the hand gesture recognition based on deep learning. 12th International Symposium on Antennas, Propagation and EM Theory (ISAPE). 2018. p.1-4. doi:10.1109/ISAPE.2018.8634348.
  8. John K, Sreekumar A, Mathew LR. Wearable gesture detection glove for mute people. 2nd International Conference on Advances in Computing, Communication, Embedded and Secure Systems (ACCESS), India. 2021. p.97-101. doi:10.1109/ACCESS51619.2021.9563296.
  9. Roy RM, Sabu B, Nisha A, Aneesh RP. Voice controlled home automation system. 7th International Conference on Bio Signals, Images, and Instrumentation (ICBSII). 2021. p.1-6. doi:10.1109/ICBSII51839.2021.9445150.
  10. Selvaganapathy G, Thirugnanam M, Sheela A. Voice assistant system. 2022 1st International Conference on Computational Science and Technology (ICCST), Chennai, India. 2022. p.1025-30. doi:10.1109/ICCST55948.2022.10040374.
  11. Khan MR, Al Ahad A, Tania AA, Das T, Das B. Smart classroom automation: a fusion of AI with voice, gesture, and face recognition attendance system. 2024 International Conference on Advances in Computing, Communication, Electrical, and Smart Systems (iCACCESS). 2024. p.1-6. doi:10.1109/iCACCESS61735.2024.10499545.
  12. Gowthamy J, Senthilselvi A, Kumar A, Aakash S, Sreedhar G. Enhanced AI voice assistance using machine learning and NLP. 2023 3rd International Conference on Smart Technologies, Communication and Robotics (STCR). 2023. p.1-5. doi:10.1109/STCR59085.2023.10396893.
  13. Kasar M, Kavimandan P, Suryawanshi T, Abbad S. AI-based real-time hand gesture-controlled virtual mouse. Aust J Electr Electron Eng. 2024;21(3):258-67. doi:10.1080/1448837X.2024.2313818.
  14. Raja M, Nagaraj P, Sathwik P, Khan KMA, Kumar NM, Prasad US. Voice assistant and gesture controlled virtual mouse using deep learning technique. 2023 International Conference on Sustainable Computing and Data Communication Systems (ICSCDS). 2023. p.156-61. doi:10.1109/ICSCDS56580.2023.10104619.
  15. Subhash S, Srivatsa PN, Siddesh S, Ullas A, Santhosh B. Artificial intelligence-based voice assistant. 2020 4th World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4). 2020. p.593-6. doi:10.1109/WorldS450073.2020.9210344.
  16. Jain P, Bhandari LS, Sampath GM, Deeksha M. Gesture controlled virtual mouse with voice assistant integration. 2025 International Conference on Artificial Intelligence and Machine Vision (AIMV). 2025. p.1-8. doi:10.1109/AIMV66517.2025.11203344.
  17. Khan I, Kanchan V, Bharambe S. Gesture controlled virtual mouse with voice assistant. Int Res J Multidiscip Scope. 2024;5(1):26-35. doi:10.47857/irjms.2024.v05i01.0127. Note: the full article identifies five authors, adding Ayush Thada and Rohini Patil.
  18. Sindhu RJ, Satija R, Sarkar S, Malarvel M, Kathiravan M. A hybrid approach to gesture recognition with biometric security. 2022 7th International Conference on Communication and Electronics Systems (ICCES). 2022. p.697-701. doi:10.1109/ICCES54183.2022.9835887.
  19. Borah AR, Nischith TN, Gupta S. Improved learning based on GenAI. 2024 2nd International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIoT), Bengaluru, India. 2024. p.1527-32. doi:10.1109/IDCIoT59759.2024.10467943.
  20. Kassim M, Zain JM, Yaacob WF, Ruhaiyem NI, Yusoff M, Nisirat MA, editors. Soft computing in data science: 8th International Conference, SCDS 2025, Penang, Malaysia, October 8-9, 2025: revised selected papers. Singapore: Springer Nature; 2026. doi:10.1007/978-981-95-8368-3.
  21. Borah AR, Krishna BVS, H SM, Gouda DN, Poojary P. The soul safety. 3rd International Conference on Communication, Computing and Industry 4.0. 2022. p.1-6. doi:10.1109/C2I456876.2022.10051384.
  22. Bhargavi MS, Prabhakar B. Deep learning approaches for early detection of Alzheimer's disease using MRI neuroimaging. International Conference on Connected Systems & Intelligence (CSI). 2022. p.1-6. doi:10.1109/CSI54720.2022.9924058.
  23. Prathyakshini P, Prathwini. Hand gesture controlled video player application. 2023 7th International Conference on Electronics, Communication and Aerospace Technology (ICECA). 2023. p.67-72. doi:10.1109/ICECA58529.2023.10395578.
  24. Xie J, Zhang Y, Xu Z, Gao Y, Bai J, Zeng J, et al. One-handed wonders: a remote control method based on hand gesture for mobile manipulator. 2024 9th IEEE International Conference on Advanced Robotics and Mechatronics (ICARM), Tokyo, Japan. 2024. p.686-91. doi:10.1109/ICARM62033.2024.10715857.
  25. Cohen CJ, Beach G, Foulk G. A basic hand gesture control system for PC applications. Proceedings of the 30th Applied Image Pattern Recognition Workshop (AIPR 2001): Analysis and Understanding of Time Varying Imagery, Washington, DC, USA. 2001. p.74-9. doi:10.1109/AIPR.2001.991206.
Support