STM Journals Peer-reviewed research from Consortium e-Learning Network

Research & Reviews: A Journal of Embedded System & Applications Review Article

Hand-Tracking-Based Mouse Control for Touchless Human-Computer Interaction: Feasibility and Future Enhancement

  1. Sameer Awasthi Computer Science Engineering in Artificial Intelligence and Machine Learning, Bansal Institute of Engineering & Technology, Lucknow
  2. Amarnath Computer Science Engineering in Artificial Intelligence and Machine Learning, Bansal Institute of Engineering & Technology, Lucknow
  3. Abhay Dixit Computer Science Engineering in Artificial Intelligence and Machine Learning, Bansal Institute of Engineering & Technology, Lucknow
  4. Arpit Singh Computer Science Engineering in Artificial Intelligence and Machine Learning, Bansal Institute of Engineering & Technology, Lucknow

Abstract

This paper presents a novel approach to human-computer interaction through a hand-tracking-based mouse control system using computer vision. The system utilizes OpenCV, MediaPipe, and the Cvzone Hand Tracking Module to identify and monitor hand movements in real-time. By utilizing hand landmarks, the system maps finger positions to cursor movement and enables essential functions such as clicking, scrolling, and double-clicking. The primary motivation for developing this system is to create a touchless alternative to traditional input devices, offering enhanced accessibility and usability. The methodology involves capturing video input via a webcam, detecting hands using MediaPipe's pre-trained models, and implementing a gesture recognition algorithm to interpret user actions. Cursor movements are mapped through an interpolation function, while clicking and scrolling gestures are detected based on finger positions and distances between key landmarks. Performance evaluation indicates an average response time of 20 to 30 ms, ensuring real-time usability. The system achieves an accuracy rate of 95% in controlled lighting conditions. However, challenges such as lighting variations and occlusion remain key areas for improvement. To mitigate unintended gestures, delays and threading mechanisms are incorporated. Future enhancements will focus on improving gesture classification with artificial intelligence models, addressing lighting-related limitations, and incorporating multi-hand interactions for increased functionality. This research contributes to the field of gesture-based computing, offering a cost-effective, hardware-independent solution for modern human-computer interaction.

Keywords

References (17)

  1. Mohamed N, Mustafa MB, Jomhari N. A review of the hand gesture recognition system: Current progress and future directions. IEEE Access. 2021;9:157422–36. doi:10.1109/ACCESS.2021. 3129650.
  2. Li J, Liu X, Wang Z, Zhang T, Qiu S, Zhao H, et al. Real-Time Hand Gesture Tracking for Human–Computer Interface Based on Multi-Sensor Data Fusion. IEEE Sensors Journal. 2021;21(23):26642-26654. doi:10.1109/jsen.2021.3122236
  3. Samkari E, Arif M, Alghamdi M, Al Ghamdi MA. Human Pose Estimation Using Deep Learning: A Systematic Literature Review. Machine Learning and Knowledge Extraction. 2023;5(4):1612-1659. doi:10.3390/make5040081
  4. Shah P, Patel M, Shah H, Patel P, Joshi S, Darji M, et al. Exploration of Historical and Modern Perspective on Hand Gesture Recognition with AI and ML. Lecture Notes in Networks and Systems. 2025:315-325. doi:10.1007/978-981-97-9523-9_27
  5. Kukreja S, Singh A, Chauhan V. Hand Gesture Vision: Integrating Computer Vision and Machine Learning for Enhanced Communication. 2024 1st International Conference on Advances in Computing, Communication and Networking (ICAC2N). 2024:35-40. doi:10.1109/icac2n63387.2024.10895070
  6. Kim J, Jang H, Kim D, Lee J. Exploration of the virtual reality teleportation methods using hand- tracking, eye-tracking, and EEG. Int J Hum Comput Interact. 2023;39:4112–25. doi:10.1080/ 10447318.2022.2109248.
  7. Lei Y, He S, Khamis M, Ye J. An End-to-End Review of Gaze Estimation and its Interactive Applications on Handheld Mobile Devices. ACM Computing Surveys. 2023;56(2):1-38. doi:10.1145/3606947
  8. Alonazi M, Ansar H, Mudawi NA, Alotaibi SS, Almujally NA, Alazeb A, et al. Smart Healthcare Hand Gesture Recognition Using CNN-Based Detector and Deep Belief Network. IEEE Access. 2023;11:84922-84933. doi:10.1109/access.2023.3289389
  9. Rehman IU, Ullah S, Ali N, Rabbi I, Khan RU. Gesture-based guidance for navigation in virtual environments. Journal on Multimodal User Interfaces. 2022;16(4):371-383. doi:10.1007/s12193-022-00395-1
  10. Ai Q, Zhao M, Chen K, Liu Q, He D, Li Z, et al. Development and application of graphene sensors in human–computer interaction: A review. IEEE Sens J. 2024;24:7406–19. doi:10.1109/JSEN. 2024.3360457.
  11. Qi J, Ma L, Cui Z, Yu Y. Computer vision-based hand gesture recognition for human-robot interaction: a review. Complex & Intelligent Systems. 2023;10(1):1581-1606. doi:10.1007/s40747-023-01173-6
  12. Ojeda-Castelo JJ, Capobianco-Uriarte MLM, Piedra-Fernandez JA, Ayala R. A survey on intelligent gesture recognition techniques. IEEE Access. 2022;10:87135–56. doi:10.1109/ACCESS.2022. 3199358.
  13. 2023 5th International Conference on Smart Systems and Inventive Technology (ICSSIT). 2023. doi:10.1109/icssit55814.2023
  14. Ruocco M, Duggan J, Jaiswal C, O'Neill B, Majeski K. Leveraging AI Face-Tracking and Gesture Recognition for Hands-Free Computing: Bridging the Gap for Users with Physical Disabilities. 2024 IEEE Global Humanitarian Technology Conference (GHTC). 2024:232-239. doi:10.1109/ghtc62424.2024.10771580
  15. Mohd Dhuzuki NH, Zainuddin AA, Kamarul Zaman NAS, Ahmad Razmi ANM, Kaitane WS, Ahmad Puzi A, et al. Design and Implementation of a Deep Learning-Based Hand Gesture Recognition System for Rehabilitation Internet-of-Things (RIoT) Environments Using MediaPipe. IIUM Engineering Journal. 2025;26(1):353-372. doi:10.31436/iiumej.v26i1.3455
  16. Dang TL, Nguyen HT, Dao DM, Nguyen HV, Luong DL, Nguyen BT, et al. SHAPE: A dataset for hand gesture recognition. Neural Comput Appl. 2022;34:21849–62. doi:10.1007/s00521-022- 07651-1.
  17. Wu D, Yang Z, Zhang P, Wang R, Yang B, Ma X. Virtual-Reality Interpromotion Technology for Metaverse: A Survey. IEEE Internet of Things Journal. 2023;10(18):15788-15809. doi:10.1109/jiot.2023.3265848
Support