Research & Reviews: A Journal of Embedded System & Applications Review Article

Hand-Tracking-Based Mouse Control for Touchless Human-Computer Interaction: Feasibility and Future Enhancement

  1. Sameer Awasthi Computer Science Engineering in Artificial Intelligence and Machine Learning, Bansal Institute of Engineering & Technology, Lucknow
  2. Amarnath Computer Science Engineering in Artificial Intelligence and Machine Learning, Bansal Institute of Engineering & Technology, Lucknow
  3. Abhay Dixit Computer Science Engineering in Artificial Intelligence and Machine Learning, Bansal Institute of Engineering & Technology, Lucknow
  4. Arpit Singh Computer Science Engineering in Artificial Intelligence and Machine Learning, Bansal Institute of Engineering & Technology, Lucknow

Abstract

This paper presents a novel approach to human-computer interaction through a hand-tracking-based mouse control system using computer vision. The system utilizes OpenCV, MediaPipe, and the Cvzone Hand Tracking Module to identify and monitor hand movements in real-time. By utilizing hand landmarks, the system maps finger positions to cursor movement and enables essential functions such as clicking, scrolling, and double-clicking. The primary motivation for developing this system is to create a touchless alternative to traditional input devices, offering enhanced accessibility and usability. The methodology involves capturing video input via a webcam, detecting hands using MediaPipe's pre-trained models, and implementing a gesture recognition algorithm to interpret user actions. Cursor movements are mapped through an interpolation function, while clicking and scrolling gestures are detected based on finger positions and distances between key landmarks. Performance evaluation indicates an average response time of 20 to 30 ms, ensuring real-time usability. The system achieves an accuracy rate of 95% in controlled lighting conditions. However, challenges such as lighting variations and occlusion remain key areas for improvement. To mitigate unintended gestures, delays and threading mechanisms are incorporated. Future enhancements will focus on improving gesture classification with artificial intelligence models, addressing lighting-related limitations, and incorporating multi-hand interactions for increased functionality. This research contributes to the field of gesture-based computing, offering a cost-effective, hardware-independent solution for modern human-computer interaction.

Keywords

References (17)

  1. Mohamed N, Mustafa MB, Jomhari N. A review of the hand gesture recognition system: Current progress and future directions. IEEE Access. 2021;9:157422–36. doi:10.1109/ACCESS.2021. 3129650.
  2. Li J, Liu X, Wang Z, Zhang T, Qiu S, Zhao H, et al. Real-time hand gesture tracking for human– computer interface based on multi-sensor data fusion. IEEE Sens J. 2021;21:26642–54. doi:10.1109/JSEN.2021.3122236.
  3. Samkari E, Arif M, Alghamdi M, Al Ghamdi MA. Human pose estimation using deep learning: A systematic literature review. Mach Learn Knowl Extr. 2023;5:1612–59. doi:10.3390/make5040081.
  4. Shah P, Patel M, Shah H, Patel P, Joshi S, Darji M, et al. Exploration of historical and modern perspective on hand gesture recognition with AI and ML. In: Tuba M, Akashe S, Joshi A, editors. ICT Systems and Sustainability. ICT4SD 2024. Lecture Notes in Networks and Systems, vol 1194. Singapore: Springer; 2025. p. 315–25. doi:10.1007/978-981-97-9523-9_27.
  5. Kukreja S, Singh A, Chauhan V. Hand gesture vision: Integrating computer vision and machine learning for enhanced communication. 2024 1st International Conference on Advances in Computing, Communication and Networking (ICAC2N), Greater Noida, India. 2024. pp. 35-40. doi:10.1109/ICAC2N63387.2024.10895070.
  6. Kim J, Jang H, Kim D, Lee J. Exploration of the virtual reality teleportation methods using hand- tracking, eye-tracking, and EEG. Int J Hum Comput Interact. 2023;39:4112–25. doi:10.1080/ 10447318.2022.2109248.
  7. Lei Y, He S, Khamis M, Ye J. An end-to-end review of gaze estimation and its interactive applications on handheld mobile devices. ACM Comput Surv. 2024;56:1–38. doi:10.1145/3606947.
  8. Alonazi M, Ansar H, Al Mudawi NA, Alotaibi SS, Almujally NA, Alazeb A, et al. Smart healthcare hand gesture recognition using CNN-based detector and deep belief network. IEEE Access. 2023;11:84922–33. doi:10.1109/ACCESS.2023.3289389.
  9. Rehman IU, Ullah S, Ali N, Rabbi I, Khan RU. Gesture-based guidance for navigation in virtual environments. J Multimodal User Interfaces. 2022;16:371–83. doi:10.1007/s12193-022-00395-1.
  10. Ai Q, Zhao M, Chen K, Liu Q, He D, Li Z, et al. Development and application of graphene sensors in human–computer interaction: A review. IEEE Sens J. 2024;24:7406–19. doi:10.1109/JSEN. 2024.3360457.
  11. Qi J, Ma L, Cui Z, Yu Y. Computer vision-based hand gesture recognition for human–robot interaction: A review. Complex Intell Syst. 2024;10:1581–606. doi:10.1007/s40747-023-01173-6.
  12. Ojeda-Castelo JJ, Capobianco-Uriarte MLM, Piedra-Fernandez JA, Ayala R. A survey on intelligent gesture recognition techniques. IEEE Access. 2022;10:87135–56. doi:10.1109/ACCESS.2022. 3199358.
  13. Wisener WJ, Rodriguez JD, Ovando A, Woolford C, Patel K. A top-view hand gesture recognition system for IoT applications. 2023 5th International Conference on Smart Systems and Inventive Technology (ICSSIT), Tirunelveli, India. 2023, pp. 430–4. doi:10.1109/ICSSIT55814.2023. 10060969.
  14. Ruocco M, Duggan J, Jaiswal C, O’Neill B, Majeski K. Leveraging AI face-tracking and gesture recognition for hands-free computing: Bridging the gap for users with physical disabilities. 2024 IEEE Global Humanitarian Technology Conference (GHTC), Radnor, PA, USA. 2024. pp. 232–9. doi:10.1109/GHTC62424.2024.10771580.
  15. Mohd Dhuzuki NH, Zainuddin AA, Kamarul Zaman NAS, Ahmad Razmi ANM, Kaitane WS, Ahmad Puzi A, et al. Design and implementation of a deep learning-based hand gesture recognition system for rehabilitation Internet-of-things (RIoT) environments using MediaPipe. IIUM Eng J. 2025;26:353–72. doi:10.31436/iiumej.v26i1.3455.
  16. Dang TL, Nguyen HT, Dao DM, Nguyen HV, Luong DL, Nguyen BT, et al. SHAPE: A dataset for hand gesture recognition. Neural Comput Appl. 2022;34:21849–62. doi:10.1007/s00521-022- 07651-1.
  17. Wu D, Yang Z, Zhang P, Wang R, Yang B, Ma X. Virtual-reality interpromotion technology for metaverse: A survey. IEEE Internet Things J. 2023;10:15788–809. doi:10.1109/JIOT.2023.3265848

Read more of it

Subscribe to one journal or build your own list of several. One invoice, one payment, access for a full year.

Subscribe to journals →

Publish your article

Peer-reviewed journals across every subject, with open access options. Send your manuscript to the right editor today.

Publish with us → Calculate your APC
Support