STM Journals Peer-reviewed research from Consortium e-Learning Network

Journal of Advancements in Robotics Original Research

Hands-Free Human–Computer Interaction for Smarter Workspaces

  1. Prerana Balip Department of Information Technology, A.C. Patil College of Engineering
  2. Dnyaneshwar Sonavane Department of Information Technology, A.C. Patil College of Engineering
  3. Shaila Pawar Department of Information Technology, A.C. Patil College of Engineering
  4. Omkar Shinde Department of Information Technology, A.C. Patil College of Engineering
  5. Ayush Deoghare Department of Information Technology, A.C. Patil College of Engineering
  6. Poonam Potraje Department of Information Technology, A.C. Patil College of Engineering

Abstract

In current digital work environments, frequent interaction with keyboards and mouse devices can interrupt workflow, especially during presentations, collaborative sessions, and multitasking activities. This study introduces a touchless human–computer interaction system that allows users to operate computer functions through hand gestures and voice instructions, minimizing the need for traditional input devices such as keyboards and mice. The proposed system integrates gesture recognition, speech processing, and optical character recognition (OCR) technologies to support a variety of interactive tasks such as slide navigation, virtual drawing, handwriting capture, and text extraction from visual input. A voice-based assistant is incorporated to execute system-level operations and respond to user commands efficiently, while a vision-based module performs real-time hand tracking and gesture detection for seamless interaction. The system is designed to improve usability, accessibility, and interaction efficiency in environments where touch-free operation is beneficial, including smart classrooms, meetings, and presentation settings. By combining computer vision and voice interaction techniques, the proposed model offers a more intuitive and flexible method of human–computer communication. Experimental results indicate that the system operates efficiently in controlled environments and can carry out various functions with high accuracy while requiring very little physical effort from the user. The project also provides scope for future enhancement in dynamic environments through improved gesture robustness, adaptive learning, and advanced AI-based interaction capabilities.

Keywords

References (11)

  1. Sharma P, Sharma N. Gesture recognition system. In: 2019 4th International Conference on Internet of Things: Smart Innovation and Usages (IoT-SIU); 2019; Ghaziabad, India. p. 1-3. doi:10.1109/IoT-SIU.2019.8777487.
  2. Dias PM, Jayakody K. Virtual assistant in native language. In: 2020 IEEE Asia-Pacific Conference on Geoscience, Electronics and Remote Sensing Technology (AGERS); 2020. p. 16-18. doi:10.1109/AGERS51788.2020.9452751.
  3. Pustode B, Pawar V, Pawar V, Pawar T, Pokale S. Smart presentation system using hand gestures [Preprint]. Research Square. 2023. doi:10.21203/rs.3.rs-2549833/v1.
  4. Umapathi N, Karthick G, Venkateswaran N, Jegadeesan R, Srinivas D. Desktop's virtual assistant using Python. Eur Chem Bull. 2023;12(S3):5975-5984. doi:10.31838/ecb/2023.12.s3.667.
  5. Soroni F, Sajid SA, Bhuiyan MNH, Iqbal J, Khan MM. Hand gesture based virtual blackboard using webcam. In: 2021 IEEE 12th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON); 2021. p. 134-140. doi:10.1109/IEMCON53756.2021.9623181.
  6. Singh S, Shukla H, Sharma K, Tyagi H, Prasad J. Digitized interaction: a gesture-controlled whiteboard system with OpenCV, MediaPipe and NumPy. IPEC J Sci Technol. 2023;2(1):43-49.
  7. Idrees M, Ahmad A, Butt MA, Danish HM. Controlling PowerPoint using hand gestures in Python. Webology. 2021;18(6):1372-1388.
  8. Powar S, Kadam S, Malage S, Shingane P. Automated digital presentation control using hand gesture technique. ITM Web Conf. 2022;44:03031. doi:10.1051/itmconf/20224403031.
  9. Abid Siddique RV, Naik S. A survey on gesture control techniques for smart object interaction in disability support. Electronics. 2023;12:512.
  10. Mohamed N, Mustafa MB, Jomhari N. A review of the hand gesture recognition system: current progress and future directions. IEEE Access. 2021;9:157422-157436. doi:10.1109/ACCESS.2021.3129650.
  11. Prasetya DD, Ashar M. Design of interactive whiteboard to support e-learning. In: Proceedings of the 1st International Conference on Vocational Education and Training (ICOVET 2017); 2017 Nov 4-5; Malang, Indonesia. Paris: Atlantis Press; 2017. p. 121-124. doi:10.2991/icovet-17.2017.26.
Support