Integrating AI and IoT in STEM Education Through a Gesture-Control Project

Petrovych, Serhii; Zhou, Stefan-Daniel

doi:Under indexing

Proceedings of International Conference on Applied Innovation in IT · 2026/04/22 · Vol. 14 · Issue 2 · pp. 399–405

Integrating AI and IoT in STEM Education Through a Gesture-Control Project

Serhii Petrovych, Stefan-Daniel Horvath and Chunfang Zhou

📄 Download PDF DOI: Under indexing

Abstract

This paper presents the development and deployment of a hand gesture-controlled lighting system specifically designed for STEM education at the undergraduate level. The primary objective of this study is to demonstrate a practical framework for integrating complex AI and IoT concepts through a hands-on, constructionist learning approach. The project combines affordable microcontroller hardware, specifically the ESP32-CAM and ESP32 Dev Board, with the MediaPipe framework and Internet of Things (IoT) protocols to transform human hand movements into interactive visual effects. By utilizing MediaPipe for real-time hand landmark detection and implementing explicit geometric rules for gesture classification, the project provides students with direct experience in hardware-software integration and distributed system control. The study details the technical architecture, including robust debouncing mechanisms to ensure operational stability. Furthermore, the system supports versatile deployment options, such as standalone executable files for Windows, enhancing classroom accessibility. This "white-box" design principle facilitates a deeper understanding of embedded programming and practical AI applications. Ultimately, the project serves as a comprehensive educational tool that successfully bridges theoretical knowledge and applied STEM skills in modern engineering curricula.

Keywords

STEM Education Hand Gesture Control Computer Vision Internet of Things.

References

S. Papert, Mindstorms: Children, Computers, and Powerful Ideas, Basic Books, 1980.
J. M. Wing, “Computational thinking,” Communications of the ACM, vol. 49, no. 3, pp. 33-35, 2006, [Online]. Available: https://doi.org/10.1145/1118178.1118215.
P. C. Blumenfeld, E. Soloway, R. W. Marx, J. S. Krajcik, M. Guzdial, and A. Palincsar, “Motivating project-based learning: Sustaining the doing, supporting the learning,” Educational Psychologist, vol. 26, no. 3-4, pp. 369-398, 1991, [Online]. Available: https://doi.org/10.1080/00461520.1991.9653139.
C. E. Hmelo-Silver, “Problem-based learning: What and how do students learn?” Educational Psychology Review, vol. 16, pp. 235-266, 2004, [Online]. Available: https://doi.org/10.1023/B:EDPR.0000034022.16470.f3.
M. Umadevi, P. Nikitha, T. S. Goud, M. Abhinay, and L. Swathi, “Gesture Math AI: Real-time math problem solving using hand gestures and computer vision,” International Research Journal on Advanced Engineering Hub, vol. 3, no. 5, 2025, [Online]. Available: https://doi.org/10.47392/IRJAEH.2025.0320.
I. D. S. Chen, C.-M. Yang, S.-S. Wu, C.-K. Yang, M.-J. Chen, C.-H. Yeh, and Y.-H. Lin, “Continuous recognition of teachers’ hand signals for students with attention deficits,” Algorithms, vol. 17, no. 7, p. 300, 2024, [Online]. Available: https://doi.org/10.3390/a17070300.
A. C. D. O. Bastos, M. F. Pinto, R. C. Coutinho, F. L. Silva, A. A. D. Lima, and G. M. Araujo, “A low-cost social robot for gesture-based educational activities and human-robot interaction in learning environments,” in Proc. XVII Simpósio Brasileiro de Robótica e Simpósio Latino Americano de Robótica (SBR/LARS), pp. 408-413, 2025, [Online]. Available: https://doi.org/10.1109/sbr/wre66973.2025.11249546.
W. Malik, G. Kulis, and K. Skabek, “Recognition of Polish sign gesture language in extended reality,” in Proc. European Conference on Modelling and Simulation (ECMS), pp. 645-652, 2025, [Online]. Available: https://doi.org/10.7148/2025-0645.
M. Arboleda, C. Vieira, and J. L. Chiu, “Opening the machine learning black box for multidisciplinary students: Scaffolding from GUI to coding,” in Proc. IEEE Frontiers in Education Conference (FIE), pp. 1-5, 2023, [Online]. Available: https://doi.org/10.1109/FIE58773.2023.10343043.
D. Wang and G. Chen, “Making AI accessible for STEM teachers: Using explainable AI for unpacking classroom discourse analysis,” IEEE Transactions on Education, vol. 67, no. 6, pp. 907-918, 2024, [Online]. Available: https://doi.org/10.1109/TE.2024.3421606.
J. M. Wing, “Computational thinking and thinking about computing,” Philosophical Transactions of the Royal Society A, vol. 366, no. 1881, pp. 3717-3725, 2008, [Online]. Available: https://doi.org/10.1098/rsta.2008.0118.
P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” in Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. I-511, 2005, [Online]. Available: https://doi.org/10.1109/cvpr.2001.990517.
C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays et al., “MediaPipe: A framework for building perception pipelines,” arXiv preprint arXiv:1906.08172, 2019, [Online]. Available: https://doi.org/10.48550/arxiv.1906.08172.
D. Long and B. Magerko, “What is AI literacy? Competencies and design considerations,” in Proc. CHI Conference on Human Factors in Computing Systems, pp. 1-16, 2020, [Online]. Available: https://doi.org/10.1145/3313831.3376727.