Development of a Pattern Recognition System for Interpreting Sign Language Using Convolutional Neural Network
DOI:
https://doi.org/10.4314/njtd.v22i1.3365Keywords:
Pattern Recognition, Deep Learning, Convolutional Neural Network, Computer Vision, Sign LanguageAbstract
Sign language is essential for communication among the deaf, hearing-impaired, and the general public. However, many people with normal hearing struggle to understand sign language, resulting in ineffective interactions with the deaf community in their daily activities. Recent studies have employed machine learning algorithms and computer application software solely to convert sign languages into text, aiming to address these challenges and bridge the communication gap among the deaf, hearing-impaired, and the public. This study developed a device utilizing a convolutional neural network (CNN) and an embedded system to translate sign languages into text and audio, enabling daily interactions among the deaf, hearing-impaired, and others in both private and public settings. The system comprises hardware components, including an OV2640 camera sensor to capture images of the signers, an ESP32 microcontroller to process the captured images and convert them into text and speech after being trained with a CNN model, a display screen to read the interpreted images as text, and a speaker that outputs the interpreted images as speech. To validate the proposed framework, 34,628 images were collected from a publicly available American Sign Language dataset. In this dataset, 80% of the images were used as training data, and 20% were reserved for testing. The results show a training accuracy of 99.76% but a low validation accuracy. The system's testing accuracy could be reliably improved with good lighting, a plain background, a microcontroller that has increased memory size, and a higher resolution camera, considering the achieved training accuracy.
References
Amosa, T. I., bt Izhar, L. I., Sebastian, P., Ismail, I. B., Ibrahim, O., & Ayinla, S. L. (2023). Clinical Errors from Acronym Use in Electronic Health Record: A Review of NLP-based Disambiguation Techniques. IEEE Access, 11, 59297-59316.
Amosa, T. I., Sebastian, P., Izhar, L. I., Ibrahim, O., Ayinla, L. S., Bahashwan, A. A., Bala, A., & Samaila, Y. A. (2023). Multi-camera multi-object tracking: a review of current trends and future advances. Neurocomputing, 552, 126558.
Aromoye, I. A., Lo, H. H., Sebastian, P., Abro, G. E. M., & Ayinla, S. L. (2025). Significant Advancements in UAV Technology for Reliable Oil and Gas Pipeline Monitoring. Computer Modeling in Engineering & Sciences (CMES), 142(2), 1155-1197.
Halder, A., & Tayade, A. (2021). Real-time vernacular sign language recognition using mediapipe and machine learning. International Journal of Research Publication and Reviews, 2(5), 9-17.
Tecperson(2018). Sign Language Modified National Institute of Standard and Technology (MNIST) [Data set]. retrieved from https://www.kaggle.com/datasets/datamunge/sign-language-mnist
Kanade, V. (2023). What Is Pattern Recognition? Working, Types, and Applications. Retrieved from https://www.spiceworks.com/tech/artificial-intelligence/articles/what-is-pattern-recognition/
Lawal, A., Cavus, N., Lawan, A. A., & Sani, I. (2024). Hausar kurma: Development and evaluation of interactive mobile app for the English-Hausa sign language alphabet. IEEE Access, 12, 46012-46023.
LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444.
Liu, Y., Jiang, X., Yu, X., Ye, H., Ma, C., Wang, W., & Hu, Y. (2023). A wearable system for sign language recognition enabled by a convolutional neural network. Nano Energy, 116, 108767.
Oguntimilehin, A., & Balogun, K. (2024). Real-time sign language fingerspelling recognition using convolutional neural network. Int. Arab J. Inf. Technol., 21(1), 158-165.
Sharma, S., & Singh, S. (2021). Vision-based hand gesture recognition using deep learning for the interpretation of sign language. Expert Systems with Applications, 182, 115657.
Smits, T., & Wevers, M. (2022). The agency of computer vision models as optical instruments. Visual Communication, 21(2), 329-349.
Vazquez Lopez, I. (2017). Hand Gesture Recognition for Sign Language Transcription [Unpublished Master's thesis], Boise State University.
Wangchuk, K., Riyamongkol, P., & Waranusast, R. (2021). Real-time Bhutanese sign language digits recognition system using convolutional neural network. ICT Express, 7(2), 215-220.
World Health Organization. (2025). Deafness and hearing loss, Key facts. Retrieved from https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Nigerian Journal of Technological Development
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
In accordance with the Copyright Act of 1976, which became effective January 1, 1978, the following statement signed by each author must accompany the manuscript submitted: "I, the undersigned author, transfer all copyright ownership of the manuscript referenced above to the Nigerian Journal of Technological Development, in the event the work is published. I warrant that the article is original, does not infringe upon any copyright or other proprietary right of any third party, is not under consideration by another journal, and has not been published previously. I have reviewed and approve the submitted version of the manuscript and agree to its publication in the Nigerian Journal of Technological Development." A copyright transfer form should be downloaded from the NJTD Website ( http://njtd.com.ng/index.php/njtd). Author(s) will be consulted, whenever possible, regarding republication of material. All authors must have access to the data presented and the authors and sponsor (if applicable) must agree to share original data with the editor if requested.
