Optimization of CNN Architectures through Fine-tuning for SIBI Classification


Nur Hilmi Insan Muhammad(1*); Abdul Syukur(2); Pujiono Pujiono(3);

(1) Universitas Dian Nuswantoro
(2) Universitas Dian Nuswantoro
(3) Universitas Dian Nuswantoro
(*) Corresponding Author

  

Abstract


This research addresses the computational optimization of convolutional neural network (CNN) architectures for the classification of Indonesian Sign Language System (Sistem Isyarat Bahasa Indonesia, SIBI) static alphabet imagery to enhance digital communication accessibility. Utilizing a domain-specific dataset comprising 1,165 images across 26 alphabet classes, this study tackles the prominent challenges of limited sample sizes and severe class imbalance. We evaluate five state-of-the-art CNN architectures MobileNetV2, DenseNet121, Xception, InceptionV3, and ResNet50V2 under four distinct training data paradigms before and after adaptive fine-tuning. To eliminate predictive bias without pixel-level distortion, oversampling is operationalized via Latent Space SMOTE on flattened vector embeddings, combined with dynamistic runtime image augmentation. The experimental results reveal that MobileNetV2, when optimized through partial layer-freezing (locking 150 baseline layers) under the integrated augmentation and oversampling combination scenario, achieved the highest macro-classification accuracy of 98.30%. This architecture also demonstrated superior efficiency, reducing the computational training latency to 0.53 minutes. The findings underscore the strategic advantage of leveraging optimized lightweight networks like MobileNetV2 for domain-specific visual recognition tasks.

Keywords


CNN; MobileNetV2; SIBI; Classification; Fine-tuning

  
  

Full Text:

PDF
  

Article Metrics

Abstract view: 56 times
PDF view: 16 times 		     

Digital Object Identifier

doi  https://doi.org/10.33096/ilkom.v18i2.2830.378-392 		  

Cite

How to cite item

References


H. Herdiansyah, M. R. Soedrajad, N. I. Hawa, and A. P. R. Dyah Ayu Arintyas, “Resetting The Relationship Between Human And Nature: Religion And The New Communication Technologies,” Journal for the Study of Religions and Ideologies, vol. 22, no. 66, 2023.

R. Z. Palindungan, “Sistem Penerjemah Bahasa Isyarat Otomatis Menggunakan Metode Deep Learning Model Convolutional Neural Network,” Pesquisa Veterinaria Brasileira, 2021.

A. Wadhawan and P. Kumar, “Sign Language Recognition Systems: A Decade Systematic Literature Review,” Archives of Computational Methods in Engineering, vol. 28, no. 3, 2021.

M. M. Alnfiai, “Deep Learning-Based Sign Language Recognition for Hearing and Speaking Impaired People,” Intelligent Automation & Soft Computing, vol. 36, no. 2, pp. 1653–1669, 2023.

C. K. M. Lee, K. K. H. Ng, C.-H. Chen, H. C. W. Lau, S. Y. Chung, and T. Tsoi, “American sign language recognition and training method with recurrent neural network,” Expert Syst. Appl., vol. 167, p. 114403, Apr. 2021.

A. Muh. A. Siddik, “Comparison of Transfer Learning Algorithm Performance in Hand Sign Language Digits Image Classification,” Jurnal Matematika, Statistika dan Komputasi, vol. 20, no. 1, 2023.

M. F. Naufal and S. F. Kusuma, “Analisis Perbandingan Algoritma Machine Learning dan Deep Learning untuk Klasifikasi Citra Sistem Isyarat Bahasa Indonesia (SIBI),” Jurnal Teknologi Informasi dan Ilmu Komputer, vol. 10, no. 4, pp. 873–882, Aug. 2023.

I. D. A. Rachmawati, R. Yunanda, M. F. Hidayat, and P. Wicaksono, “Deep Transfer Learning for Sign Language Image Classification: A Bisindo Dataset Study,” Engineering, MAthematics and Computer Science Journal (EMACS), vol. 5, no. 3, 2023.

A. Hussain, S. Ul Amin, M. Fayaz, and S. Seo, “An Efficient and Robust Hand Gesture Recognition System of Sign Language Employing Finetuned Inception-V3 and Efficientnet-B0 Network,” Computer Systems Science and Engineering, vol. 46, no. 3, pp. 3509–3525, 2023.

J. P. Sahoo, A. J. Prakash, P. Pławiak, and S. Samantray, “Real-Time Hand Gesture Recognition Using Fine-Tuned Convolutional Neural Network,” Sensors, vol. 22, no. 3, p. 706, Jan. 2022.

D. M. Cherian and J. J. Fernandez, “Real-time sign language recognition and speech conversion using VGG16,” Int. J. Comput. Vis. Robot., vol. 13, no. 2, 2023.

C. Yin, Y. Zhu, J. Fei, and X. He, “A Deep Learning Approach for Intrusion Detection Using Recurrent Neural Networks,” IEEE Access, vol. 5, 2017.

B. Maity, A. Alim, S. Bhattacharjee, and S. Nandi, “DeHonk: A deep learning based system to characterize vehicular honks in presence of ambient noise,” Pervasive Mob. Comput., vol. 88, 2022.

Mark Sandler, A. Howard, M. Zhu, A. Zhmoginov, and Liang-Chieh Chen, “MobileNetV2: Inverted Residuals and Linear Bottlenecks Mark,” Convolutional Neural Networks with Swift for Tensorflow, 2019.

K. Weiss, T. M. Khoshgoftaar, and D. D. Wang, “A survey of transfer learning,” J. Big Data, vol. 3, no. 1, 2016.

C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu, “A survey on deep transfer learning,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2018.

X. Zhao, L. Wang, Y. Zhang, X. Han, M. Deveci, and M. Parmar, “A review of convolutional neural networks in computer vision,” Artif. Intell. Rev., vol. 57, no. 4, 2024.

S. Sharmin, T. Ahammad, M. A. Talukder, and P. Ghose, “A Hybrid Dependable Deep Feature Extraction and Ensemble-Based Machine Learning Approach for Breast Cancer Detection,” IEEE Access, vol. 11, 2023.

M. Buda, A. Maki, and M. A. Mazurowski, “A systematic study of the class imbalance problem in convolutional neural networks,” Neural Networks, vol. 106, 2018.

C. Shorten and T. M. Khoshgoftaar, “A survey on Image Data Augmentation for Deep Learning,” J. Big Data, vol. 6, no. 1, 2019.

C. Szegedy et al., “Going deeper with convolutions,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2015.

L. Alzubaidi et al., “Review of deep learning: concepts, CNN architectures, challenges, applications, future directions,” J. Big Data, vol. 8, no. 1, 2021.

S. Sharma and S. Singh, “Vision-based hand gesture recognition using deep learning for the interpretation of sign language,” Expert Syst. Appl., vol. 182, 2021.

N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: Synthetic minority over-sampling technique,” Journal of Artificial Intelligence Research, vol. 16, 2002.

H. Li, H. Liu, and Y. Hu, “Prediction of Unbalanced Financial Risk Based on GRA-TOPSIS and SMOTE-CNN,” Sci. Program., vol. 2022, 2022.

E. Ileberi, Y. Sun, and Z. Wang, “Performance Evaluation of Machine Learning Methods for Credit Card Fraud Detection Using SMOTE and AdaBoost,” IEEE Access, vol. 9, pp. 165286–165294, 2021.

J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are features in deep neural networks?,” in Advances in Neural Information Processing Systems, 2014.

Y. Nakamura and L. Jing, “Skeleton-Based Data Augmentation for Sign Language Recognition Using Adversarial Learning,” IEEE Access, 2024.

N. Aloysius and M. Geetha, “A scale space model of weighted average CNN ensemble for ASL fingerspelling recognition,” International Journal of Computational Science and Engineering, vol. 22, no. 1, p. 154, 2020.

G. Vrbancic and V. Podgorelec, “Transfer Learning With Adaptive Fine-Tuning,” IEEE Access, vol. 8, pp. 196197–196211, 2020.

S. Kornblith, J. Shlens, and Q. V. Le, “Do better imagenet models transfer better?,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019.

J. Ungredda and J. Branke, “Bayesian Optimisation for Constrained Problems,” ACM Transactions on Modeling and Computer Simulation, vol. 34, no. 2, 2024.

Q. Wu, J. Qi, D. Zhang, H. Zhang, and J. Tang, “Fine-Tuning for Few-Shot Image Classification by Multimodal Prototype Regularization,” IEEE Trans. Multimedia, vol. 26, pp. 8543–8556, 2024.

G. M. Kumar and A. Pandian, “A constructive deep convolutional network model for analyzing video-to-image sequences,” Data Knowl. Eng., vol. 144, 2023.

D. S. Breland, S. B. Skriubakken, A. Dayal, A. Jha, P. K. Yalavarthy, and L. R. Cenkeramaddi, “Deep Learning-Based Sign Language Digits Recognition from Thermal Images with Edge Computing System,” IEEE Sens. J., vol. 21, no. 9, 2021.

W. Nazih, A. O. Aseeri, O. Y. Atallah, and S. El-Sappagh, “Vision Transformer Model for Predicting the Severity of Diabetic Retinopathy in Fundus Photography-Based Retina Images,” IEEE Access, vol. 11, 2023.

M. Abadi, “TensorFlow: learning functions at scale,” ACM SIGPLAN Notices, vol. 51, no. 9, 2016.

H. Mokayed, T. Z. Quan, L. Alkhaled, and V. Sivakumar, “Real-Time Human Detection and Counting System Using Deep Learning Computer Vision Techniques,” Artificial Intelligence and Applications, vol. 1, no. 4, 2022.

Q. M. Areeb, Maryam, M. Nadeem, R. Alroobaea, and F. Anwer, “Helping Hearing-Impaired in Emergency Situations: A Deep Learning-Based Approach,” IEEE Access, vol. 10, 2022.

A. Wadhawan and P. Kumar, “Deep learning-based sign language recognition system for static signs,” Neural Comput. Appl., vol. 32, no. 12, 2020.

M. Heydarian, T. E. Doyle, and R. Samavi, “MLCM: Multi-Label Confusion Matrix,” IEEE Access, vol. 10, 2022.


Refbacks

  • There are currently no refbacks.


Copyright (c) 2026 Nur Hilmi Insan Muhammad, Abdul Syukur, Pujiono Pujiono

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.