DIET Classifier Model Analysis for Words Prediction in Academic Chatbot


Wistiani Astuti(1*); Aji Prasetya Wibawa(2); Haviluddin Haviluddin(3); Herdianti Darwis(4);

(1) fakultas ilmu komputer universitas muslim indonesia
(2) Universitas Negeri Malang
(3) Universitas Mulawarman
(4) Univeristas Muslim Indonesia
(*) Corresponding Author

  

Abstract


One prevalent conversational system within the realm of natural language processing (NLP) is chatbots, designed to facilitate interactions between humans and machines. This study focuses on predicting frequently asked questions by students using the Duel Intent and Entity Transformer (DIET) Classifier method and assessing the performance of this method. The research involves employing 300 epochs with an 80% training data and 20% testing data split. In this study, the DIET Classifier adopts a multi-task transformer architecture to simultaneously handle classification and entity recognition tasks. Notably, it possesses the capability to integrate diverse word embeddings, such as BERT and GloVe, or pre-trained words from language models, and blend them with sparse words and n-gram character-level features in a plug-and-play manner. Throughout the training process of the DIET Classifier model, data loss and accuracy from both training and testing datasets are monitored at each epoch. The evaluation of the text classification model utilizes a confusion matrix. The accuracy results for testing the DIET Classifier method are presented through four case studies, each comprising 25 text messages and 15 corresponding chatbot responses. The obtained accuracy values range from 0.488 to 0.551, F1-Score values range from 0.427 to 0.463, and precision range from 0.417 to 0.457.


Keywords


Chatbot, Confusion Matrix, Diet Classifier.

  
     

Article Metrics

Abstract view: 11 times
     

Digital Object Identifier

doi  https://doi.org/10.33096/ilkom.v16i1.1598.%25p 		  

Cite

How to cite item

References


D. Biswas, “Self-improving Chatbots based on Reinforcement Learning Self-improving Chatbots based on Reinforcement Learning,” no. May, 2019.

A. Jiao, “An Intelligent Chatbot System Based on Entity Extraction Using RASA NLU and Neural Network,” J. Phys. Conf. Ser., vol. 1487, no. 1, 2020, doi: 10.1088/1742-6596/1487/1/012014.

P. Lauren and P. Watta, “A Conversational User Interface for Stock Analysis,” Proc. - 2019 IEEE Int. Conf. Big Data, Big Data 2019, pp. 5298–5305, 2019, doi: 10.1109/BigData47090.2019.9005635.

S. Raj, Building Chatbots with Python. 2019.

R. K. Sharma, “An Analytical Study and Review of open S ource Chatbot framework , R ASA,” vol. 9, no. 06, pp. 1011–1014, 2020.

T. Bunk, D. Varshneya, V. Vlasov, and A. Nicho, “DIET: Lightweight language understanding for dialogue systems,” arXiv, 2020.

D. A. N. Mysql, “Bot Whatsapp Sebagai Pemberi Data Statistik,” vol. 1, no. 2, pp. 282–293, 2020.

K. Chung and R. C. Park, “Chatbot-based heathcare service with a knowledge base for cloud computing,” Cluster Comput., vol. 22, pp. 1925–1937, 2019, doi: 10.1007/s10586-018-2334-5.

T. N. Stephens, A. Joerin, M. Rauws, and L. N. Werk, “Feasibility of pediatric obesity and prediabetes treatment support through Tess, the AI behavioral coaching chatbot,” Transl. Behav. Med., vol. 9, no. 3, pp. 440–447, 2019, doi: 10.1093/tbm/ibz043.

F. Patel, R. Thakore, I. Nandwani, and S. K. Bharti, “Combating depression in students using an intelligent ChatBot: A cognitive behavioral therapy,” 2019 IEEE 16th India Counc. Int. Conf. INDICON 2019 - Symp. Proc., pp. 1–4, 2019, doi: 10.1109/INDICON47234.2019.9030346.

R. Singh, M. Paste, N. Shinde, H. Patel, and N. Mishra, “Chatbot using TensorFlow for small Businesses,” Proc. Int. Conf. Inven. Commun. Comput. Technol. ICICCT 2018, no. Icicct, pp. 1614–1619, 2018, doi: 10.1109/ICICCT.2018.8472998.

F. P. Putri, H. Meidia, and D. Gunawan, “Designing intelligent personalized chatbot for hotel services,” ACM Int. Conf. Proceeding Ser., pp. 468–472, 2019, doi: 10.1145/3377713.3377791.

A. Nursetyo, D. R. I. M. Setiadi, and E. R. Subhiyakto, “Smart chatbot system for E-commerce assitance based on AIML,” 2018 Int. Semin. Res. Inf. Technol. Intell. Syst. ISRITI 2018, pp. 641–645, 2018, doi: 10.1109/ISRITI.2018.8864349.

W. Astuti, D. P. I. Putri, A. P. Wibawa, Y. Salim, Purnawansyah, and A. Ghosh, “Predicting Frequently Asked Questions (FAQs) on the COVID-19 Chatbot using the DIET Classifier,” Apr. 2021, doi: 10.1109/EIConCIT50028.2021.9431913.

T. Bocklisch, J. Faulkner, N. Pawlowski, and A. Nichol, “Rasa: Open source language understanding and dialogue management,” arXiv, pp. 1–9, 2017.

T. Desot, S. Raimondo, A. Mishakova, F. Portet, and M. Vacher, “Towards a french smart-home voice command corpus: Design and NLU experiments,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 11107 LNAI, pp. 509–517, 2018, doi: 10.1007/978-3-030-00794-2_55.

A. Singh, K. Ramasubramanian, S. Shivam, A. Singh, K. Ramasubramanian, and S. Shivam, Introduction to Microsoft Bot, RASA, and Google Dialogflow. 2019.

V. A. Bhagwat, “Deep Learning for ChatBots,” Scholarworks.Sjsu.Edu, p. 56, 2018, [Online]. Available: https://scholarworks.sjsu.edu/cgi/viewcontent.cgi?article=1645&context=etd_projects.

J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” arXiv, pp. 4171–4186, 2018.

M. Henderson et al., “A repository of conversational datasets,” arXiv, 2019, doi: 10.18653/v1/w19-4101.

and C. Jeffrey Pennington, Richard Socher and Manning., “No Title,” in Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 1532–1543.

Ł. K. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, “Attention Is All You Need Ashish,” 2017, doi: 10.1109/2943.974352.

P. Shaw, J. Uszkoreit, and A. Vaswani, “Self-attention with relative position representations,” NAACL HLT 2018 - 2018 Conf. North Am. Chapter Assoc. Comput. Linguist. Hum. Lang. Technol. - Proc. Conf., vol. 2, pp. 464–468, 2018, doi: 10.18653/v1/n18-2074.

L. Wu, A. Fisch, S. Chopra, K. Adams, A. Bordes, and J. Weston, “StarSpace: Embed All The Things!,” 2018.

V. Vlasov, J. E. M. Mosig, and A. Nichol, “Dialogue Transformers,” pp. 1–8.


Refbacks

  • There are currently no refbacks.


Copyright (c) 2024 Wistiani Astuti

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.