A Hybrid BERT–RAG Model for Developing Knowledge-Validated Conversational Systems


Desi Anggreani(1*); Ismawati Ismawati(2); A. Inayah Auliyah(3); Lukman Lukman(4); Aedah Abd Rahman(5); Nurmisba Nurmisba(6); Muh Ilham Akbar(7);

(1) Universitas Muhammadiyah Makassar
(2) Universitas Muhammadiyah Makassar
(3) Institut Teknologi Bacharuddin Jusuf Habibie
(4) Universitas Muhammadiyah Makassar
(5) Asia E University
(6) Universitas Muhammadiyah Makassar
(7) Universitas Muhammadiyah Makassar
(*) Corresponding Author

  

Abstract


The transition of freshmen into the university environment requires adaptive and responsive information support. This study develops a chatbot system based on a hybrid BERT–RAG architecture integrated with the FAISS Index to provide automated consultation services for new students. The novelty of this research lies in the implementation of a faculty-based hierarchical knowledge structure and an adaptive multi-domain context mechanism—an approach not previously found in studies involving BERT–RAG for university onboarding services. This design enables the chatbot to deliver more relevant, personalized, and faculty-specific responses. The dataset was derived from three primary sources of information: the Faculty of Economics and Business (FEB), the Faculty of Teacher Training and Education (FKIP), and the Faculty of Engineering (FT), which were structured into a validated knowledge base in documents.json format. System evaluation was conducted across ten interaction scenarios using performance metrics including BERT Similarity, BLEU Score, ROUGE-1, ROUGE-2, and ROUGE-L. The system achieved excellent results, with average scores of 0.905 (BERT Similarity), 0.844 (BLEU), 0.876 (ROUGE-1), 0.820 (ROUGE-2), and 0.871 (ROUGE-L) and standard deviations below 0.1 across all metrics. Strong metric correlations (0.85–0.99) further indicate consistency between semantic understanding and generated text quality. Furthermore, the system effectively minimizes hallucination through validated knowledge integration and faculty-based reranking strategies. Overall, this research provides a significant contribution to the development of institutionally contextual educational chatbots capable of delivering accurate, natural, and responsive communication to support new student orientation in higher education


Keywords


BERT, Retrieval-Augmented Generation (RAG), Educational Chatbot, University Freshmen, FAISS Index, Natural Language Processing

  
  

Full Text:

PDF
  

Article Metrics

Abstract view: 92 times
PDF view: 56 times 		     

Digital Object Identifier

doi  https://doi.org/10.33096/ilkom.v18i1.3126.30-42 		  

Cite

How to cite item

References


L. M. Ramjan et al., "Academic support strategies for nursing students with a disability at university — An integrative review," Collegian, vol. 32, no. 4, pp. 195–211, Aug. 2025. doi: 10.1016/j.colegn.2025.05.001.

J. Wang, "Impact of natural disasters on student enrollment in higher education programs: A systematic review," Heliyon, vol. 10, no. 6, e27705, Mar. 2024. doi: 10.1016/j.heliyon.2024.e27705.

B. Shang and Z. He, "AI-assisted symbolic healing for digital natives: A Jungian perspective based on the S-O-R and Fogg behavior models," Acta Psychologica, vol. 260, 105472, Oct. 2025. doi: 10.1016/j.actpsy.2025.105472.

Y. Wang et al., "Stressors in university life and anxiety symptoms among international students: a sequential mediation model," BMC Psychiatry, vol. 23, no. 1, p. 556, Aug. 2023. doi: 10.1186/s12888-023-05046-7.

K. Zhang, Z. Mi, E. J. Parks-Stamm, W. Cao, Y. Ji, and R. Jiang, "Adaptability protects university students from anxiety, depression, and insomnia during remote learning: A three-wave longitudinal study from China," Frontiers in Psychiatry, vol. 13, 868072, 2022. doi: 10.3389/fpsyt.2022.868072.

T. Rohimah, Firman, N. S. Neviyarni, and M. A. C. Amat, "Optimizing counseling programs in higher education and their future implications," Quality: Journal of Education, Arabic and Islamic Studies, vol. 3, no. 1, 2025. doi: 10.58355/qwt.v3i1.97.

D. Gilani, "Student attitudes and preferences towards communications from their university – a meta-analysis of student communications research within UK higher education institutions," Journal of Higher Education Policy and Management, vol. 46, no. 3, pp. 274–290, 2024. doi: 10.1080/1360080X.2024.2344234.

T. Ratnasari, "Persepsi mahasiswa terhadap layanan online aplikasi pelayanan terpadu satu pintu pada Biro Administrasi Umum Akademik dan Kemahasiswaan IAIN Kudus," Diplomatika: Jurnal Kearsipan Terapan, vol. 6, no. 1, pp. 17–26, 2022. doi: 10.22146/diplomatika.8312117.

S. H. Anwar, K. M. Abouaish, E. M. Matta, A. K. Farouq, A. A. Ahmed, and N. K. Negied, "Academic assistance chatbot – a comprehensive NLP and deep learning-based approaches," Indonesian Journal of Electrical Engineering and Computer Science, vol. 33, no. 2, pp. 1042–1056, Feb. 2024. doi: 10.11591/ijeecs.v33.i2.pp1042-1056.

A. Puspitasari, A. N. Paradhita, Y. W. Tineka, V. Sulistyowati, N. K. S. Noriska, and Haryanto, "Natural language processing (NLP) technology for chatbot website," Jurnal Penelitian Pendidikan IPA, vol. 10, Special Issue, pp. 319–324, 2024. doi: 10.29303/jppipa.v10iSpecialIssue.8241.

C. Suardi, D. Anggeani, A. P. Wibawa, N. Murtadlo, I. A. E. Zaeni, and N. A. M. Jabari, "Asking a chatbot for food ingredients halal status," in Halal Development: Trends, Opportunities and Challenges, pp. 14–20, 2021.

C. Eang and S. Lee, "Improving the accuracy and effectiveness of text classification based on the integration of the BERT model and a recurrent neural network (RNN_BERT_Based)," Applied Sciences, vol. 14, no. 18, p. 8388, Sep. 2024. doi: 10.3390/app14188388.

Y. Sun, "The evolution of transformer models from unidirectional to bidirectional in natural language processing," Applied and Computational Engineering, Feb. 2024. doi: 10.54254/2755-2721/42/20230794.

Z. Levonian, C. Li, W. Zhu, A. Gade, O. Henkel, M.-E. Postle, and W. Xing, "Retrieval-augmented generation to improve math question-answering: Trade-offs between groundedness and human preference (Version 2)," arXiv, 2023. doi: 10.48550/arXiv.2310.03184.

O. Ayala and P. Bechard, "Reducing hallucination in structured outputs via retrieval-augmented generation," in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2024, pp. 228–238. doi: 10.18653/v1/2024.naacl-industry.19.

J. Swacha and M. Gracel, "Retrieval-augmented generation (RAG) chatbots for education: A survey of applications," Applied Sciences, vol. 15, no. 8, p. 4234, Apr. 2025. doi: 10.3390/app15084234.

W. C. Choi and C. I. Chang, "A survey of techniques, design, applications, challenges, and student perspective of chatbot-based learning tutoring system supporting students to learn in education," Preprints, 2025, 2025031134. doi: 10.20944/preprints202503.1134.v1.

K. Saluja, S. Agarwal, S. Kumar, and T. Choudhury, "Evaluating performance of conversational bot using Seq2Seq model and attention mechanism," EAI : Endorsed Transactions on Scalable Information Systems, vol. 24, no. 6, Mar. 2024. doi: 10.4108/eetsis.5457.

Z. Xu, D. Chen, J. Kuang, Z. Yi, Y. Li, and Y. Shen, "Dynamic Demonstration Retrieval and Cognitive Understanding for Emotional Support Conversation," International ACM SIGIR Conference on Research and Development in Information Retrieval, 2024. doi: 10.1145/3626772.3657695.

O. Henkel, Z. Levonian, C. Li, and M. Postle, "Retrieval-augmented Generation to Improve Math Question-Answering: Trade-offs Between Groundedness and Human Preference," in Proceedings of the 17th International Conference on Educational Data Mining, Jul. 2024, pp. 315–320, doi: 10.5281/zenodo.12729824.

H.-T. Ho, T.-T.-H. Nguyen, D. N. M. Huy, and L. V. Nguyen, "Bio-Inspired Algorithms in NLP Techniques: Challenges, Limitations and Its Applications," Computers, Materials and Continua, vol. 83, no. 3, pp. 3945–3973, May 2025, doi: 10.32604/cmc.2025.063099.

R. C. Torres, "AI Hallucination in the Context of Education: Exploring College Students’ Use of Generative AI for Academic Tasks," 2025 16th International Conference on E-Education, E-Business, E-Management and E-Learning (IC4e), Tokyo, Japan, 2025, pp. 445-449, doi: 10.1109/IC4e65071.2025.11075444.

D. Uysal, D. Toka, E. Bozkurt, O. Kumaş, and H. H. Yılmaz, "Enhancing Financial NLP with Supercomputing: Spell Correction and Domain-Specific BERT Pretraining," Procedia Computer Science, vol. 267, pp. 176–186, 2025, doi: 10.1016/j.procs.2025.08.244.

R. Datu, D. F. Antara, J. J. Sumangando, H. R. Kakunsi, A. E. Handoko, and J. R. Patroli, "Online Behavior and the Transformation of Interpersonal Communication in the Social Media Era," Jurnal Syntax Admiration, vol. 6, no. 2, Feb. 2025, doi: 10.46799/jsa.v6i2.2141.

X. Zhang, W. Wang, and Q. Jin, "IntentionESC: An Intention-Centered Framework for Enhancing Emotional Support in Dialogue Systems," arXiv preprint arXiv:2506.05947, Jun. 2025, doi: 10.48550/arXiv.2506.05947.

J. Liu, "Deep Semantic Analysis and Thematic Evolution Prediction of American Literary Texts Based on BERT and Knowledge Graph," 2025 IEEE International Conference on Electronics, Energy Systems and Power Engineering (EESPE), Shenyang, China, 2025, pp. 218-224, doi: 10.1109/EESPE63401.2025.10986868.

N. Reddy, "Design and Implementation of an AI-Based Chatbot Framework with Retrieval-Augmented Generation and Integrated Recommender System for Interactive User Support," SSRN, 2025, [Online]. Available: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5250507.

M. Ahmad, “Toward a unified framework for information retrieval in large language model applications: Balancing textual and graph-based knowledge sources”, Master’s thesis, School of Engineering, May 2025. [Online]. Available: https://aaltodoc.aalto.fi/items/8c94c500-844b-4587-97df-6bfe733e8816.

A. A. Aliero, B. S. Adebayo, H. O. Aliyu, A. G. Tafida, B. U. Kangiwa, and N. M. Dankolo, "Systematic Review on Text Normalization Techniques and its Approach to Non-Standard Words," International Journal of Computer Applications, vol. 185, no. 33, pp. 44–55, Sep. 2023, doi: 10.5120/ijca2023923106.

L. Hui and M. Belkin, "Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks," arXiv preprint arXiv:2006.07322, Oct. 2021, doi: 10.48550/arXiv.2006.07322.

S. Pal, M. Chang, and M. F. Iriarte, "Summary Generation Using Natural Language Processing Techniques and Cosine Similarity," in Intelligent Systems Design and Applications, 2022, vol. 418, pp. 485–497, doi: 10.1007/978-3-030-96308-8_47.

M. T. Rustam, M. Muhatri, dan A. B. Nasution, "Design and Construction of QA (Question Answering) System Based on Artificial Intelligence Using FAISS and Mixtral Language Model," IT Journal, vol. 13, no. 1, pp. 44–55, 2025. [Online]. https://upu-journal.potensi-utama.org/index.php/itjournal/article/view/370?utm_source=chatgpt.com

Z. Tang, H. Fan, X. Gu, J. Zhou, H. Ma, A. V. Vasilakos, and B. Li, "Enabling efficient and accurate semantic search over encrypted cloud data," Information Sciences, vol. 719, Nov. 2025, Art. no. 122437, doi: 10.1016/j.ins.2025.122437.

A. Ranjan and M. Ravinder, "Text Extraction from Blurred Images through NLP-based Post-processing," in A Handbook of Computational Linguistics: Artificial Intelligence in Natural Language Processing, 2024, pp. 285–300, doi: 10.2174/97898152384881240201.

J. Junadhi, A. Agustin, L. Efrizoni, F. Okmayura, D. R. Habibie, and M. Muslim, "Improving Evaluation Metrics for Text Summarization: A Comparative Study and Proposal of a Novel Metric," Journal of Applied Data Sciences, vol. 6, no. 2, Jun. 2025, doi: 10.47738/jads.v6i2.547.

M. Ghassemiazghandi, "An Evaluation of ChatGPT's Translation Accuracy Using BLEU Score," Theory and Practice in Language Studies, vol. 14, no. 4, pp. 985–994, Apr. 2024, doi: 10.17507/tpls.1404.07.

S. Kumar and A. Solanki, "ROUGE-SS: A New ROUGE Variant for Evaluation of Text Summarization," Authorea, Jul. 20, 2023, doi: 10.22541/au.168984209.92955863/v1.


Refbacks

  • There are currently no refbacks.


Copyright (c) 2026 Desi Anggreani, Ismawati Ismawati, A. Inayah Auliyah, Lukman, Aedah Abd Rahman, Nurmisba Nurmisba, Muh Ilham Akbar

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.