Evaluation of lambung mangkurat university student academic portal using user experience questionnaire (UEQ)

The student academic portal is one of the Academic Information Systems at The University of Lambung Mangkurat (ULM). The ULM student academic portal can only be accessed by the ULM students. It can be used for academic guidance, managing student study plans, printing test cards, filling out questionnaires


Introduction
Academic Information System (AIS) is a system that processes academic data and manages business processes in educational institutions. AIS plays a vital role in educational institutions because it can manage major business processes such as those in universities. Most higher education institutions use a web based AIS that is connected to the intranet or the internet. A well-managed AIS can produce good quality information that is useful in terms of decision making, such as strategic plans to support the achievement of organizational goals [1][2] [3].
The Academic Information System of Lambung Mangkurat University (AIS ULM) can be used to manage academic activities in the University such as faculty data management, department and study programs data, student data, curriculum data, registration of student in each semester, academic supervisor and academic guidance, Class settings, lecture attendance, and the assessment. In addition, there are two AIS used in ULM; the lecturer academic portal and student academic portal [4]. Although the website-based student academic portal has been released and used since 2016, there is still no publication on usability evaluation on the website-based ULM student academic portal. Evaluation is one of the stages in the Software Development Life Cycle (SDLC). This stage allows the user to provide an assessment of the system [5]. Usability is an important issue that is inseparable from Human-Computer Interaction (HCI) because it is an important aspect to assess the quality of the user interface of an interactive system [6][7][8] [9]. Therefore, usability evaluation must be carried out as part of developing a system that meets HCI standards. Even usability assessments carried out before users use the system can reduce system maintenance and training costs and increase the system's usability [8] [10].
Research conducted by A. A. I. I. Paramitha in 2018 [11] evaluated the interface and user experience of SIsKA. SIsKA is a web-based information system for managing the thesis of postgraduate students majoring in computer science at Ganesha University of Education. This evaluation is carried out to find out what problems are experienced by the user when using the system and measure the user's response when interacting through the system. The evaluation was carried out using a user experience questionnaire (UEQ). The results of the UEQ, after comparing the average scores of each aspect using the UEQ Data Analysis Tool, it was found that the aspect of novelty, efficiency, Sari, et. al. (Evaluation of lambung mangkurat university student academic portal using user experience questionnaire (UEQ)) perspicuity, and attractiveness in SIsKA were categorized as good. In addition, the dependability aspect is classified above average, and the stimulation aspect is classified as very good.
Another study was conducted by N. Limantara, et al, in 2019 [12], to evaluate the BeeCEO application before it was implemented. This is to find out if BeeCEO has met all dimensions in terms of user experience or still needs improvement. BeeCEO can be classified as a digital educational game. The user experience dimension can be assessed using the User Experience Questionnaire (UEQ). UEQ will evaluate software based on 6 dimensions: attractiveness, perspicuity, efficiency, reliability, stimulation, and novelty. The evaluation results show that the efficiency dimension gets the lowest average value of 0.547, with the speed item having an average value of -0.9. Meanwhile, the stimulation dimension has the highest average of 1,422.
The solution proposed is based on the above problems. It is necessary to evaluate the usability of the Lambung Mangkurat University student portal using the User Experience Questionnaire (UEQ). It is expected that by conducting a usability evaluation on the ULM student academic portal, it can be seen how the user's assessment of the running system and can be a reference for improvement.

A. Population and Research Sample
Population is a set of observation objects that have certain characteristics to be studied. In a study, it is often impossible to use the entire population. Thus, the researcher will use a sample to represent the entire population [13]. In this study, the population taken is Lambung Mangkurat University students. The number of students at Lambung Mangkurat University based on data from the Higher Education Database of the Ministry of Research, Technology and Higher Education (PDDIKTI) is 24,525 students [14] The research sample is a small part of the population with certain similarities and is considered to represent the entirety of the existing population. Determination of the number of samples is done in certain ways. The aim is to get a picture of the object's characteristics being observed and estimate population parameters and make generalizations. This is done if the population to be studied relatively large, so it is impossible to conduct research directly on all individuals or units in the population [15]. For data collection using UEQ, 20-30 samples can provide relatively stable results. Previous studies that used the UEQ also used the number of samples as a reference, such as a study conducted [12], which used 16 samples. Then the research was conducted [16], which used 20 samples, then [17], which used 38 samples in the study.
There are two sampling techniques that can generally be used: non-probability sampling and probability sampling [15]. However, this research will use simple random sampling and probability sampling techniques. This technique is used because it is assumed that all populations, which are the students of Lambung Mangkurat University, have used the Student Academic Portal. Therefore, all population units can be sampled because they are considered to have the same opportunities. To determine the number of samples to be used, it is calculated using the slovin formula with a margin of error of 10% [18] [19]. The calculation can be seen formula (1).

B. User Experience Questionnaire (UEQ)
The User Experience Questionnaire (UEQ) measures user experience using a questionnaire that includes 6 aspects: attractiveness, novelty, stimulation, dependability, efficiency, and perspicuity. Attractiveness has 6 items to be assessed, while the other 5 scales have 4 items. Each item is displayed in two terms that have opposite meanings. A seven-level scale was used to reduce bias, where each item was rated from -3 to +3. The UEQ is already available in Indonesian, and the outline of the UEQ questionnaire is shown in Table 1 [20].

Table 1. Outline of the questionnaire Aspect Indicators
Attractiveness How fun the system is used by the user How good the system is used by the user How users like the system The Convenience of user in using the system How attractive the system is used by the user How friendly the system is used by the user Perspicuity How easy the system is understood by the user How easy the system is learned by the user How easy the system is used by the user How clear the system is used by the user

Efficiency
System speed in processing tasks The efficiency of the system in processing tasks Simplicity of the system in processing assignments How organized the system is processing the task

Dependability
The system can be predicted The system can support the user System security System suitability with expectations Stimulation How valuable the system for the user How exciting the system for the user How interesting the system is for the user How motivated the system for the user Novelty Creative system Ingenious system The leading system Innovative system The ULM student academic portal has been used since 2016 to carry out academic activities such as academic guidance, managing student study plans, printing test cards, filling out questionnaires, and viewing exam results (assessment). UEQ is used to measure the user experience of the ULM student academic portal user to find out how users think about the student academic portal from various aspects. It is necessary so that users can participate in future system development. Existing questionnaires will be distributed to Lambung Mangkurat University students according to the predetermined sample. The results of this questionnaire will show which aspects, according to the user, are still lacking.
The collected questionnaire data will be managed using the Data Analysis Tool. The Data Analysis Tool is a data processing tool provided by UEQ, this tool is in the form of an excel file with many worksheets and built-in formulas. The display of this tool is shown in Figure 1.

A. Questionnaire Distribution
In this study, questionnaires were distributed to Lambung Mangkurat University students who are the ULM Student Academic Portal users. The target respondents of this questionnaire are 100 students who are determined using simple random sampling and probability sampling techniques. The questionnaires in this study were distributed using an online survey using Google Form.

B. User Experience Questionnaire (UEQ) results
To get the results of the analysis of the UEQ questionnaire, researchers used the UEQ Data Analysis Tool provided by UEQ. Based on 100 respondents, the results of the assessment were based on aspects of attractiveness, perspicuity (clarity), efficiency, dependability (accuracy), stimulation, and novelty, as shown in Table 2 and Figure 2. The values above 0.8 are considered positive, -0.8 to 0.8 is considered neutral, and values below -0.8 are considered negative.   Table 2 and Figure 2, the 5-scale value is above 0.8, which means positive, while the novelty scale is between -0.8 and 0.8, which means neutral. Therefore, the highest value is 1.458 (Perspicuity), and the lowest value is 0.653 (Novelty).
The UEQ Analysis Tool has a feature to compare the product or system being evaluated with the data in the Benchmark feature. This data set consists of data from 20,190 people from 452 studies on various products, such as business software, web pages, webshops, and social networks. The comparison of the Student Academic Portal with the data in the benchmark is shown in Table 3 and Figure 3.

Scale
Average   Table 3 and Figure 3, the perspicuity, accuracy, and stimulation scales are above the average, which means that there are 50% of the data in the Benchmark which is worse than the ULM Student Academic Portal, while the attractiveness, efficiency, and novelty scales are below the average. There are only 25% data in the Benchmark, which is worse than the ULM Student Academic Portal.

Conclusion
Conclusions that can be drawn based on the formulation of the problem raised are about evaluating the academic portal of the Mangkurat University students by using the User Experience Questionnaire (UEQ). The aspect of perspicuity (clarity) gets the highest value among the other aspects, namely 1,458, which means users find it easy to understand how to use the ULM student academic portal. The novelty aspect gets the lowest value of 0.653 which means that the user considers ULM student academic portal has not yet innovative and creative. Based on the results of data processing using the Benchmark feature on the UEQ Data Analysis Tool, it is known that there are three aspects that are above average, namely aspects of perspicuity, accuracy, and stimulation. It means that there are 50% of the data in the Benchmark, which is worse than the ULM Student Academic Portal. Other aspects, attractiveness, efficiency, and novelty are below the average, which means that only 25% of data in the Benchmark is worse than the ULM Student Academic Portal. Compared with SisKA in research conducted by AAII Paramitha [11], the average value of aspects on the ULM Student Academic Portal is below SisKA. It is because the attractiveness, perspicuity, efficiency, and novelty aspects of SIsKA are categorized as good, the accuracy aspect is categorized above average, and the stimulation aspect was categorized as very good. The evaluation research of the ULM Student Academic Portal using the User Experience Questionnaire (UEQ) only resulted in general assessments from users regarding the UEQ aspects contained in the ULM Student Academic Portal. Further research can use other models or methods that are considered more suitable and can evaluate ULM student academic portals more detail than various aspects to produce better improvement results and suggestions.