DEVELOPMENT OF AN AUTOMATIC EVALUATION MODEL FOR ASSESSING THE QUALITY OF INDONESIAN DIGITAL TRANSLATION

  • Rocia Fani Kaban Universitas HKBP Nommensen
  • Erika Sinambela Universitas HKBP Nommensen
  • Arsen Nahum Situmorang Universitas HKBP Nommensen
DOI: https://doi.org/10.36655/jetal.v8i1.2470
Keywords: Automatic Evaluation Model, Indonesian Translation, TQA, Machine Translation

Abstract

This study aims to develop an automatic evaluation model for assessing the quality of Indonesian digital translations and to produce a simple bilingual translation application called Translate You. The research addresses the limitations of existing evaluation metrics, such as BLEU, METEOR, BERTScore, and COMET, which are not fully suitable for Indonesian linguistic characteristics. This study uses a Research and Development (R&D) method combined with a quantitative approach. The data consist of bilingual sentences collected from online articles. Translation outputs are evaluated using an automatic model and a professional translator as the expert validator. The model integrates semantic similarity, fluency, and Indonesian linguistic features. The results show that the developed model produces evaluation scores that are more aligned with expert judgment compared to traditional metrics. The Translate You application also successfully integrates text translation, voice input, and automatic evaluation in a simple system. This research contributes to Translation Quality Assessment (TQA) by providing an Indonesian-specific evaluation model and a practical prototype for translation assessment.

References

Dahan, N., Bawden, R., Dahan, N., & Yvon, F. (2024). Survey of Automatic Metrics for Evaluating Machine Translation at the Document Level October 2024 Survey of Automatic Metrics for Evaluating Machine Translation at the Document-Level. October, 1–55.
DS Nugraha. (2025). Exploring Integrative Complexity in the Morphology of Indonesian Language.
Effendi, O. D., Ravindran, L., Amini, M., & Alipour, M. (2024). International Journal Of Language Education And Applied Linguistics Exploring Strategies and Perceptions of Quality in English to Indonesian Translation of Idioms in the Short Story “ Dear Life .” 14(2), 7–19.Golovatska, I. (2024). The Use of Digital Technologies to Prepare Future Translators for the Modern Requirements of the Linguistic Services Market. 17(1), 175–187.
H Rahmani, M. K. (2025). The impact of globalization and modern on the English language. 7(2), 31–41.
Haseeb, M., & Abbasi, W. S. (2025). Social science review archives. 885–894.
Kurniadi, P., & Erwin, E. (2024). Language Development in the Digital Age , A Literature Review on the Influence of Technology on Human Communication. 1(01), 1–7.
L Han, A Smeaton, G. J. (2025). Translation Quality Assessment : A Brief Survey on Manual and Automatic Methods. Cl.
Leiter, C., Fomicheva, M., & Eger, S. (2024). Towards Explainable Evaluation Metrics for Machine Translation. 25, 1–49.
Manipuspika, Y. S. (2021). The Effectiveness of Translation Quality Assessment ( TQA ) Models in the Translation of Indonesian Students. 0299(c), 287–297. https://doi.org/10.32996/ijllt
Mienye, I. D. (2024). A Comprehensive Review of Deep Learning : Architectures , Recent Advances , and Applications.
Mohamed, Y. A., Khanan, A., Bashir, M., & Elsadig, M. A. (2024). The Impact of Artificial Intelligence on Language Translation : A Review. January, 25553–25579.
Nida, E. A., & Taber, C. R. (1969). The Theory and Practice of Translation. Leiden: E. J. Brill.
Park, W., & Id, S. K. (2022). Integrating quantitative and qualitative methodologies to build a national R & D plan using data envelopment analysis based on R & D stakeholders ’ perspectives. 1–20. https://doi.org/10.1371/journal.pone.0265058
Rejuna, C., Brahmana, P. S., Kurnia, D., Ningrum, S., & Harahap, A. I. (2022). Translation Quality Assessments : Translators Perspectives TALENTA Conference Series Translation Quality Assessments : Translators Perspectives. 8(3). https://doi.org/10.32734/lwsa.v8i3.2500
Saadany, H., & Or, C. (2021). BLEU , METEOR , BERTScore : Evaluation of Metrics Performance in Assessing Critical Translation Errors in Sentiment-oriented Text. 48–56.
Sinambela, E.(2024). Advancements in Translation Technology : Innovations and Challenges. Medan : LPPM UHN PRESS
Sinambela, E., Sipayung, R. W., Herman, H., Purba, R., Fatmawati, E., Ngongo, M., & Manurung, L. W. (2024). Investigating translation strategies used by students in translating metaphors from English into Indonesian: A case study on translation. Research Journal in Advanced Humanities, 5(4).
Sneddon, J. N., Adelaar, K. A., Djenar, D. N., & Ewing, M. C. (2010). Indonesian reference grammar (2nd ed.). Allen & Unwin.
Susilawati, A., Sh, A., Al, M., Abduh, A., & Irwansyah, F. S. (2025). Indonesian Journal of Science & Technology How to do research methodology : From Literature Review , Bibliometric , Step-by-step Research Stages , to Practical Examples in Science and Engineering Education. 10(1), 1–40.
Thi, N., Ngoc, N., Truc, T. T., Ngoc, N., An, H., & Phat, L. H. (2025). The Benefits and Challenges of AI Translation Tools in Translation Education at the Tertiary Level : A Systematic Review. 5(2), 132–148.
Wajdi, M., & Nurani, S. G. (2025). English-Indonesian Translation Majid Wajdi.
Zandian, S., Ketabi, S., & Dastjerdi, H. V. (2021). In Keeping with New Translation Quality Assessment (TQA) Trends: From Planning to Applying a Viable Rubric to Assess Quality of Literary Translation. 1.
ZS Almalki. (2023). Production and Judgements of Complaints in Instant Messaging : A Cross-cultural and Interlingual Approach.
Published
2026-09-28
Issue
Vol 8 No 1 (2026): September
Section
Articles
Creative Commons License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0) that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.

Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.

Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).