European Journal of Business Science and Technology 2024, 10(2):173-184 | DOI: 10.11118/ejobsat.2024.010

Improving Automated Categorization of Customer Requests with Recent Advances in Natural Language Processing

Filip Koukal1, František Dařena1, Roman Ježdík2, Jan Přichystal1
1 Mendel University in Brno, Czech Republic
2 ALVAO, s. r. o., Žďár nad Sázavou, Czech Republic

In this paper, we focus on the categorization of tickets in service desk systems. We employ modern neural network-based artificial intelligence methods to improve the performance of current systems and address typical problems in the domain. Special attention is paid to balancing the ticket categories, selecting a suitable representation of text data, and choosing a classification model. Based on experiments with two real-world datasets, we conclude that text preprocessing, balancing the ticket categories, and using the representations of texts based on fine-tuned transformers are crucial for building successful classifiers in this domain. Although we could not directly compare our work to other research the results demonstrate superior performance to similar works.

Keywords: service desk systems, customer requests classification, transformer models, machine learning
JEL classification: C89, L86

Received: September 13, 2023; Revised: April 12, 2024; Accepted: April 15, 2024; Published: December 31, 2024  Show citation

ACS AIP APA ASA Harvard Chicago IEEE ISO690 MLA NLM Turabian Vancouver
Koukal, F., Dařena, F., Ježdík, R., & Přichystal, J. (2024). Improving Automated Categorization of Customer Requests with Recent Advances in Natural Language Processing. European Journal of Business Science and Technology, 10(2), 173-184. doi: 10.11118/ejobsat.2024.010
Share...
Download citation
  • BibTeX (.bib)
  • Bookends (.ris)
  • EasyBib (.ris)
  • EndNote (.enw)
  • EndNote 8 (.xml)
  • ISI WoS (.isi)
  • Medlars (.medlars)
  • Mendeley (.ris)
  • MODS (.xml)
  • Papers (.ris)
  • RefWorks (.txt)
  • RefManager (.ris)
  • RIS (.ris)
  • MS Word (.xml)
  • Zotero (.ris)
    • Open full article

    References

    1. Aggarwal, C. C. 2020. Data Classification: Algorithms and Applications. Boca Raton: Chapman and Hall/CRC.
    2. Al-Hawari, F. & Barham, H. 2021. A Machine Learning Based Help Desk System for IT Service Management. Journal of King Saud University - Computer and Information Sciences, 33 (6), 702-718. DOI: 10.1016/j.jksuci.2019.04.001 Go to original source...
    3. Arkhipov, M., Trofimova, M., Kuratov, Y. & Sorokin, A. 2019. Tuning Multilingual Transformers for Language-Specific Named Entity Recognition. In Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing, pp. 89-93. DOI: 10.18653/v1/W19-3712 Go to original source...
    4. Bojanowski, P., Grave, E., Joulin, A. & Mikolov, T. 2017. Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics, 5, 135-146. DOI: 10.1162/tacl_a_00051 Go to original source...
    5. Campese, S., Agostini, F., Pazzini, J. & Pozza, D. 2022. Beyond Transformers: Fault Type Detection in Maintenance Tickets with Kernel Methods, Boost Decision Trees and Neural Networks. In 2022 International Joint Conference on Neural Networks (IJCNN), 1-8. DOI: 10.1109/IJCNN55064.2022.9892980 Go to original source...
    6. Chawla, N. V., Bowyer, K. W., Hall, L. O. & Kegelmeyer, W. P. 2022. SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research, 16, 321-357. DOI: 10.1613/jair.953 Go to original source...
    7. Chen, T. & Guestrin, C. 2016. XGBoost: A Scalable Tree Boosting System. In KDD '16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785-794. DOI: 10.1145/2939672.2939785 Go to original source...
    8. Choi, H., Kim, J., Joe, S. & Gwon, Y. 2021. Evaluation of BERT and ALBERT Sentence Embedding Performance on Downstream NLP Tasks. ArXiv: 2101.10642v1. DOI: 10.48550/arXiv.2101.10642 Go to original source...
    9. Coulombe, C. 2018. Text Data Augmentation Made Simple by Leveraging NLP Cloud APIs. ArXiv: 1812.04718v1. DOI: 10.48550/arXiv.1812.04718 Go to original source...
    10. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. ArXiv: 1810.04805v2. DOI: 10.48550/arXiv.1810.04805 Go to original source...
    11. Eichhorn, G. 2020. Predict IT Support Tickets with Machine Learning and NLP [online]. Available at: https://towardsdatascience.com/predict-it-support-tikets-with-machine-learning-and-nlp-a87ee1cb66fc. [Accessed 2023, May 1].
    12. Goutte, C. & Gaussier, E. 2005. A Probabilistic Interpretation of Precision, Recall and F-Score, with Implication for Evaluation. In Losada, D. E. & Fernández-Luna, J. M. (eds.). Advances in Information Retrieval, Lecture Notes in Computer Science, vol. 3408, pp. 345-359. Springer, Berlin, Heidelberg. Go to original source...
    13. Herzig, K., Just, S. & Zeller, A. 2013. It's Not a Bug, It's a Feature: How Misclassification Impacts Bug Prediction. In Proceedings of the 2013 International Conference on Software Engineering, 392-401. DOI: 10.1109/ICSE.2013.6606585 Go to original source...
    14. Jäntti, M. 2012. Examining Challenges in IT Service Desk System and Processes: A Case Study. In ICONS 2012: The Seventh International Conference on Systems, 105-108. ISBN 978-1-61208-184-7.
    15. Landsman, I. 2015. A Guide to Support Ticket Categorization [online]. Available at: https://www.helpspot.com/blog/a-guide-to-support-ticket-categorization. [Accessed 2023, January 15].
    16. Liu, Y., Loh, H. T. & Sun, A. 2009. Imbalanced Text Classification: A Term Weighting Approach. Expert Systems with Applications, 36 (1), 690-701. DOI: 10.1016/j.eswa.2007.10.042 Go to original source...
    17. Menken, I. & Blokdijk, G. 2009. Support Center Complete Handbook: How to Analyze, Assess, Manage and Deliver Customer Business Needs and Exceed Customer Expectations with Help Desk, Support Center and Service Desk. Brisbane: Emereo Publishing.
    18. Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad, N., Chenaghlu, M. & Gao, J. 2021. Deep Learning Based Text Classification: A Comprehensive Review. ArXiv: 2004.03705v3. DOI: 10.48550/arXiv.2004.03705 Go to original source...
    19. Olson, S. 2018. 10 Help Desk Metrics for Service Desks and Internal Help Desks [online]. Available at: https://www.zendesk.com/in/blog/top-10-help-desk-metrics/. [Accessed 2022, November 13].
    20. Opuchlich, P. 2019. Text Classification of Support Ticket Data of SAP [online]. Available at: https://www.researchgate.net/publication/340492353_Text_Classification_of_Support_tiket_Data_of_SAP. [Accessed 2023, January 20].
    21. Paramesh, S. P., Ramya, C. & Shreedhara, K. S. 2018. Classifying the Unstructured IT Service Desk Tickets Using Ensemble of Classifiers. In 3rd International Conference on Computational Systems and Information Technology for Sustainable Solutions (CSITSS), 221-227. DOI: 10.1109/CSITSS.2018.8768734 Go to original source...
    22. Parmar, P. S., Biju, P. K., Snahkar, M. & Kadiresan, N. 2018. Multiclass Text Classification and Analytics for Improving Customer Support Response through Different Classifiers. In International Conference on Advances in Computing, Communications and Informatics (ICACCI), 538-542. DOI: 10.1109/ICACCI.2018.8554881 Go to original source...
    23. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M. & Duchesnay, É. 2011. Scikit-Learn: Machine Learning in Python. Journal of Machine Learning Research, 12 (85), 2825-2830.
    24. Pennington, J., Socher, R. & Manning, C. 2014. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1532-1543. DOI: 10.3115/v1/D14-1162 Go to original source...
    25. Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N. & Huang, X. 2020. Pre-Trained Models for Natural Language Processing: A Survey. Science China Technological Sciences, 63, 1872-1897. DOI: 10.1007/s11431-020-1647-3 Go to original source...
    26. Rajapakse, T. 2023. Simple Transformers [online]. Available at: http://simpletransformers.ai. [Accessed 2023, September 1].
    27. Rogers, A., Kovaleva, O. & Rumshisky, A. 2020. A Primer in BERTology: What We Know About How BERT Works. Transactions of the Association for Computational Linguistics, 8, 842-866. DOI: 10.1162/tacl_a_00349 Go to original source...
    28. Sanh, V., Debut, L., Chaumond, J. & Wolf, T. 2019. DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter. ArXiv: 1910.01108v4. DOI: 10.48550/arXiv.1910.01108 Go to original source...
    29. Singh, G., Mittal, N. & Chouhan, S. S. 2022. A Systematic Review of Deep Learning Approaches for Natural Language Processing in Battery Materials Domain. IETE Technical Review, 39 (5), 1046-1057. DOI: 10.1080/02564602.2021.1984323 Go to original source...
    30. Sun, C., Qui, X., Xu, Y. & Huang, X. 2019. How to Fine-Tune BERT for Text Classification? ArXiv: 1905.05583v3. DOI: 10.48550/arXiv.1905.05583 Go to original source...
    31. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł. & Polosukhin, I. 2017. Attention Is All You Need. ArXiv: 1706.03762. DOI: 10.48550/arXiv.1706.03762 Go to original source...
    32. Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, Ł., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith, J., Riesa, J., Rudnick, A., Vinyals, O., Corrado, G., Hughes, M. & Dean, J. 2016. Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. ArXiv: 1609.08144. DOI: 10.48550/arXiv.1609.08144 Go to original source...
    33. Xu, Y. & Goodacre, R. 2018. On Splitting Training and Validation Set: A Comparative Study of Cross-Validation, Bootstrap and Systematic Sampling for Estimating the Generalization Performance of Supervised Learning. Journal of Analysis and Testing, 2, 249-262. DOI: 10.1007/s41664-018-0068-2 Go to original source...
    34. Zangari, A., Marcuzzo, M., Schiavinato, M., Gasparetto, A. & Albarelli, A. 2023. Ticket Automation: An Insight into Current Research with Applications to Multi-Level Classification Scenarios. Expert Systems with Applications, 225, 119984. DOI: 10.1016/j.eswa.2023.119984 Go to original source...
    35. Zhong, J. & Li, W. 2019. Predicting Customer Call Intent for the Auto Dealership Industry from Analyzing Phone Call Transcripts with CNN for Multi-Class Classification. International Journal on Soft Computing, Artificial Intelligence and Applications, 8 (3), 13-25. DOI: 10.5121/ijscai.2019.8302 Go to original source...
    36. Żak, K., Glavota, F., Mironica, I., Dinu, B., Marin, B., Vinca, F., Raducanu, I. & Tipau, A. 2021. GitHub - karolzak/support-tickets-classification/ [online]. Available at: https://github.com/karolzak/support-tickets-classification. [Accessed 2023, April 5].

    This is an open access article distributed under the terms of the Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0), which permits use, distribution, and reproduction in any medium, provided the original publication is properly cited. No use, distribution or reproduction is permitted which does not comply with these terms.


    Return to the content