Abstract
Dialogue acts (DAs) can represent conversational actions of tutors or students that take place during tutoring dialogues. Automating the identification of DAs in tutoring dialogues is significant to the design of dialogue-based intelligent tutoring systems. Many prior studies employ machine learning models to classify DAs in tutoring dialogues and invest much effort to optimize the classification accuracy by using limited amounts of training data (i.e., low-resource data scenario). However, beyond the classification accuracy, the robustness of the classifier is also important, which can reflect the capability of the classifier on learning the patterns from different class distributions. We note that many prior studies on classifying educational DAs employ cross entropy (CE) loss to optimize DA classifiers on low-resource data with imbalanced DA distribution. The DA classifiers in these studies tend to prioritize accuracy on the majority class at the expense of the minority class which might not be robust to the data with imbalanced ratios of different DA classes. To optimize the robustness of classifiers on imbalanced class distributions, we propose to optimize the performance of the DA classifier by maximizing the area under the ROC curve (AUC) score (i.e., AUC maximization). Through extensive experiments, our study provides evidence that (i) by maximizing AUC in the training process, the DA classifier achieves significant performance improvement compared to the CE approach under low-resource data, and (ii) AUC maximization approaches can improve the robustness of the DA classifier under different class imbalance ratios.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
FP, Positive Feedback “Well done!”, same abbreviation from [25].
- 2.
Our study has 31 dialogue acts as the classes to be classified.
References
Al-Luhaybi, M., Yousefi, L., Swift, S., Counsell, S., Tucker, A.: Predicting academic performance: a bootstrap** approach for learning dynamic Bayesian networks. In: Isotani, S., Millán, E., Ogan, A., Hastings, P., McLaren, B., Luckin, R. (eds.) AIED 2019. LNCS (LNAI), vol. 11625, pp. 26–36. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23204-7_3
Boyer, K., Ha, E.Y., Phillips, R., Wallis, M., Vouk, M., Lester, J.: Dialogue act modeling in a complex task-oriented domain. In: Proceedings of the SIGDIAL 2010 Conference, pp. 297–305 (2010)
Cavalcanti, A.P., et al.: How good is my feedback? A content analysis of written feedback. In: Proceedings of the LAK, LAK 2020, pp. 428–437. ACM, New York (2020)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT, Minneapolis, Minnesota, pp. 4171–4186. Association for Computational Linguistics (2019)
D’Mello, S., Olney, A., Person, N.: Mining collaborative patterns in tutorial dialogues. J. Educ. Data Min. 2(1), 1–37 (2010)
Du Boulay, B., Luckin, R.: Modelling human teaching tactics and strategies for tutoring systems: 14 years on. Int. J. Artif. Intell. Educ. 26(1), 393–404 (2016)
Ezen-Can, A., Boyer, K.E.: Understanding student language: an unsupervised dialogue act classification approach. J. Educ. Data Min. 7(1), 51–78 (2015)
Ezen-Can, A., Grafsgaard, J.F., Lester, J.C., Boyer, K.E.: Classifying student dialogue acts with multimodal learning analytics. In: Proceedings of the Fifth LAK, pp. 280–289 (2015)
Lin, J., et al.: Is it a good move? Mining effective tutoring strategies from human–human tutorial dialogues. Futur. Gener. Comput. Syst. 127, 194–207 (2022)
Lin, J., et al.: Enhancing educational dialogue act classification with discourse context and sample informativeness. IEEE TLT (under reviewing process)
Min, W., et al.: Predicting dialogue acts for intelligent virtual agents with multimodal student interaction data. Int. Educ. Data Min. Soc. (2016)
Nguyen, N.D., Tan, W., Buntine, W., Beare, R., Chen, C., Du, L.: AUC maximization for low-resource named entity recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence (2023)
Nye, B.D., Graesser, A.C., Hu, X.: Autotutor and family: a review of 17 years of natural language tutoring. Int. J. Artif. Intell. Educ. 24(4), 427–469 (2014)
Nye, B.D., Morrison, D.M., Samei, B.: Automated session-quality assessment for human tutoring based on expert ratings of tutoring success. Int. Educ. Data Min. Soc. (2015)
Raković, M., et al.: Towards the automated evaluation of legal casenote essays. In: Rodrigo, M.M., Matsuda, N., Cristea, A.I., Dimitrova, V. (eds.) AIED 2022. LNCS, vol. 13355, pp. 167–179. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-11644-5_14
Rus, V., Maharjan, N., Banjade, R.: Dialogue act classification in human-to-human tutorial dialogues. In: Innovations in Smart Learning. LNET, pp. 183–186. Springer, Singapore (2017). https://doi.org/10.1007/978-981-10-2419-1_25
Rus, V., et al.: An analysis of human tutors’ actions in tutorial dialogues. In: The Thirtieth International Flairs Conference (2017)
Samei, B., Li, H., Keshtkar, F., Rus, V., Graesser, A.C.: Context-based speech act classification in intelligent tutoring systems. In: Trausan-Matu, S., Boyer, K.E., Crosby, M., Panourgia, K. (eds.) ITS 2014. LNCS, vol. 8474, pp. 236–241. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07221-0_28
Samei, B., Rus, V., Nye, B., Morrison, D.M.: Hierarchical dialogue act classification in online tutoring sessions. In: EDM, pp. 600–601 (2015)
Sha, L., et al.: Is the latest the greatest? A comparative study of automatic approaches for classifying educational forum posts. IEEE Trans. Learn. Technol. 1–14 (2022)
Tan, W., Du, L., Buntine, W.: Diversity enhanced active learning with strictly proper scoring rules. In: Advances in Neural Information Processing Systems, vol. 34, pp. 10906–10918 (2021)
Tan, W., et al.: Does informativeness matter? Active learning for educational dialogue act classification. In: Wang, N., et al. (eds.) AIED 2023. LNAI, vol. 13916, pp. 176–188. Springer, Cham (2023)
Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., Schmidt, L.: Measuring robustness to natural distribution shifts in image classification. In: Advances in Neural Information Processing Systems, vol. 33, pp. 18583–18599 (2020)
Vail, A.K., Grafsgaard, J.F., Boyer, K.E., Wiebe, E.N., Lester, J.C.: Predicting learning from student affective response to tutor questions. In: Micarelli, A., Stamper, J., Panourgia, K. (eds.) ITS 2016. LNCS, vol. 9684, pp. 154–164. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-39583-8_15
Vail, A.K., Boyer, K.E.: Identifying effective moves in tutoring: on the refinement of dialogue act annotation schemes. In: Trausan-Matu, S., Boyer, K.E., Crosby, M., Panourgia, K. (eds.) ITS 2014. LNCS, vol. 8474, pp. 199–209. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07221-0_24
VanLehn, K., Graesser, A.C., Jackson, G.T., Jordan, P., Olney, A., Rosé, C.P.: When are tutorial dialogues more effective than reading? Cogn. Sci. 31(1), 3–62 (2007)
Yang, T., Ying, Y.: AUC maximization in the era of big data and AI: a survey. ACM Comput. Surv. 55(8) (2022). https://doi.org/10.1145/3554729
Ying, Y., Wen, L., Lyu, S.: Stochastic online AUC maximization. In: Advances in Neural Information Processing Systems, vol. 29 (2016)
Yuan, Z., Yan, Y., Sonka, M., Yang, T.: Large-scale robust deep AUC maximization: a new surrogate loss and empirical studies on medical image classification. In: 2021 IEEE/CVF ICCV, Los Alamitos, CA, USA, pp. 3020–3029. IEEE Computer Society (2021)
Yuan, Z., Guo, Z., Chawla, N., Yang, T.: Compositional training for end-to-end deep AUC maximization. In: International Conference on Learning Representations (2022). https://openreview.net/forum?id=gPvB4pdu_Z
Zhao, L., et al.: METS: multimodal learning analytics of embodied teamwork learning. In: LAK23: 13th International Learning Analytics and Knowledge Conference, pp. 186–196 (2023)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Lin, J. et al. (2023). Robust Educational Dialogue Act Classifiers with Low-Resource and Imbalanced Datasets. In: Wang, N., Rebolledo-Mendez, G., Matsuda, N., Santos, O.C., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2023. Lecture Notes in Computer Science(), vol 13916. Springer, Cham. https://doi.org/10.1007/978-3-031-36272-9_10
Download citation
DOI: https://doi.org/10.1007/978-3-031-36272-9_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-36271-2
Online ISBN: 978-3-031-36272-9
eBook Packages: Computer ScienceComputer Science (R0)