Abstract
Learning analytics dashboards have been developed to facilitate teacher guidance in computer-supported collaborative learning (CSCL). As yet, little is known about how teachers interpret dashboard information to facilitate guidance in their teaching practice. This study examined teachers’ behavior patterns in interpreting information from dashboards, and obtained their views about the potential barriers in interpreting dashboard information. Fourteen pre-service teachers participated in the study and data were collected from multiple sources. In total, 1,346 min of video data on teachers’ guiding behavior and approximately 27,000 words from a cued retrospective report and interview data were generated. A two-stage approach was adopted to process the data. Based on the video analysis in the first stage, we extracted teachers’ four typical behavior patterns in finding and reading dashboard information and two behavior patterns when explaining information from dashboards. Thematic analysis at the second stage identified useful indicators for teacher guidance in CSCL and some major barriers teachers encountered in interpreting information. These findings may help improve the design of dashboards and show how teachers integrate dashboards into their daily teaching practice, thereby enhancing students’ collaboration and learning.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Asterhan, C. S. C. (2011). Assessing e-moderation behavior from synchronous discussion protocols with a multi-dimensional methodology. Computers in Human Behavior, 27(1), 449–458. https://doi.org/10.1016/j.chb.2010.09.008
Asterhan, C. S. C., Schwarz, B. B., & Gil, J. (2012). Small-group, computer-mediated argumentation in middle-school classrooms: The effects of gender and different types of online teacher guidance. British Journal of Educational Psychology, 82(3), 375–397. https://doi.org/10.1111/j.2044-8279.2011.02030.x
Bao, H. G., Li, Y. Y., Su, Y., **ng, S., Chen, N. S., & Rose, C. P. (2021). The effects of a learning analytics dashboard on the performance of teachers’ diagnosis and intervention in computer-supported collaborative learning. Technology, Pedagogy and Education, 30(2), 287–303. https://doi.org/10.1080/1475939X.2021.1902383
Bodily, R., Ikahihifo, T. K., Mackley, B., & Graham, C. R. (2018). The design, development, and implementation of student-facing learning analytics dashboards. Journal of Computing in Higher Education, 30(3), 572–598. https://doi.org/10.1007/s12528-018-9186-0
Boyatzis, R. E. (1998). Transforming qualitative information: Thematic analysis and code development. Thousand Oaks, CA: Sage Publications
Brown, M. (2020). Seeing students at scale: How faculty in large lecture courses act upon learning analytics dashboard data. Teaching in Higher Education, 25(4), 384–400. https://doi.org/10.1080/13562517.2019.1698540
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Chen, J., Wang, M., Kirschner, P. A., & Tsai, C. C. (2018). The role of collaboration, computer use, learning environments, and supporting strategies in CSCL: A meta-analysis. Review of Educational Research, 88(6), 799–843. https://doi.org/10.3102/0034654318791584
Chounta, I. A., & Avouris, N. (2016). Towards the real-time evaluation of collaborative activities: Integration of an automatic rater of collaboration quality in the classroom from the teacher’s perspective. Education and Information Technologies, 21(4), 815–835. https://doi.org/10.1007/s10639-014-9355-3
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational & Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104
Dix, A., & Leavesley, J. (2015). Learning analytics for the academic: An action perspective. Journal of Universal Computer Science, 21(1), 48–65. https://doi.org/10.3217/jucs-021-01-0048
Galleta, A., & Cross, W. E. (2013). Conducting the interview: The role of reciprocity and reflexivity. In A. Galleta (Ed.), Mastering the semi-structured interview and beyond: From research design to analysis and publication (pp. 100–125). Oxford: Oxford University Press
Gillies, R. M., Ashman, A., & Terwel, J. (Eds.). (2008). The teacher’s role in implementing cooperative learning in the classroom. New York, NY: Springer
Gross, C., Stary, & A Totter. (2005). User centered awareness in computer-supported cooperative work-systems. Computers in Human Behavior, 18(3), 323–360. https://doi.org/10.1207/s15327590ijhc1803_5
Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., & Hlosta, M. (2019). A large-scale implementation of predictive learning analytics in higher education: The teachers’ role and perspective. Educational Technology Research and Development, 67(5), 1273–1306. https://doi.org/10.1007/s11423-019-09685-0
Hoogland, I., Schildkamp, K., van der Kleij, F., Heitink, M., Kippers, W., Veldkamp, B., & Dijkstra, A. M. (2016). Prerequisites for data-based decision making in the classroom: Research evidence and practical illustrations. Teaching and Teacher Education, 60, 377–386. https://doi.org/10.1016/j.tate.2016.07.012
Janssen, J., & Bodemer, D. (2013). Coordinated computer-supported collaborative learning: Awareness and awareness tools. Educational Psychologist, 48(1), 40–55. https://doi.org/10.1080/00461520.2012.749153
Jivet, I., Scheffel, M., Specht, & Drachsler, H. (2018). License to evaluate: Preparing learning analytics dashboards for educational practice, in Proceedings of the 8th International Conference on Learning Analytics and Knowledge, pp. 31–40
Joyce-Gibbons, A. (2017). Observe, interact and act: Teachers’ initiation of mini-plenaries to scaffold small-group collaboration. Technology, Pedagogy and Education, 26(1), 51–68. https://doi.org/10.1080/1475939X.2016.1173089
Kaendler, C., Wiedmann, M., Leuders, T., Rummel, N., & Spada, H. (2016). Monitoring student interaction during collaborative learning: Design and evaluation of a training program for pre-service teachers. Psychology Learning & Teaching, 15(1), 44–64. https://doi.org/10.1177/1475725716638010
Klein, C., Lester, J., Rangwala, H., & Johri, A. (2019). Technological barriers and incentives to learning analytics adoption in higher education: Insights from users. Journal of Computing in Higher Education, 31(3), 604–625. https://doi.org/10.1007/s12528-019-09210-5
Landis, J. L., & Koch, G. G. (1977). An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics, 33(2), 363–374. https://doi.org/10.2307/2529786
Matcha, W., Uzir, N. A., Gasevic, D., & Pardo, A. (2020). A systematic review of empirical studies on learning analytics dashboards: A self-regulated learning perspective. IEEE Transactions on Learning Technologies, 13(2), 226–245. https://doi.org/10.1109/TLT.2019.2916802
Martinez-Maldonado, R. (2019). A handheld classroom dashboard: Teachers’ perspectives on the use of real-time collaborative learning analytics. International Journal of Computer-Supported Collaborative Learning, 14(2), 1–29. https://doi.org/10.1007/s11412-019-09308-z
Molenaar, I., & Knoop-van Campen, C. A. N (2019). How teachers make dashboard information actionable. IEEE Transactions on Learning Technologies, 12(3), 347–355. https://doi.org/10.1109/TLT.2018.2851585
Rummel, N. (2018). One framework to rule them all? Carrying forward the conversation started by Wise and Schwarz. International Journal of Computer-Supported Collaborative Learning, 13(1), 123–129. https://doi.org/10.1007/s11412-018-9273-2
Schwendimann, B. A., Rodriguez-Triana, M. J., Vozniuk, A., Prieto, L. P., Boroujeni, M. S., Holzer, A. … Dillenbourg, P. (2017). Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies, 10(1), 30–41. https://doi.org/10.1109/TLT.2016.2599522
Schwarz, B. B., & Asterhan, C. A. (2011). E-Moderation of synchronous discussions in educational settings: A nascent practice. Journal of the Learning Sciences, 20(3), 395–442. https://doi.org/10.1080/10508406.2011.553257
Sergis, S., & Sampson, D. G. (2017). Teaching and learning analytics to support teacher inquiry: A systematic literature review. In A. Peña-Ayala (Ed.), Learning analytics: Fundaments, applications, and trends (pp. 25–63). New York: Springer
Sedrakyan, G., Mannens, E., & Verbert, K. (2019). Guiding the choice of learning dashboard visualizations: Linking dashboard design and data visualization concepts. Journal of Visual Languages & Computing, 50, 19–38. https://doi.org/10.1016/j.jvlc.2018.11.002
Shibani, A., Knight, S., & Buckingham Shum, S. (2020). Educator perspectives on learning analytics in classroom practice. The Internet and Higher Education, 46, 1–14. https://doi.org/10.1016/j.iheduc.2020.100730
Siemens, G. (2013). Learning analytics: The emergence of a discipline. American Behavioral Scientist, 57(10), 1380–1400. https://doi.org/10.1177/0002764213498851
Tissenbaum, M., & Slotta, J. (2019). Supporting classroom orchestration with real-time feedback: A role for teacher dashboards and real-time agents. International Journal of Computer-Supported Collaborative Learning, 14(7), 1–27. https://doi.org/10.1007/s11412-019-09306-1
Van den Bergh, L., Ros, A., & Beijaard, D. (2013). Teacher feedback during active learning: Current practices in primary schools. British Journal of Educational Psychology, 83(2), 341–362. https://doi.org/10.1111/j.2044-8279.2012.02073.x
Van Es, E. A., & Sherin, M. G. (2002). Learning to notice: Scaffolding new teachers’ interpretations of classroom interactions. Journal of Technology and Teacher Education, 10(4), 571–596. https://www.learntechlib.org/primary/p/9171/
Van Gog, T., Paas, F., van Merriënboer, J. J. G., & Witte, P. (2005). Uncovering the problem-solving process: Cued retrospective reporting versus concurrent and retrospective reporting. Journal of Experimental Psychology: Applied, 11(4), 237–244. https://doi.org/10.1037/1076-898X.11.4.237
Van Leeuwen, A. (2019). Teachers’ perceptions of the usability of learning analytics reports in a flipped university course: When and how does information become actionable knowledge? Educational Technology Research and Development, 67(5), 1043–1064. https://doi.org/10.1007/s11423-018-09639-y
Van Leeuwen, A., & Janssen, J. (2019). A systematic review of teacher guidance during collaborative learning in primary and secondary education. Educational Research Review, 27, 71–89. https://doi.org/10.1016/j.edurev.2019.02.001
Van Leeuwen, A., Janssen, J., Erkens, G., & Brekelmans, M. (2013). Teacher interventions in a synchronous, co-located CSCL setting: Analyzing focus, means, and temporality. Computers in Human Behavior, 29(4), 1377–1386. https://doi.org/10.1016/j.chb.2013.01.028
Van Leeuwen, A., Janssen, J., Erkens, G., & Brekelmans, M. (2014). Supporting teachers in guiding collaborating students: Effects of learning analytics in CSCL. Computers & Education, 79, 28–39. https://doi.org/10.1016/j.compedu.2014.07.007
Van Leeuwen, A., Janssen, J., Erkens, G., & Brekelmans, M. (2015a). Teacher regulation of cognitive activities during student collaboration: Effects of learning analytics. Computers & Education, 90, 80–94. https://doi.org/10.1016/j.compedu.2015.09.006
Van Leeuwen, A., Janssen, J., Erkens, G., & Brekelmans, M. (2015b). Teacher regulation of multiple computer-supported collaborating groups. Computers in Human Behavior, 52, 233–242. https://doi.org/10.1016/j.chb.2015.05.058
Van Leeuwen, A., & Rummel, N. (2020). Comparing teachers’ use of mirroring and advising dashboards. In Proceedings of the 10th International Conference on Learning Analytics & Knowledge (pp. 26–34), New York, USA: ACM
Van Leeuwen, A., Rummel, N., & van Gog, T. (2019). What information should CSCL teacher dashboards provide to help teachers interpret CSCL situations? International Journal of Computer-Supported Collaborative Learning, 14(3), 261–289. https://doi.org/10.1007/s11412-019-09299-x
Van Leeuwen, A., van Wermeskerken, M., Erkens, G., & Rummel, N. (2017). Measuring teacher sense making strategies of learning analytics: A case study. Learning: Research and Practice, 3(1), 42–58. https://doi.org/10.1080/23735082.2017.1284252
Vieira, C., Parsons, P., & Byrd, V. (2018). Visual learning analytics of educational data: A systematic literature review and research agenda. Computers & Education, 122, 119–135. https://doi.org/10.1016/j.compedu.2018.03.018
Wise, A. F., & Jung, Y. (2019). Teaching with analytics: Towards a situated model of instructional decision-making. Journal of Learning Analytics, 6(2), 53–69. https://doi.org/10.18608/jla.2019.62.4
Zheng, Y., Xu, C., Li, Y., & Su, Y. (2018). Measuring and visualizing group knowledge elaboration in online collaborative discussions. Educational Technology & Society, 21(1), 91–103. https://www.jstor.org/stable/26273871
Acknowledgements
The authors would like to thank all the anonymous reviewers for their enlightening comments and suggestions. We also thank Michelle Pascoe, PhD, from Liwen Bianji (Edanz) (www.liwenbianji.cn/) for editing the English text of a draft of this manuscript.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Disclosure of potential conflicts of interest
This work was supported by National Natural Science Foundation of China (Grant numbers: 61,877,003), the International Joint Research Project of Faculty of Education, Bei**g Normal University, and National Natural Science Foundation of China (62,107,003). The authors declare that they have no conflict of interest.
Research involving human participants
The procedures used in this study adhere to the tenets of the Declaration of Helsinki. This research involves no more than minimal risk to the research subjects.
Informed consent
All individual participants are more than 16 years old. Informed consent was obtained from all individual participants included in the study. Verbal informed consent was also obtained prior to the interview.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Li, Y., Zhang, M., Su, Y. et al. Examining teachers’ behavior patterns in and perceptions of using teacher dashboards for facilitating guidance in CSCL. Education Tech Research Dev 70, 1035–1058 (2022). https://doi.org/10.1007/s11423-022-10102-2
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11423-022-10102-2