SpringerLink
Log in

High School English Teachers Reflect on Their Talk: A Study of Response to Automated Feedback with the Teacher Talk Tool

  • ARTICLE
  • Published:
International Journal of Artificial Intelligence in Education Aims and scope Submit manuscript

Abstract

We present the Teacher Talk Tool, which automatically analyzes classroom audio and provides formative feedback on key aspects of teachers’ classroom discourse (e.g., use of open-ended questions). The tool was designed to promote teacher learning by focusing attention and sense-making on their discourse. We conducted a feedback-response study where five English & Language Art teachers used the Teacher Talk Tool in eight classroom sessions. Teachers completed repeated-measure surveys and semi-structured interviews providing quantitative and qualitative evidence of feedback response. Results indicated that the majority of automated feedback was perceived to be accurate and prompted a high degree of reflection, focusing teachers’ attention on the measured talk constructs. This feedback also led teachers to engage in a process of sense-making, linking the measured talk features to classroom processes and contexts. However, evidence of feedback uptake was more limited. Overall, results contribute to the nascent literature on the efficacy of automated feedback on instructional practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Canada)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Notes

  1. The robustness with which uptake is identified may depend strongly on the specific definition and context of uptake. In our prior work in English language arts contexts, we employ Nystrand and Gamoran’s very strict definition of uptake. Other researchers have had success coding more expansive treatments of uptake in computer science classrooms (Demszky et al., 2023).

  2. The studies cited here are focused more on the determinants of trust than characterizing the central tendency in trust, which would be aided by a frame of reference (i.e., common measures used with respondents in multiple occupations) not present in the data.

  3. In the case of the Teacher Talk Tool the feedback did entail a comparison (to normative data), but because the talk constructs are not defined with reference to effectiveness, participants were not forced to infer any given comparative score was “good” or “bad.” There are also some basic system differences to Jacobs et al. (2022): the features themselves are different, the audio recording systems are very different, and Jacobs et al. (2022) provided feedback on a continuous (0–100%) scale. Differences in the underlying computational models and their validation are not discussed here.

  4. The first author participated in the Nystrand and Gamoran studies beginning with the national or five-state study (Gamoran & Kelly, 2003), and then the Partnership for Literacy Study (Kelly, 2008).

  5. Models were trained and evaluated using tenfold teacher-level cross-validation, where all utterances for a given teacher were either in the training set or the testing set, but never in both. If a teacher in the present study also contributed data used to train the models (prior data collection), the models were retrained after removing utterances from that teacher prior to use in the current study. In this fashion, there was no data overlap between model development (training) and deployment.

  6. The first participant received labels based on 15–85 cut points, which we quickly realized obfuscated far too much important variation in talk features.

  7. Efficacy items were newly developed for this study based on learning/standards goals from the Common Core State Standards for English language arts and Literacy. Although the items appear highly internally consistent (Cronbach’s alpha of above .9), various survey response processes (e.g., adjacency effects) can artificially inflate such statistics. The mean of the efficacy items was 3.33 at the start of data collection (on a 4-point scale), increasing to 3.64 at the end of the study.

  8. Accuracies were reported like this in two places: in the initial overview presentation to teachers, and in information screens on the Webapp. When the system was switched to tercile cut points for low, medium, and high scores, we failed to correctly update the cutoff-based accuracies reported to users (i.e., we continued to use the accuracies corresponding to original 15/85 cut points). This error notwithstanding, we are confident users were well-apprised that the system is not fully accurate.

  9. We designate this statistic as informal because the ICC in such small samples is readily impacted by chance/random differences across teachers.

  10. Quotes included in results here have been corrected for word substitutions and various errors.

  11. The instructional talk measure could be very useful in other contexts, such as making more highly aggregated appraisals (e.g., across schools or districts), or in larger scale studies where the tails of the distribution would be relevant.

  12. This quote illustrates the depth of Erin’s puzzlement over the instructional talk feature but also fundamental misunderstanding of the features; instructional talk is estimated, by definition, orthogonally to the other features. The inter-relationships among features is understandably challenging for a new user to understand.

  13. We experimented with that in this study in the second interview, and users generally responded positively to specific examples.

References

  • Ahuja, K., Kim, D., Xhakaj, F., Varga, V., **e, A., Zhang, S., Townsend, J. E., Harrison, C., Ogan, A., & Agarwal, Y. (2019). EduSense: Practical classroom sensing at scale. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 3(3), 1–26.

    Article  Google Scholar 

  • Alic, S., Demszky, D., Mancenido, Z., Liu, J., Hill, H., & Jurafsky, D. (2022). Computationally identifying funneling and focusing questions in classroom discourse, ar**v preprint ar**v:2208.04715 .

  • Archer, J., Cantrell, S., Holtzman, S. L., Joe, J. N., Tocci, C. M., & Wood, J. (2016). Better feedback for better teaching: A practical guide to improving classroom observations. John Wiley & Sons.

    Google Scholar 

  • Aroyo, A. M., De Bruyne, J., Dheu, O., Fosch-Villaronga, E., Gudkov, A., Hoch, H., ... & Tamò-Larrieux, A. (2021). Overtrusting robots: Setting a research agenda to mitigate overtrust in automation. Paladyn, Journal of Behavioral Robotics, 12, 423–436.

  • Aucejo, E., Coate, P., Fruehwirth, J. C., Kelly, S., & Mozenter, Z. (2022). Teacher effectiveness and classroom composition: Understanding match effects in the classroom. The Economic Journal, 132, 3047–3064.

    Article  Google Scholar 

  • Azevedo, R., & Bernard, R. M. (1995). A meta-analysis of the effects of feedback in computer-based instruction. Journal of Educational Computing Research, 13(2), 111–127.

    Article  Google Scholar 

  • Bell, C. A., Qi, Y., Croft, A. J., Leusner, D., McCaffrey, D. F., Gitomer, D. H., & Pianta, R. C. (2014). Improving observational score quality. In T. Kane, K. Kerr, & R. Pianta (Eds.), Designing teacher evaluation systems: New guidance from the measures of effective teaching project (pp. 50–97). Jossey-Bass.

    Google Scholar 

  • Blanchard, N., Brady, M., Olney, A., Glaus, M., Sun, X., Nystrand, M., Samei, B., Kelly, S., & D’Mello, S. K. (2015). A study of automatic speech recognition in noisy classroom environments for automated dialog analysis. In C. Conati, N. Heffernan, A. Mitrovic, & M. F. Verdejo (Eds.), Proceedings of the 17th international conference on artificial intelligence in education (AIED 2015) (pp. 23–33). Springer-Verlag.

  • Brett, J. F., & Atwater, L. E. (2001). 360° feedback: Accuracy, reactions, and perceptions of usefulness. Journal of Applied Psychology, 86, 930–942.

    Article  Google Scholar 

  • Camburn, E. M. (2010). Embedded teacher learning opportunities as a site for reflective practice: An exploratory study. American Journal of Education, 116, 463–489.

    Article  Google Scholar 

  • Camburn, E. M., & Han, S. W. (2015). Infrastructure for teacher reflection and instructional change: An exploratory study. Journal of Educational Change, 16, 511–533.

    Article  Google Scholar 

  • Campbell, S. L., & Ronfeldt, M. (2018). Observational evaluation of teachers: Measuring more than we bargained for? American Educational Research Journal, 55, 1233–1267.

    Article  Google Scholar 

  • Cao, J., Ganesh, A., Cai, J., Southwell, R., Perkoff, M., Regan, M., Kann, K., Martin, J., Palmer, M., & D’Mello, S. K. (2023). A comparative analysis of automatic speech recognition errors in small group classroom discourse. In Proceedings of the ACM International Conference on User Modeling, Adaptation and Personalization (UMAP 2023) (pp. 250–262). ACM.

  • Caughlan, S., Juzwik, M. M., Borsheim-Black, C., Kelly, S., & Fine, J. G. (2013). English teacher candidates develo** dialogically organized instructional practices. Research in the Teaching of English, 47, 212–246.

    Article  Google Scholar 

  • Chawla, N., Gabriel, A. S., da Motta Veiga, S. P., & JSlaughter, J. E. (2019). Does feedback matter for job search self-regulation? It depends on feedback quality. Personnel Psychology, 72, 513–541.

    Article  Google Scholar 

  • Chen, G., Chan, C. K. K., Chan, K. K. H., Clarke, S. N., & Resnick, L. B. (2020). Efficacy of video-based teacher professional development for increasing classroom discourse and student learning. Journal of the Learning Sciences, 29, 642–680.

    Article  Google Scholar 

  • Cherasaro, T. L., Brodersen, R. M., Reale, M. L., & Yanoski, D. C. (2016). Teachers’ responses to feedback from evaluators: What feedback characteristics matter? (REL 2017–190). Regional Educational Laboratory Central.

  • Chiu, J. L., Bywater, J. P., & Lilly, S. (2022). The role of AI to support teacher learning and practice: A review and future directions. In F. Ouyang, P. Jiao, B. McLaren, & A. Alavi (Eds.), Artificial intelligence in STEM education: The paradigmatic shifts in research, education, and technology (pp. 163–173). CRC Press.

    Chapter  Google Scholar 

  • Clarke, D., & Hollingsworth, H. (2002). Elaborating a model of teacher professional growth. Teaching and Teacher Education, 18, 947–967.

    Article  Google Scholar 

  • Close, K., Amrein-Beardsley, A., & Collins, C. (2018). State-level assessments and teacher evaluation systems after the passage of the every student succeeds act: Some steps in the right direction. National Education Policy Center.

    Google Scholar 

  • Cohen, J., & Goldhaber, D. (2016). Building a more complete understanding of teacher evaluation using classroom observations. Educational Researcher, 45, 378–387.

    Article  Google Scholar 

  • Colestock, A., & Sherin, M. G. (2009). Teachers’ sense-making strategies while watching video of mathematics instruction. Journal of Technology and Teacher Education, 17, 7–29.

    Google Scholar 

  • d’Anjou, B., Bakker, S., An, P., & Bekker, T. (2019). How peripheral data visualisation systems support secondary school teachers during VLE-supported lessons. In Proceedings of the 2019 on designing interactive systems conference (pp. 859–870).

  • D’Mello, S. K., Lehman, B., & Person, N. (2010). Expert tutors feedback is immediate, direct, and discriminating. In C. Murray & H. Guesgen (Eds.), Proceedings of the 23rd Florida Artificial Intelligence Research Society Conference (pp. 595–560). AAAI Press.

  • D’Mello, S. K., Olney, A. M, Blanchard, N., Sun, X., Ward, B., Samei, B., & Kelly, S. (2015). Multimodal capture of teacher-student interactions for automated dialogic analysis in live classrooms. Proceedings of the 17th ACM International Conference on Multimodal Interaction (ICMI 2015) (Multimodal Learning Analytics Grand Challenge MLA’15). (pp. 557–566). ACM.

  • Dale, M., Godley, A., Capello, S., Donnelly, P., D’Mello, S., & Kelly, S. (2022). Toward the automated analysis of teacher talk in secondary ELA classrooms. Teaching and Teacher Education, 110, 103584.

    Article  Google Scholar 

  • Datta, D., Bywater, J. P., Phillips, M., Lilly, S., Chiu, J. L., Watson, G. S., & Brown, D. E. (2023). Classifying mathematics teacher questions to support mathematical discourse. In International Conference on Artificial Intelligence in Education (pp. 372–377). Springer Nature Switzerland.

  • Demszky, D., Liu, J., Hill, H. C., Jurafsky, D., & Piech, C. (2023). Can automated feedback improve teachers’ uptake of student ideas? Evidence from a randomized controlled trial in a large-scale online course. Educational Evaluation and Policy Analysis. https://doi.org/10.3102/01623737231169270

  • Demszky, D. (2022). Using natural language processing to support student-centered education, Doctoral dissertation, Stanford University.

  • Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. ar**v, ar**v:1810.04805 .

  • Dwyer, C. A., & Stufflebeam, D. S. (1996). Evaluation for effective teaching. In D. Berliner & R. Calfee (Eds.), Handbook of research in educational psychology. Macmillan.

    Google Scholar 

  • Ericsson, K. A., Krampe, R., & Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100(3), 363.

    Article  Google Scholar 

  • Fadde, P. J., & Klein, G. A. (2010). Deliberate performance: Accelerating expertise in natural settings. Performance Improvement, 49(9), 5–14.

    Article  Google Scholar 

  • Gamoran, A., & Nystrand, M. (1992). Taking students seriously. In F. Newmann (Ed.), Student engagement and achievement in American secondary schools. Teachers College Press.

    Google Scholar 

  • Gamoran, A., Nystrand, M., Berends, M., & Lepore, P. C. (1995). An organizational analysis of the effects of ability grou**. American Educational Research Journal, 32, 687–715.

    Article  Google Scholar 

  • Gamoran, A., & Kelly, S. (2003) Tracking, instruction, and unequal literacy in secondary school English. In M. T. Hallinan, A. Gamoran, W. Kubitschek, and T. Loveless (Eds.), Stability and Change in American Education: Structure, Processes and Outcomes (pp. 109–126). Eliot Werner Publications.

  • Gerard, L., Wiley, K., Bradford, A., Chen, J. K., Lim-Breitbart, J., & Linn, M. (2020). Impact of a teacher action planner that captures student ideas on teacher customization decisions. In Proceedings of the 14th international society for learning sciences conference (pp. 2077–2084).

  • Gitomer, D. H., Bell, C. A., Qi, Y., McCaffrey, D. F., Hamre, B. K., & Pianta, R. C. (2014). The instructional challenge in improving teaching quality: Lessons from a classroom observation protocol. Teachers College Record, 116(6), 1–32.

    Article  Google Scholar 

  • Grissom, J. A., Blissett, R. S., & Mitani, H. (2018). Evaluating school principals: Supervisor ratings of principal practice and principal job performance. Educational Evaluation and Policy Analysis, 40(3), 446–472.

    Article  Google Scholar 

  • Grossman, P., Loeb, S., Cohen, J., & Wyckoff, J. (2013). Measure for measure: The relationship between measures of instructional practice in middle school English language arts and teachers’ value-added scores. American Journal of Education, 19, 45–470.

    Google Scholar 

  • Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61, 29–48.

    Article  MathSciNet  Google Scholar 

  • Hennessy, S., Howe, C., Mercer, N., & Vrikki, M. (2020). Coding classroom dialogue: Methodological considerations for researchers. Learning, Culture, and Social Interaction, 25, 100404.

    Article  Google Scholar 

  • Ho, A. D., & Kane, T. J. (2013). The reliability of classroom observations by school personnel. (Tech. Rep.). Bill & Melinda Gates Foundation, Measures of Effective Teaching Project.

  • Hoff, K. A., & Bashir, M. (2015). Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors, 57, 407–434.

    Article  Google Scholar 

  • Huang, G. Y., Chen, J., Liu, H., Fu, W., Ding, W., Tang, J., ... & Liu, Z. (2020). Neural multi-task learning for teacher question detection in online classrooms. In Artificial Intelligence in Education: 21st International Conference, AIED 2020, Ifrane, Morocco, July 6–10, 2020, Proceedings, Part I 21 (pp. 269–281). Springer International Publishing.

  • Humphry, S. M., & Heldsinger, S. A. (2014). Common structural design features of rubrics may represent a threat to validity. Educational Researcher, 43, 253–263.

    Article  Google Scholar 

  • Jacobs, J., Scornavacco, K., Harty, C., Suresh, A., Lai, V., & Sumner, T. (2022). Promoting rich discussion in mathematics classrooms: Using personalized automated feedback to support reflection and instructional change. Teaching and Teacher Education, 112, 103611.

    Article  Google Scholar 

  • Jensen, E., Dale, M., Donnelly, P. J., Stone, C., Kelly, S., Godley, A., & S. K. D’Mello. (2020). Toward automated feedback on teacher discourse to enhance teaching effectiveness. Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (CHI 2020): Association for Computing Machinery. pp 1–13.

  • Jensen, E., Pugh, S., & D’Mello, S. K. (2021). A deep transfer learning approach to automated teacher discourse feedback. In Proceedings of the 11th Learning Analytics & Knowledge Conference (LAK 2021). ACM.

  • Kelly, S. (2007). Classroom discourse and the distribution of student engagement. Social Psychology of Education, 10, 331–352.

    Article  Google Scholar 

  • Kelly, S. (2008). Race, social class, and student engagement in middle school English classrooms. Social Science Research, 37, 434–448.

    Article  Google Scholar 

  • Kelly, S. (2023). Agnosticism in instructional observation systems. Education Policy Analysis Archives, 31(7). https://doi.org/10.14507/epaa.31.7493

  • Kelly, S., & Abruzzo, E. (2021). Using lesson-specific teacher reports of student engagement to investigate innovations in curriculum and instruction. Educational Researcher, 50, 306–314.

    Article  Google Scholar 

  • Kelly, S., Olney, A. M., Donnelly, P., Nystrand, M., & D’Mello, S. K. (2018). Automatically measuring question authenticity in real-world classrooms. Educational Researcher, 47, 451–464.

    Article  Google Scholar 

  • Kelly, S., Bringe, R., Aucejo, E., & Fruehwirth, J. (2020a). Using global observation protocols to inform research on teaching effectiveness and school improvement: Strengths and emerging limitations. Education Policy Analysis Archives, 28, 62.

    Article  Google Scholar 

  • Kelly, S., Mozenter, Z., Aucejo, E., & Fruehwirth, J. (2020b). School-to-school differences in instructional practice: New descriptive evidence on opportunity to learn. Teachers College Record, 122(11), 1–38.

    Article  Google Scholar 

  • Klette, K., Blikstad-Balas, M., & Roe, A. (2017). Linking instruction and student achievement. Acta Didactica, 11(3), 10.

    Article  Google Scholar 

  • Korban, M., Youngs, P., & Acton, S. T. (2023). A Multi-Modal Transformer network for action detection. Pattern Recognition, 142, 109713.

    Article  Google Scholar 

  • Kraft, M. A., & Christian, A. (2019). In search of high-quality evaluation feedback: An administrator training field experiment. Ed-Working Paper 19–62, Annenberg Institute at Brown University, Providence, RI.

  • Kraft, M. A., & Gilmour, A. F. (2016). Can principals promote teacher development as evaluators? A case study of principals’ views and experiences. Educational Administration Quarterly, 52, 711–753.

    Article  Google Scholar 

  • Kraft, M. A., & Novicoff, S. (2024). Time in school: A conceptual framework, synthesis of the causal research, and empirical exploration. American Educational Research Journal, 0(0). https://doi.org/10.3102/00028312241251857

  • Langer, J. A. (2001). Beating the odds: Teaching middle and high school students to read and write well. American Educational Research Journal, 38, 837–880.

    Article  Google Scholar 

  • Liu, S., Bell, C. A., Jones, N. D., & McCaffrey, D. F. (2019). Classroom observation systems in context: A case for the validation of observation systems. Educational Assessment, Evaluation, and Accountability, 31, 61–95.

    Article  Google Scholar 

  • Logg, J. M., Minson, J. A., & Moore, D. A. (2019). Algorithmic appreciation: People prefer algorithmic to human judgement. Organizational Behavior and Human Decision Processes, 151, 90–103.

    Article  Google Scholar 

  • Lugini, L., Litman, D., Godley, A., & Olshefski, C. (2019). Annotating student talk in text-based classroom discussions, ar**v preprint ar**v:1909.03023.

  • McCaffrey, D. F., Yuan, K., Savitsky, T. D., Lockwood, J. R., & Edelen, M. O. (2015). Uncovering multivariate structure in classroom observations in the presence of rater errors. Educational Measurement: Issues and Practice, 34(2), 34–46.

    Article  Google Scholar 

  • McKeown, M. G., & Beck, I. L. (2015). Effective classroom talk is reading comprehension instruction. In L. B. Resnick, C. S. C. Asterhan, & S. N. Clarke (Eds.), Socializing intelligence through academic talk and dialogue (pp. 51–62). American Educational Research Association.

    Chapter  Google Scholar 

  • Miles, Huberman, A. M., & Saldaña, J. (2020). Qualitative data analysis : a methods sourcebook. (Fourth edition.). SAGE.

  • Murphy, P. K., Wilkinson, I. A. G., Soter, A. O., Hennessey, M. N., & Alexander, J. F. (2009). Examining the effects of classroom discussion on students’ high-level comprehension of text: A meta-analysis. Journal of Educational Psychology, 101, 740–764.

    Article  Google Scholar 

  • Palonsky, S. B. (1986). 900 Shows a year: A look at teaching from a teacher’s side of the desk. McGraw-Hill.

    Google Scholar 

  • Pan, S. J., & Yang, Q. (2010). A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10), 1345–1359.

    Article  Google Scholar 

  • Pepper, M. J., Ehlert, M. W., Parsons, E. S., Stahlheber, S. W., & Burns, S. F. (2015). Educator evaluations in Tennessee: Findings from the 2014 First to the Top survey. Tennessee Consortium on Research, Evaluation, & Development. Vanderbilt.

  • Praetorius, A. K., & Charalambous, C. Y. (2018). Classroom observation frameworks for studying instructional quality: Looking back and looking forward. ZDM Mathematics Education, 50(3), 535–553.

    Article  Google Scholar 

  • Price, H. E. (2012). Principal-teacher interactions: How affective relationships shape principal and teacher attitudes. Educational Administration Quarterly, 48, 39–85.

    Article  Google Scholar 

  • Price, H. E. (2021). Weathering fluctuations in teacher commitment: Leaders relational failures, with improvement prospects. Journal of Educational Administration, 59, 493–513.

    Article  Google Scholar 

  • Putnam, R. T., & Borko, H. (2000). What do new views of knowledge and thinking have to say about research on teacher learning? Educational Researcher, 29, 4–15.

    Article  Google Scholar 

  • Quintelier, A., De Maeyer, S., & Vanhoof, J. (2020). Determinants of teachers’ feedback acceptance during a school inspection visit. School Effectiveness and School Improvement, 31, 529–547.

    Article  Google Scholar 

  • Resnick, L. B., Asterhan, C. S. C., Clarke, S. N., & Schantz, F. (2018). Next generation research in dialogic learning. In G. E. Hall, L. F. Quinn, & D. M. Gollnick (Eds.), Wiley handbook of teaching and learning (pp. 323–338). Wiley-Blackwell.

    Chapter  Google Scholar 

  • Reznitskaya, A., Anderson, R. C., McNurlen, B., Nguyen-Jahiel, K., Archodidou, A., & Kim, S.-O. (2001). Influence of oral discussion on written argument. Discourse Processes, 32, 155–175.

    Article  Google Scholar 

  • Sankaranarayanan, S., Kandimalla, S. R., Hasan, S., An, H., Bogart, C., Murray, R. C., ... & Rosé, C. (2020). Agent-in-the-loop: conversational agent support in service of reflection for learning during collaborative programming. In Artificial Intelligence in Education: 21st International Conference, AIED 2020, Ifrane, Morocco, July 6–10, 2020, Proceedings, Part II 21 (pp. 273–278). Springer International Publishing.

  • Schaefer, K. E., Chen, J. Y., Szalma, J. L., & Hancock, P. A. (2016). A meta-analysis of factors influencing the development of trust in automation: Implications for understanding autonomy in future systems. Human Factors, 58, 377–400.

    Article  Google Scholar 

  • Shernoff, D. J. (2013). Optimal learning environments to promote student engagement. Springer.

    Book  Google Scholar 

  • Shute, V. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189.

    Article  Google Scholar 

  • Song, Y., Lei, S., Hao, T., Lan, Z., & Ding, Y. (2021). Automatic classification of semantic content of classroom dialogue. Journal of Educational Computing Research, 59, 496–521.

    Article  Google Scholar 

  • Southwell, R., Pugh, S., Perkoff, E. M., Clevenger, C., Bush, J. B., Lieber, R., ... & D’Mello, S. (2022). Challenges and Feasibility of Automatic Speech Recognition for Modeling Student Collaborative Discourse in Classrooms. International Educational Data Mining Society.

  • Stigler, J. W., & Miller, K. F. (2018). Expertise and expert performance in teaching. In The Cambridge handbook of expertise and expert performance (pp. 431–452). Cambridge University Press. https://doi.org/10.1017/9781316480748.024

  • Suresh, A., Sumner, T., Huang, I., Jacobs, J., Foland, B., & Ward, W. (2018). Using deep learning to automatically detect talk moves in teachers’ mathematics lessons. In 2018 IEEE International Conference on Big Data (Big Data), 5445–5447.

  • Suresh, A., Sumner, T., Jacobs, J., Foland, B., & Ward, W. (2019). Automating analysis and feedback to improve mathematics teachers’ classroom discourse. Proceedings of the AAAI Conference on Artificial Intelligence.

  • Suresh, A., Jacobs, J., Perkoff, M., Martin, J., & Sumner, T. (2022). Fine-tuning transformers with additional context to classify discursive moves in mathematics classrooms. 17th Workshop on Innovative Use of NLP for Building Educational Applications.

  • Taylor, B. M., Pearson, P. D., Peterson, D. P., & Rodriguez, M. C. (2005). The CIERA School change framework: An evidence-based approach to professional development and school reading improvement. Reading Research Quarterly, 40, 40–69.

    Article  Google Scholar 

  • Tran, N., Pierce, B., Litman, D., Correnti, R., & Matsumura, L. C. (2023). Utilizing natural language processing for automated assessment of classroom discussion. In International Conference on Artificial Intelligence in Education (pp. 490–496). Springer Nature Switzerland.

  • Tschannen-Moran, M., & Hoy, W. (1998). Trust in schools: A conceptual and empirical analysis. Journal of Educational Administration, 36, 334–352.

    Article  Google Scholar 

  • van de Grift, W. J. (2014). Measuring teaching quality in several European countries. School Effectiveness and School Improvement, 25, 295–311.

    Article  Google Scholar 

  • Van Maele, D., & Van Houtte, M. (2009). Faculty trust and organizational school characteristics: An exploration across secondary schools in Flanders. Educational Administration Quarterly, 45, 556–589.

    Article  Google Scholar 

  • Vanover, C., Mihas, P., & Saldaña, J. (Eds.). (2021). Analyzing and interpreting qualitative research: After the interview. SAGE Publications.

    Google Scholar 

  • Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, & R. Garnett (Eds.), Advances in neural information processing systems, 30: Annual conference on neural information processing systems 2017 (pp. 5998–6008).

  • White, M. C. (2018). Rater performance standards for classroom observation measures. Educational Researcher, 47, 492–501.

    Article  Google Scholar 

  • White, M., & Klette, K. (2023). What’s in a score? Problematizing interpretations of observation scores. Studies in Educational Evaluation, 77, 101238.

    Article  Google Scholar 

  • Wieczorek, D., Aguilar, I., & Mette, I. (2022). System-level leaders’ local control of teacher supervision and evaluation under every student succeeds act. AASA Journal of Scholarship & Practice, 19(3), 10–31.

  • Wilkinson, I. A., Soter, A., & Murphy, P. (2010). Develo** a model of quality talk about literary text. In M. G. McKeown & L. Kucan (Eds.), Bringing reading research to life (pp. 142–169). Guilford.

    Google Scholar 

  • Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., ... & Rush, A. M. (2019). HuggingFace’s transformers: State-of-the-art natural language processing. ar** a formative assessment protocol to support professional growth. Educational Assessment, 25(4), 314–330.

    Article  Google Scholar 

Download references

Funding

This research was supported by the National Science Foundation (NSF IIS 1735785). Any opinions, findings, conclusions, or recommendations expressed in this paper are those of the author and do not represent the views of the funding agencies.

Author information

Authors and Affiliations

  1. Department of Educational Foundations, Organizations, and Policy, University of Pittsburgh, Pittsburgh, PA, USA

    Sean Kelly & Gizem Guner

  2. Institute of Cognitive Science, University of Colorado Boulder, Boulder, CO, USA

    Nicholas Hunkins & Sidney K. D’Mello

Authors
  1. Sean Kelly
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gizem Guner
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicholas Hunkins
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sidney K. D’Mello
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Author 1: Conceptualization; Data Collection; Methodology-Surveys; Analysis; Writing-Composing, Review, and Editing. Author 2: Analysis; Editing. Author 3: Methodology-Software and Automation; Data Collection-user support. Author 4: Conceptualization; Methodology-Software and Automation; Writing-Composing, Review, and Editing.

Corresponding author

Correspondence to Sean Kelly.

Ethics declarations

Competing Interests

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 75 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kelly, S., Guner, G., Hunkins, N. et al. High School English Teachers Reflect on Their Talk: A Study of Response to Automated Feedback with the Teacher Talk Tool. Int J Artif Intell Educ (2024). https://doi.org/10.1007/s40593-024-00417-x

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s40593-024-00417-x

Keywords

Search

Navigation