SpringerLink
Log in

Who’s Hel** Who? When Students Use ChatGPT to Engage in Practice Lab Sessions

  • Conference paper
  • First Online:
Artificial Intelligence in Education (AIED 2024)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14829))

Included in the following conference series:

  • 45 Accesses

Abstract

Little is understood about how chatbots powered by Large Language Models (LLMs) impact teaching and learning, and how to effectively integrate them into educational practices. This study examined whether and how using ChatGPT in a graduate-level robotics course impacts performance and learning with data from 64 students (40 ChatGPT users and 24 non-users). Regression analyses revealed complex interactions between ChatGPT use, task performance, and learning of course-related concepts: using ChatGPT significantly improved task performance, but not necessarily learning outcomes. Task performance positively correlated with learning only for students with low and medium prior knowledge who did not use ChatGPT, suggesting that performance translated to learning only for non-ChatGPT users overall. Clustering ChatGPT-users’ prompts helped identify three types of usage that differed in terms of learning and performance. Although all ChatGPT users had improved performance, Debuggers (who requested solutions and error fixes) outperformed the other clusters. In terms of learning, Conceptual Explorers (who sought to understand concepts, tasks or codes) had higher learning outcomes than Debuggers and Practical Developer (who exclusively asked for task solutions). The behaviors elicited by students in the Practical Developer and Debuggers clusters therefore were less likely to translate performance into conceptual understanding, while the Conceptual Explorers’ behaviors were more conducive to learning. This empirical study helps improve our understanding of the complex dynamics between how ChatGPT is used, performance, and learning outcomes.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
EUR 29.95
Price includes VAT (Germany)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
EUR 68.47
Price includes VAT (Germany)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
EUR 85.59
Price includes VAT (Germany)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Ethically approved by EPFL’s Ethics Committee (HREC No: 075-2023/18.09.2023).

  2. 2.

    On October 31st 2023, 39 students used ChatGPT 3.5, and one used ChatGPT 4.0.

  3. 3.

    We excluded 25 prompts unrelated to solving the tasks (e.g., “hello”, “thank you”).

References

  1. Adamson, V., Bägerfeldt, J.: Assessing the effectiveness of ChatGPT in generating Python code (2023)

    Google Scholar 

  2. Azaria, A., Azoulay, R., Reches, S.: ChatGPT is a remarkable tool–for experts. ar**v preprint ar**v:2306.03102 (2023). https://doi.org/10.48550/ar**v.2306.03102

  3. Baidoo-Anu, D., Ansah, L.O.: Education in the era of generative artificial intelligence (AI): understanding the potential benefits of ChatGPT in promoting teaching and learning. J. AI 7(1), 52–62 (2023)

    Article  Google Scholar 

  4. Becker, B.A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., Santos, E.A.: Programming is hard - or at least it used to be: educational opportunities and challenges of AI code generation. In: Proceedings of the 54th ACM Technical Symposium on Computer Science Education, vol. 1, pp. 500–506. SIGCSE 2023, Association for Computing Machinery, New York, NY, USA (2023)

    Google Scholar 

  5. Bird, C., et al.: Taking Flight with Copilot: early insights and opportunities of AI-powered pair-programming tools. Queue 20(6), 10:35–10:57 (2023). https://doi.org/10.1145/3582083

  6. Hwang, G.J., Chang, C.Y.: A review of opportunities and challenges of chatbots in education. Interact. Learn. Environ. 31(7), 4099–4112 (2023)

    Article  Google Scholar 

  7. Kasneci, E., et al.: ChatGPT for good? On opportunities and challenges of large language models for education. Learn. Individ. Differ. 103, 102274 (2023)

    Article  Google Scholar 

  8. Kazemitabaar, M., Chow, J., Ma, C.K.T., Ericson, B.J., Weintrop, D., Grossman, T.: Studying the effect of AI code generators on supporting novice learners in introductory programming. In: CHI’2023, pp. 1–23. ACM (2023)

    Google Scholar 

  9. Kiesler, N., Lohr, D., Keuning, H.: Exploring the potential of large language models to generate formative programming feedback. In: 2023 IEEE Frontiers in Education Conference (FIE), pp. 1–5 (2023)

    Google Scholar 

  10. Konak, A., Clarke, C.J.S.F.: Augmenting critical thinking skills in programming education through leveraging chat GPT: analysis of its opportunities and consequences. In: Fall Mid Atlantic Conference: Meeting Our Students Where They are and Getting them Where They Need to be (2023)

    Google Scholar 

  11. Kuhail, M.A., Alturki, N., Alramlawi, S., Alhejori, K.: Interacting with educational chatbots: a systematic review. Educ. Inf. Technol. 28(1), 973–1018 (2023)

    Article  Google Scholar 

  12. Lau, S., Guo, P.: From “Ban It Till We Understand It” to “Resistance is Futile”: how university programming instructors plan to adapt as more students use AI code generation and explanation tools such as ChatGPT and GitHub copilot. In: ICER’2023. vol. 1, pp. 106–121. ACM (2023)

    Google Scholar 

  13. Likas, A., Vlassis, N., Verbeek, J.J.: The global k-means clustering algorithm. Pattern Recogn. 36(2), 451–461 (2003)

    Article  Google Scholar 

  14. Logozar, R., Mikac, M., Hizak, J.: ChatGPT on the freshman test in C/C++ programming (2023)

    Google Scholar 

  15. Loubier, M.: ChatGPT: A good computer engineering student?: An experiment on its ability to answer programming questions from exams (2023)

    Google Scholar 

  16. Memarian, B., Doleck, T.: ChatGPT in education: methods, potentials, and limitations. Comput. Hum. Behav.: Artif. Hum. 1(2), 100022 (2023)

    Google Scholar 

  17. Oosterwyk, G., Tsibolane, P., Kautondokwa, P., Canani, A.: Beyond the Hype: a cautionary tale of ChatGPT in the programming classroom (2023)

    Google Scholar 

  18. OpenAI: Chatgpt (2023). https://openai.com/chatgpt. Accessed 31 Oct 2023

  19. Pérez, J.Q., Daradoumis, T., Puig, J.M.M.: Rediscovering the use of chatbots in education: a systematic literature review. Comput. Appl. Eng. Educ. 28(6), 1549–1565 (2020). https://doi.org/10.1002/cae.22326

    Article  Google Scholar 

  20. Qureshi, B.: Exploring the use of ChatGPT as a tool for learning and assessment in undergraduate computer science curriculum: opportunities and challenges. In: 2023 9th International Conference on e-Society, e-Learning and e-Technologies, pp. 7–13 (2023). https://doi.org/10.1145/3613944.3613946

  21. Reiche, M., Leidner, J.L.: Bridging the programming skill gap with ChatGPT: a machine learning project with business students. In: European Conference on Artificial Intelligence, pp. 439–446. Springer (2023). https://doi.org/10.1007/978-3-031-50485-3_42

  22. Rousseeuw, P.J.: Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20, 53–65 (1987)

    Article  Google Scholar 

  23. Shoufan, A.: Can students without prior knowledge use ChatGPT to answer test questions? An empirical study. ACM Trans. Comput. Educ. 23(4), 45:1–45:29 (2023)

    Google Scholar 

  24. Shoufan, A.: Exploring students’ perceptions of ChatGPT: thematic analysis and follow-up survey. IEEE Access 11, 38805–38818 (2023)

    Article  Google Scholar 

  25. Shute, V.J., Sun, C., Asbell-Clarke, J.: Demystifying computational thinking. Educ. Res. Rev. 22, 142–158 (2017)

    Article  Google Scholar 

  26. Singh, H., Tayarani-Najaran, M.H., Yaqoob, M.: Exploring computer science students’ perception of ChatGPT in higher education: a descriptive and correlation study. Educ. Sci. 13(9), 924 (2023)

    Article  Google Scholar 

  27. Susnjak, T.: ChatGPT: the end of online exam integrity? ar**v preprint ar**v:2212.09292 (2022). https://doi.org/10.48550/ar**v.2212.09292

  28. Tian, H., et al.: Is ChatGPT the ultimate programming assistant–how far is it? ar**v preprint ar**v:2304.11938 (2023). https://doi.org/10.48550/ar**v.2304.11938

  29. Welsh, M.: The end of programming. Commun. ACM 66(1), 34–35 (2023)

    Article  MathSciNet  Google Scholar 

  30. Yilmaz, R., Karaoglan Yilmaz, F.G.: Augmented intelligence in programming learning: examining student views on the use of ChatGPT for programming learning. Comput. Hum. Behav.: Artif. Hum. 1(2), 100005 (2023)

    Google Scholar 

  31. Yu, H.: Reflection on whether Chat GPT should be banned by academia from the perspective of education and teaching. Front. Psychol. 14 (2023)

    Google Scholar 

Download references

Acknowledgements

We would like to thank our colleagues (J.W., N.A, A.B, R.B, C.C, S.A), and the students who volunteered to participate in the study.

Author information

Authors and Affiliations

  1. MOBOTS & LEARN, EPFL - École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland

    Jérôme Brender & Francesco Mondada

  2. University of Teacher Education (Haute École Pédagogique) Vaud, Lausanne, Switzerland

    Jérôme Brender, Laila El-Hamamsy & Engin Bumbacher

Authors
  1. Jérôme Brender
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Laila El-Hamamsy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Francesco Mondada
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Engin Bumbacher
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jérôme Brender .

Editor information

Editors and Affiliations

  1. University of Memphis, Memphis, TN, USA

    Andrew M. Olney

  2. University of Duisburg-Essen, Duisburg, Germany

    Irene-Angelica Chounta

  3. **an University, Guangzhou, China

    Zitao Liu

  4. UNED, Madrid, Spain

    Olga C. Santos

  5. Universidade Federal de Alagoas, Maceio, Brazil

    Ig Ibert Bittencourt

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Brender, J., El-Hamamsy, L., Mondada, F., Bumbacher, E. (2024). Who’s Hel** Who? When Students Use ChatGPT to Engage in Practice Lab Sessions. In: Olney, A.M., Chounta, IA., Liu, Z., Santos, O.C., Bittencourt, I.I. (eds) Artificial Intelligence in Education. AIED 2024. Lecture Notes in Computer Science(), vol 14829. Springer, Cham. https://doi.org/10.1007/978-3-031-64302-6_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-64302-6_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-64301-9

  • Online ISBN: 978-3-031-64302-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation