Qian HUANG1,*, Ameek Kaur2, Thijs WILLEMS1
1Lee Kuan Yew Centre for Innovative Cities, Singapore University of Technology and Design (SUTD)
2NUS Business School
Huang, Q., Kaur, A., & Willems, T. (2024). Leveraging ChatGPT for analysing student reflections in a design thinking course [Paper presentation]. In Higher Education Conference in Singapore (HECS) 2024, 3 December, National University of Singapore. https://blog.nus.edu.sg/hecs/qhuang-et-al/
SUB-THEME
Opportunities from Generative AI
KEYWORDS
Generative AI, ChatGPT, large-scale reflection, qualitative analysis, design education
CATEGORY
Paper Presentation
EXTENDED ABSTRACT
Generative Artificial Intelligence (Gen-AI) is increasingly being integrated into teaching and research methodologies, particularly since the advent of ChatGPT (Albdrani & Al-shargabi, 2023; Hwang & Chen, 2023). As educators navigate this evolving landscape, it becomes crucial to understand how to effectively and critically utilise Gen-AI tools in academic settings. This study explores the application of ChatGPT in analysing student reflections in a design thinking course at a university in Singapore. The course involved 550 first-year students across 11 cohorts, each student required to write four reflections over a semester. The significant volume of reflections presented a unique opportunity to deploy ChatGPT-4.0 for large-scale qualitative analysis.
Initially, researchers manually analysed the reflections of 50 students from one class to establish a benchmark. These manual analyses were then compared to ChatGPT’s results to verify the reliability of the AI-driven approach. Upon confirming ChatGPT’s reliability, the tool was employed to analyse reflections from the entire cohort through the semester (550 students X four phases). The analysis focused on two primary objectives: first, to assess the impact of pedagogical interventions on students’ Affect, Behavior, and Cognition (ABC); and second, to understand how students applied these interventions and the frequency of their application.
The study aimed to uncover how specific pedagogical interventions influenced students’ emotional responses, behavioural changes, and cognitive developments by using ChatGPT. For instance, it was observed that interventions such as confirmation bias were frequently applied by students during site visits to explore problems from multiple perspectives. This detailed analysis provided insights into the effectiveness of various teaching strategies and highlighted areas for potential improvement.
Key findings from the study revealed several noteworthy trends. Firstly, some interventions, including case studies and activities, did not significantly impact students’ affective responses to the ideas emphasised in these interventions. This suggests that educators may need to refine these interventions to better support students emotionally. Secondly, the analysis highlighted variations in the delivery and emphasis of interventions across different cohorts, attributable to individual teaching styles of different instructors. ChatGPT’s analysis provided a nuanced understanding of how these differences influenced student outcomes.
By leveraging ChatGPT, the research team was able to conduct a comprehensive analysis of a large dataset, providing valuable insights that might not have been feasible through manual analysis alone. The findings underscore the potential of Gen-AI tools in educational research, particularly in scaling qualitative analyses and uncovering patterns that inform pedagogical practices.
In summary, this study demonstrates the utility of ChatGPT in analysing student reflections to gauge the impact of pedagogical interventions on students’ action, emotion, and cognition. The application of Gen-AI in this context not only facilitated the processing of a large volume of qualitative data but also offered educators deeper insights into how classroom interventions can be optimised to achieve desired educational outcomes. This method represents a significant advancement in educational research, providing a scalable and reliable approach to understanding and enhancing student learning experiences.
This study contributes to the growing body of literature on the use of AI in education and offers practical implications for educators seeking to integrate Gen-AI tools into their teaching practices. Future research could expand on these findings by exploring the application of ChatGPT in different educational contexts and with diverse student populations to further validate and refine this approach.
REFERENCES
Albdrani, R., & Al-shargabi, A. (2023). Investigating the effectiveness of ChatGPT for providing personalized learning experience: A case atudy. International Journal of Advanced Computer Science and Applications. https://doi.org/10.14569/ijacsa.2023.01411122.
Hwang, G. J. & Chen., N. S. (2023). Exploring the potential of generative artificial intelligence in education: Applications, challenges, and future research directions. Educational Technology & Society, 26(2). https://doi.org/10.30191/ETS.202304_26(2).0014