Exploring The Role of Generative AI As a Training Tool for Medical Undergraduates in Discharge Summary Writing-Methodology And Study Design

Nathasha LUKE 2, *, CHUA Chun En1, and Desmond B. TEO1

1Department of Medicine, NUHS
2Department of Physiology, Yong Loo Lin School of Medicine, NUS

*nathasha@nus.edu.sg

Luke, N., Chua, C.E., & Teo D.B. (2024). Exploring The Role of Generative AI As a Training Tool for Medical Undergraduates in Discharge Summary Writing -Methodology And Study Design[Lightning Talk]. In Higher Education Conference in Singapore (HECS) 2024, 3 December, National University of Singapore. https://blog.nus.edu.sg/hecs/hecs2024-luke-et-al

SUB-THEME

Opportunities from Generative AI

KEYWORDS

Discharge summary, Generative AI, Chatbot, Large Language Models

CATEGORY

Lightning Talk

INTRODUCTION

A discharge summary is a permanent record of a patient’s hospitalisation, which should be concise, yet contain adequate and accurate information regarding the hospitalisation (Ando et al., 2022). Substandard discharge summaries result in gaps in subsequent patient follow-ups, clinical coding of data, hospital subvention, and medical insurance (Sukanya, 2017). Globally, discharge summaries are authored by junior doctors but there is little formal teaching and quality assessment in most training programs. An initial audit of 100 discharge summaries within the Department of Medicine, National University Hospital, in January 2021 revealed that only 21% had complete information.

 

To address this gap, a teaching program was implemented to train medical students on discharge summary writing and hands-on, case-based sessions where the students drafted discharge summaries for tutors to provide feedback. This programme demonstrated an improvement in the quality of discharge summaries over the years (Chua & Teo, 2023). However, conducting this program was challenging due to limitations in the number of facilitators to conduct these sessions and provide one-to-one feedback. Hence, we planned a project to evaluate the capability of Generative Artificial Intelligence (Gen AI) to provide feedback in discharge summary writing training.

METHODOLOGY AND WORKFLOW

To ensure sustainability without the need for facilitator manpower, this project caters to an interactive e-learning module complemented by Gen AI to provide feedback on discharge summaries written by students based on case scenarios. Gen AI will assess the accuracy and quality of discharge summaries based on a rubric to provide individualised feedback.

 

This study will be conducted in two phases, where in the initial phase, researchers will evaluate different Gen AI platforms to decide on the best platform to provide feedback. In the subsequent phase, the students will directly interact with the selected platform to receive feedback, in which the researchers will evaluate the learning experience.

 

In the first phase, an e-learning module will be implemented to train students followed by a formative assessment component where students create and submit their discharge summaries through the LMS. Each discharge summary will be subjected to feedback from five arms, (1) an experienced clinician, and generative AI platforms which include (2) Llama 3, (3) Gemini AI, (4) Co-Pilot, and (5) GPT-4 powered Chatbot. The feedback provided by these five arms will then be objectively evaluated by an expert in a blinded manner, to identify the best platform.

 

In the second phase, the students will directly interact with the selected platform as guided by the study team to receive feedback for discharge summaries. The generative AI outputs and student feedback will be evaluated to determine the efficacy and identify the best strategies to implement the programme.

FIGURES AND TABLES

a22 - Fig 1

Figure 1. Methodology for Phase 1

 

Figure 2. Methodology for Phase II

 

REFERENCES

Ando, K., Okumura, T., Komachi, M., Horiguchi, H., & Matsumoto, Y. (2022). Is artificial intelligence capable of generating hospital discharge summaries from inpatient records?. PLOS Digital Health, 1(12). https://doi.org/10.1371/journal.pdig.0000158

Sukanya, C. (2017). Validity of principal diagnoses in discharge summaries and ICD-10 coding assessments based on national health data of Thailand. Healthcare Informatics Research, 23(4), 293-303. https://doi.org/10.4258/hir.2017.23.4.293

Chua, C. E., & Teo, D. B. (2023). Writing a high‐quality discharge summary through structured training and assessment. Medical Education, 57(8), 773–774. https://doi.org/10.1111/medu.15102

Econ on the Go: Chatbot-Guided Scavenger Hunt for Large Economics Classes 

Timothy WONG*, CHAN Kok Hoe, and ONG EeCheng

Department of Economics, Faculty of Arts and Social Sciences (FASS)

ecstwcj@nus.edu.sg

Wong, T. C. J., Chan, K. H., & Ong, E. C. (2024). Econ on the Go: Chatbot-guided scavenger hunt for large economic classes [Poster presentation]. In Higher Education Conference in Singapore (HECS) 2024, 3 December, National University of Singapore. https://blog.nus.edu.sg/hecs/hecs2024-twong-chankh-ongec/ 

SUB-THEME

Others – Opportunities from AI 

KEYWORDS

Experiential learning, field trip, scavenger hunt, chatbot 

CATEGORY

Poster Presentation

EXTENDED ABSTRACT

We designed a chatbot-guided scavenger hunt for students in our core introductory course, EC1101E: “Introduction to Economic Analysis”, which enrols 300–600 students each semester. This project is funded by a Teaching Enhancement Grant (TEG). 

To design the scavenger hunt, we first identified economic concepts that can be matched to locational features and histories. For example, the concept of technological change as a decrease in inputs is manifested in the transition post World War II from rickshaws to trishaws; the inputs here are labor hours and energy. This story is explained on a plaque at Jinricksha Station. 

Students visit various locations to collaboratively solve the puzzles by applying economic knowledge. As Mayer (2004) states, “meaningful learning occurs when the learner strives to make sense of the presented material by selecting relevant incoming information, organizing it into a coherent structure, and integrating it with other organized knowledge.” Here, the “presented material” is the puzzle, the “relevant incoming information” is their surroundings, and “other organized knowledge” is the economic theories they have learned in class. After students solve a puzzle, they are guided by the chatbot to the next location where a new puzzle is revealed. 

The chatbot will record students’ responses to the puzzles and clues, enabling us to evaluate students’ strengths and weaknesses in their understanding of specific economic concepts and in their ability to apply such concepts to novel contexts. We may use this information to adjust how we teach a particular concept, to modify the puzzles, and/or to improve the chatbot in future iterations. 

Chatbots offer scalability and accessibility, and their potential as a pedagogical tool is being explored (Wollny et. al., 2021) including in the area of experiential learning (Casillo et al., 2022). While experiential field trips provide unique opportunities for students to identify economic concepts at play in the real world and to recognize the limits of economic theories and models (Ong & Wong, 2023), these field trips are typically led by human guides who can lead only a small group of students at a time. Meanwhile, chatbot guides can accommodate hundreds of students and permit flexible scheduling. 

We believe that the chatbot-guided scavenger hunt promotes several positive learning outcomes. First, we hope that this activity will help students to see the value of economics in understanding the world, thereby increasing their engagement with economics (Eccles & Wigfield, 2020). Studies show that intrinsic interest in a subject drives deep learning while a focus on extrinsic rewards leads to surface learning (Laird & Garver, 2010; Entwistle, 2009). 

Second, this experiential learning activity presents a unique opportunity for students to practice applying economic concepts in the field. The ability to recognise which economic theories apply to which real-world scenarios will enable them to transfer their learning to their subsequent courses and careers (Green et al., 2013). 

Third, we aim to inculcate in our students intellectual curiosity, where they continually observe, question, and discover. Students engage with one another in a social, collaborative setting to solve puzzles. This experience may shape their perception of how and when learning happens, leading them to recognise that learning is not circumscribed to the classroom. 

We will conduct surveys at the beginning of the semester and after the field trip to evaluate the efficacy of the chatbot-guided field trip. We are interested in the following outcomes: (i) students’ knowledge of economic theory; (ii) students’ ability to apply economic theory to new contexts; (iii) students’ interest in economics; (iv) students’ perception of the relevance of economics in their lives; (v) students’ perception of and attitudes toward learning; (vi) students’ engagement with their classmates.

REFERENCES

Casillo, M., De Santo, M., Mosca, R., & Santaniello, D. (2022). An ontology-based chatbot to enhance experiential learning in a cultural heritage scenario. Frontiers in Artificial Intelligence 5. https://doi.org/10.3389/frai.2022.808281 

Eccles, J. S., & Wigfield, A. (2020). From expectancy-value theory to situated expectancy-value theory: A developmental, social cognitive, and sociocultural perspective on motivation. Contemporary Educational Psychology, 61, Article 101859. https://doi.org/10.1016/j.cedpsych.2020.101859 

Entwistle, N. (2009). Teaching for understanding at university: Deep approaches and distinctive ways of thinking. Universities into the 21st Century Series. Palgrave Macmillan. 

Green, G., Bean, J., & Peterson, D. (2013). Deep learning in intermediate microeconomics: using scaffolding assignments to teach theory and promote transfer. Journal of Economic Education, 44(2). 142-157. https://doi.org/10.1080/00220485.2013.770338 

Laird, T. N., & Garver, A. K. (2010). The effect of teaching general education courses on deep approaches to learning: How disciplinary context matters. Research in Higher Education, 51(3), 248–265, https://doi.org/10.1007/s11162-009-9154-7 

Mayer, R. E. (2004). Should there be a three-strikes rules against pure discovery learning? The case for guided methods of instruction. American Psychologist, 59(1), 14. https://doi.org/10.1037/0003-066X.59.1.14

Ong, E. C., & Wong, T. (2023). Bringing the classroom to the real world: Field trips to marginalized neighborhoods. Journal of Economic Education, 54:(3), 267–280. 10.1080/00220485.2023.2200409 

Wollny, S., Schneider, J., Di Mitri, D., Weidlich, J., Rittberger, M., & Drachsler, H. (2021). Are we there yet? A systematic literature review on chatbots in education. Frontiers in Artificial Intelligence 4. http://dx.doi.org/10.3389/frai.2021.654924 

Viewing Message: 1 of 1.
Success

Blog.nus login is now via SSO. Don't see or cannot edit your blogs after logging in? Please get in touch with us, and we will fix that. (More information.)