Exploring The Role of Generative AI As a Training Tool for Medical Undergraduates in Discharge Summary Writing-Methodology And Study Design

Nathasha LUKE ^{2, _*}, CHUA Chun En¹, and Desmond B. TEO¹

¹Department of Medicine, NUHS
²Department of Physiology, Yong Loo Lin School of Medicine, NUS

^*nathasha@nus.edu.sg

Luke, N., Chua, C.E., & Teo D.B. (2024). Exploring The Role of Generative AI As a Training Tool for Medical Undergraduates in Discharge Summary Writing -Methodology And Study Design[Lightning Talk]. In Higher Education Conference in Singapore (HECS) 2024, 3 December, National University of Singapore. https://blog.nus.edu.sg/hecs/hecs2024-luke-et-al

SUB-THEME

Opportunities from Generative AI

KEYWORDS

Discharge summary, Generative AI, Chatbot, Large Language Models

INTRODUCTION

A discharge summary is a permanent record of a patient’s hospitalisation, which should be concise, yet contain adequate and accurate information regarding the hospitalisation (Ando et al., 2022). Substandard discharge summaries result in gaps in subsequent patient follow-ups, clinical coding of data, hospital subvention, and medical insurance (Sukanya, 2017). Globally, discharge summaries are authored by junior doctors but there is little formal teaching and quality assessment in most training programs. An initial audit of 100 discharge summaries within the Department of Medicine, National University Hospital, in January 2021 revealed that only 21% had complete information.

To address this gap, a teaching program was implemented to train medical students on discharge summary writing and hands-on, case-based sessions where the students drafted discharge summaries for tutors to provide feedback. This programme demonstrated an improvement in the quality of discharge summaries over the years (Chua & Teo, 2023). However, conducting this program was challenging due to limitations in the number of facilitators to conduct these sessions and provide one-to-one feedback. Hence, we planned a project to evaluate the capability of Generative Artificial Intelligence (Gen AI) to provide feedback in discharge summary writing training.

METHODOLOGY AND WORKFLOW

To ensure sustainability without the need for facilitator manpower, this project caters to an interactive e-learning module complemented by Gen AI to provide feedback on discharge summaries written by students based on case scenarios. Gen AI will assess the accuracy and quality of discharge summaries based on a rubric to provide individualised feedback.

This study will be conducted in two phases, where in the initial phase, researchers will evaluate different Gen AI platforms to decide on the best platform to provide feedback. In the subsequent phase, the students will directly interact with the selected platform to receive feedback, in which the researchers will evaluate the learning experience.

In the first phase, an e-learning module will be implemented to train students followed by a formative assessment component where students create and submit their discharge summaries through the LMS. Each discharge summary will be subjected to feedback from five arms, (1) an experienced clinician, and generative AI platforms which include (2) Llama 3, (3) Gemini AI, (4) Co-Pilot, and (5) GPT-4 powered Chatbot. The feedback provided by these five arms will then be objectively evaluated by an expert in a blinded manner, to identify the best platform.

In the second phase, the students will directly interact with the selected platform as guided by the study team to receive feedback for discharge summaries. The generative AI outputs and student feedback will be evaluated to determine the efficacy and identify the best strategies to implement the programme.

FIGURES AND TABLES

Figure 1. Methodology for Phase 1

Figure 2. Methodology for Phase II

REFERENCES

Ando, K., Okumura, T., Komachi, M., Horiguchi, H., & Matsumoto, Y. (2022). Is artificial intelligence capable of generating hospital discharge summaries from inpatient records?. PLOS Digital Health, 1(12). https://doi.org/10.1371/journal.pdig.0000158

Sukanya, C. (2017). Validity of principal diagnoses in discharge summaries and ICD-10 coding assessments based on national health data of Thailand. Healthcare Informatics Research, 23(4), 293-303. https://doi.org/10.4258/hir.2017.23.4.293

Chua, C. E., & Teo, D. B. (2023). Writing a high‐quality discharge summary through structured training and assessment. Medical Education, 57(8), 773–774. https://doi.org/10.1111/medu.15102