Knowledge Attainment and Engagement Among Medical Students: A Comparison of Three Forms of Online Learning

Introduction

Over the past decade, remote learning has gained increased prominence in higher education.¹ Undergraduate medicine, which has gradually been evolving from lecture-based curricula towards more integrated, collaborative approaches,^2,3 has also followed this trend. The COVID-19 pandemic has accelerated the pace of change and necessitated a faster transformation to remote learning in order to meet complex learning objectives.⁴

One important method of delivering remote medical teaching has been the use of case-based learning. Students are presented with a short text-based clinical vignette and discuss the case with a tutor remotely. This approach uses authentic patient cases to provide a relevant and realistic experience to students, aiming to connect existing theory to clinical practice.⁵ Studies have demonstrated its benefits, which include improving knowledge attainment, enhancing patient care outcomes, and yielding student satisfaction.^6,7

As an extension to this, medical educational multimedia such as video have also become a prominent method of delivering case-based teaching, as opposed to using written vignettes alone. Video-based case materials have been shown to be an engaging way of learning, although some studies have identified limitations, such as a reduction in deep thinking compared with written vignettes.^8,9

A more novel and less developed form of remote learning is the use of Extended Reality (XR). This is an umbrella term encompassing virtual reality (VR), augmented reality (AR) and mixed reality (MR). In contrast to VR, which involves a completely digital environment, AR is a form of technology which superimposes digital interfaces upon the physical world. MR, an extension to this, enables the user to directly interact with digital holograms placed in the real environment. Advances in visual technology over recent years have resulted in an increase in the use of XR in medical education. The fields of anatomy, surgical simulation and telementoring have started to integrate XR into their training programmes.^10–12

One pioneering XR device is Microsoft HoloLens 2, released in November 2019. This mixed reality technology creates an immersive environment using head-mounted glasses with a built-in camera. Previous research has demonstrated its feasibility, acceptability, and effectiveness in delivering remote-access bedside teaching, with the use of patient-based cases, to clinical-year medical students.¹³ The headset, worn by the tutor, livestreams their environment to students remotely, allowing them to view and interact with both the tutor and the patient. Moreover, the tutor can superimpose holograms, such as scans, onto the live environment as an adjunct to the session. Although the processes of delivering remote clinical teaching using MR have been described, adequate work has yet to demonstrate its utility on a large scale and compare its effectiveness directly with other forms of remote learning.

This study aimed to directly compare three forms of remote learning: clinical vignette, video-based case and MR teaching, via the Microsoft Hololens 2, delivered remotely to a cohort of third year medical students at one UK medical school. We set out to evaluate the feasibility of conducting MR teaching on a large scale, and to establish how this form of teaching compares to case-based learning and video-based learning with regards to student knowledge attainment and engagement.

Materials and Methods

Study Population

Medical students in Year 3 at Imperial College London participated in three online teaching sessions, one in each format. All students were expected to attend the teaching sessions and to complete the formative assessment, as the session was part of their scheduled learning. The topics taught aligned directly with the students’ curriculum. Inclusion of their data used as part of the research trial was optional.

The cohort were divided into a morning and afternoon session based on their current placement locations in order to minimise disruption to their clinical attachments. Students were randomised in three equal subgroups using the RANK-L method on Microsoft Excel.

All participants received an information sheet prior to the study with detailed information about the study (Appendix 1). At the beginning of the session, a researcher explained the information to participants and allowed time for answering any questions. Consent to use answers from the formative assessment was gained via Mentimeter at the beginning of the formative assessment. Students were informed they would be able to participate in all aspects regardless of whether they had provided consent for the study. All participants were able to withdraw their consent up to 2 weeks after the completion of data collection, after which all answers were anonymised.

Patients were consented to partake in the MR sessions using pre-approved Imperial College GDPR protocol and were informed that sessions would be recorded. Curtains were pulled around patients and signs placed in surrounding areas to ensure privacy for the patient and prevent other ward patients or staff being inadvertently filmed.

Materials

Three standardised scripts for each of the tutorial topics were devised; Painful Hands, Chest Pain and Breathlessness. Each script was tailored to the requirements of the specific teaching method, but contained the same framework, depth and detail to ensure congruence between the material being taught. Each script was rehearsed with individual tutors on two occasions to ensure homogeneity in teaching styles across all three tutorials.

Three simultaneous online teaching sessions were delivered; one HoloLens tutorial, one video-based tutorial, and one vignette-based tutorial – all delivered via Microsoft Teams. The tutorials were on the topics of chest pain, breathlessness, and painful hands. Each tutorial lasted 25 minutes and students rotated from one to the next, with 5-minute breaks between each tutorial, until they had experienced all three styles. There were approximately 42 students in each session.

The MR session featured a live video stream of a tutor on a hospital medical ward, interacting with and examining a patient, whilst also displaying holograms on the screen such as investigation results. Students were able to watch this session directly on their electronic devices and interact in real time with the tutor and patient using the chat and microphone facilities on Microsoft Teams.

The video-based tutorial consisted of students watching a 2–3 minute pre-recorded patient-testimony video taken from the learning platform Speaking Clinically, a national resource provided by the Medical Schools Council of video archives of real patient encounters. The tutor then guided students through a teaching session related to the clinical condition that the patient described in the video.

The vignette tutorial featured a written case, which the students read in the first 2–3 minutes of the session, and then analysed with the tutor.

All three tutorials were designed to be interactive in nature, with questions and discussions actively encouraged.

A formative assessment was then developed which featured very short answer (VSA) questions.^14,15 There were 60 questions in total, with 20 on each of the three topics being taught. The questions were written by the researchers with guidance from expert question-writers at Imperial College London. Questions were standard-set by a select group of six medical educators using the Ebel method. An inclusion threshold of 50% of the items per topic was set; students who had answered at least 10 out of 20 items in a particular topic were included. This was done in order to prevent bias caused by dropout. If the inclusion criteria were met, blank answers were marked as wrong.

A questionnaire was also developed to assess engagement and enjoyment of the tutorials. Participants were asked to rate their enjoyment and engagement for each teaching method using a 5-point Likert scale.

Assessment

Immediately after completion of the three teaching sessions, all students joined a single Microsoft Teams call and participated in the online formative assessment and enjoyment and engagement questionnaire via the quiz platform Mentimeter. The formative assessment was approximately 1 hour in length and students completed these questions remotely under usual exam-conditions and time-constraints.

All students were asked to rate their enjoyment of and engagement with each form of online session on a Likert scale from 1 to 5: the exact questions are outlined in Appendix 2.

Marking and Feedback

VSAs were marked electronically and verified by two researchers.¹⁴ A third clinician was available to arbitrate any queries.

Once all students had completed the assessment, the formative paper with answers and explanations was made available through direct email to the students.

Power Analysis

We performed an a priori power analysis to determine the appropriate sample size for the study. Our primary outcome measure was performance on the formative assessment. The expected mean (SD) for the formative assessment was 65.7% (16.5%). We performed the power analysis for four levels of effect size, as calculated by Cohen’s d (0.3, 0.4, 0.5 and 0.7) for a one-way ANOVA comparing the means of the three groups, with one group performing better and the other two groups performing equal to the expected mean. We aimed for a minimum power value of 0.8. Table 1 shows the results of the power analysis for the four levels of effect size.

Table 1 Results of Power Analysis for the Four Levels of Effect Size, Alpha of X

We opted for a target sample size of 100 per group, which gave an over 80% power to detect differences in the mean scores on the formative assessment of effect sizes of 0.4 or above (Cohen’s d).

Statistical Analysis

The study administration team produced an Excel file containing answers and scores for each student for each question. Each student was allocated a numerical code to ensure anonymity. The data were transferred into R version 4.1.0 for analysis. Comparison between performances on the formative assessment by tutorial method were investigated using a repeated measures two-way ANOVA with factors “Topic” (chest pain, breathlessness, painful hands) and “Tutorial Method” (“MR”, “video-based” or “case vignette”) and their interactions. Where the ANOVA main effects were significant, we performed pairwise comparisons with paired t-tests. We used the Holm correction to adjust p-values for multiple comparisons in the ANOVAs and in post hoc tests. Engagement and enjoyment were also analysed in the same manner.

Patient and Public Involvement

This study was conducted with medical students as the participants. Patients were involved only for the bedside teaching sessions themselves. We thoroughly briefed patients beforehand and discussed the nature of our study, as well as providing them with an information sheet and the opportunity to ask any questions. We complied and adhered to all GDPR policies of Imperial College London School of Medicine, and Imperial College London NHS Trust.

Results will be disseminated to study participants (medical students) via Professor Amir Sam, Head of Imperial College School of Medicine.

Results

A total of 324 students participated in the teaching sessions; of those, 252 students (78%) consented to participate in the study. Students in each session were randomised into three subgroups (see Table 2 for combinations of tutorial methods and topics per group). A total of 84, 82 and 86 participants from groups A, B and C, respectively, entered at least one answer on the formative assessment.

Table 2 Means and Standard Deviations of Scores on the Formative Assessment for Each Topic and Tutorial Method Combination

Clinical Knowledge Attainment and Performance

Table 2 shows the combinations of tutorial methods and topics per group and the means and standard deviations on the assessment of each tutorial by topic combination. Participants with over 50% missing data on any topic were removed from the analysis. In Groups A, B and C, respectively, 60 (71.4%), 62 (78%) and 60 (69.8%) had complete data for at least 50% of the assessments and were included in the analyses, totaling 182 participants. For these participants, missing answers were treated as incorrect. Cronbach’s alpha was calculated for each topic, collapsed over tutorial method. Cronbach’s alpha for Chest Pain was 0.57, for Breathlessness was 0.64 and for Painful Hands was 0.71.

Mean knowledge attainment scores differed across the three tutorial methods, shown by a significant main effect of Tutorial Method (F(2, 2) = 6.31, p = 0.004). When we compared each pair of tutorial methods using post hoc two-tailed t-tests, there was a trend towards higher scores on the video-based tutorial than the other two methods (p < 0.1), which did not survive correction for multiple comparisons.

Student Perceptions, Engagement and Enjoyment

Table 3 shows the means and standard deviations and medians and interquartile ranges of ratings of enjoyment and engagement for each tutorial by topic combination).

Table 3 Means, Standard Deviations and Medians and Interquartile Ranges of Engagement and Enjoyment Ratings (from 1 to 5, Where Higher Scores Indicate Higher Engagement/Enjoyment) for Each Combination of Topic and Method

Mean enjoyment ratings differed across the three tutorial methods, shown by a significant main effect of Tutorial Method (F(2, 2) = 15.68, p < 0.001). The main effect of Topic was not significant. When we compared each pair of tutorial methods using post hoc two-tailed t-tests, participants reported higher enjoyment for the case vignette method, compared to the other two (p< 0.001). We found no difference in enjoyment ratings between MR and the video-based methods. Of the included participants, 31 did not enter an enjoyment rating for one or more of the three methods.

Mean engagement ratings differed across the three tutorial methods, shown by a significant main effect of Tutorial Method (F(2, 2) = 13.49, p < 0.001). After adjustment for multiple comparisons, there was a trend towards a main effect of Topic (F(2,2) = 3.61, p = 0.056). When we compared each pair of tutorial methods, using post hoc two-tailed t-tests, participants reported higher engagement ratings for the case vignette method, compared to the other two (p<0.001). We found no difference in engagement ratings between MR and the video-based methods. Of the included participants, 40 did not enter an engagement rating for one or more of the three methods.

Participants indicated which of the tutorial methods was their favourite. A total of 79 participants (43.4%) preferred the case vignette, 57 participants (31.3%) preferred the MR session and 28 participants (15.4%) preferred the video-based tutorial; 28 of the included participants (15.4%) did not enter an answer.

Discussion

The trend towards remote learning, increased by the COVID-19 pandemic, has driven medical schools around the world to meet students’ learning requirements using methods different from traditional classroom-based teaching approaches. Owing to advancements in visual technologies, interest in the use of augmented reality in medical education has started to increase, though research into this topic is currently limited.

Previous knowledge-attainment studies into the use of augmented and mixed reality in medical education have mainly focused on physical anatomy or neuroanatomy.¹⁶ Many studies have also sought to ascertain its effectiveness as a tool to teach technical skills, such as suturing.¹⁷ Our research focused on core medical conditions in the undergraduate medical curriculum, and examined the use of this technology as a virtual ward round and bedside teaching experience, taking it beyond a skills-based teaching tool and into a knowledge delivery tool.

This study directly compared knowledge attainment and student engagement between three forms of online learning, one of which being the use of Microsoft Hololens 2, a mixed reality headset, at the bedside. Moreover, we aimed to explore the feasibility of applying MR as a teaching tool on a large scale, following previous research demonstrating its proof-of-concept. Our research demonstrated that the implementation of MR is an effective, acceptable and feasible way of teaching clinical medicine to undergraduate students on a large scale. Our results revealed that knowledge attainment of students using the Microsoft HoloLens was equivalent compared with students taught via live online case-based and video-based tutorials.

There exists extensive research into the effectiveness of case-based and video-based learning as knowledge-imparting tools.^18,19 Our results are important as they demonstrate the potential of MR as a valid scalable method of teaching undergraduate medical students where face-to-face exposure to certain patients is logistically challenging.

Our results also show that the case vignette tutorial was regarded by students as more engaging and enjoyable than the video-based or MR formats. Furthermore, the vignette-style was the least resource-intensive form of learning. However, both video-based tutorials and MR tutorials are authentic, requiring the student to actively listen to real patients. The use of these real patients poses the challenge of information being presented in piecemeal and tangential ways (as it is in clinical practice). Information is not given to students in the concisely presented way of a case-vignette, creating more uncertainty and the need for higher cognitive processing. The information is tidily presented to students in case vignettes and does not require any further information gathering, thus may be perceived as more enjoyable to students, despite being not as authentic an experience.

This study had a large sample size, used random allocation, and is the first large-scale study of its kind to directly compare MR technology with other forms of remote learning with regards knowledge attainment. Limitations included the fact that invigilation of a remote formative examination was not possible. However, we mitigated for this by having a strict time control per question, and ensuring that the questions were constructed in such a way that they were not easy to search online. The nature of the online assessment made it possible for students to drop-out part way through. We therefore used an inclusion threshold of 50% complete data on each of the formative assessments. The sample size for analysis was lower than intended in the power analysis, due to missing data and drop-out of students during the session.

Another limitation was assessment of knowledge attainment, rather than assessment of clinical and professional skills after the sessions. One could postulate that sessions involving live patient consultations, streamed by the HoloLens, might be associated with better performance in assessments of clinical and professional skills, and this should be explored in future studies.

Our study evaluated the effectiveness of three online teaching modalities at two levels of the Kirkpatrick model: reaction (engagement) and learning (knowledge attainment).²⁰ Future studies could explore the impact of these teaching modalities on the trainees’ behaviour and the organisation for which they work. Moreover, more work needs to be done into whether XR is most effective using smaller groups, rather than large cohorts of students. With regards to engagement, further work could be conducted to explore the reasons for differing student engagement and enjoyment ratings among the three styles of teaching.

Conclusion

Our study demonstrated that short-term knowledge attainment was comparable with remote online video-based, case-vignette and MR teaching. We were able to successfully upscale MR to teach an entire year group over the course of the day. Student engagement and enjoyment was greater with simple case-vignette teaching. This may be explained by its comparatively more structured presentation and lack of information-gathering which is required in real-time and clinically authentic consultations.