Back to Journals » Advances in Medical Education and Practice » Volume 14

Knowledge Attainment and Engagement Among Medical Students: A Comparison of Three Forms of Online Learning

Authors Stackhouse AA, Rafi D , Walls R, Dodd RV, Badger K, Davies DJ, Brown CA, Cowell A, Meeran K , Halse O, Kinross J, Lupton M, Hughes EA, Sam AH

Received 10 October 2022

Accepted for publication 28 February 2023

Published 20 April 2023 Volume 2023:14 Pages 373—380


Checked for plagiarism Yes

Review by Single anonymous peer review

Peer reviewer comments 2

Editor who approved publication: Dr Md Anwarul Azim Majumder

Ashlyn A Stackhouse,1,* Damir Rafi,1,* Risheka Walls,1,2 Rebecca V Dodd,1 Kerry Badger,1 Daniel J Davies,1 Celia A Brown,3 Adrian Cowell,1 Karim Meeran,1 Omid Halse,1 James Kinross,4 Martin Lupton,1 Elizabeth A Hughes,5 Amir H Sam1

1Imperial College School of Medicine, Imperial College London, London, UK; 2Imperial College Healthcare NHS Trust, London, UK; 3Warwick Medical School, Department of Medicine, Warwick, UK; 4Department of Surgery and Cancer, Imperial College London, London, UK; 5Health Education England, London, UK

*These authors contributed equally to this work

Correspondence: Amir H Sam, Imperial College London, Charing Cross Campus, Reynolds Building, St Dunstan’s Road, London, W6 8RP, UK, Tel +442075895111, Email [email protected]

Objective: This study compared knowledge attainment and student enjoyment and engagement between clinical case vignette, patient-testimony videos and mixed reality (MR) teaching via the Microsoft HoloLens 2, all delivered remotely to third year medical students. The feasibility of conducting MR teaching on a large scale was also assessed.
Setting & Participants: Medical students in Year 3 at Imperial College London participated in three online teaching sessions, one in each format. All students were expected to attend these scheduled teaching sessions and to complete the formative assessment. Inclusion of their data used as part of the research trial was optional.
Primary and Secondary Outcome Measures: The primary outcome measure was performance on a formative assessment, which served to compare knowledge attainment between three forms of online learning. Moreover, we aimed to explore student engagement with each form of learning via a questionnaire, and also feasibility of applying MR as a teaching tool on a large scale. Comparisons between performances on the formative assessment between the three groups were investigated using a repeated measures two-way ANOVA. Engagement and enjoyment were also analysed in the same manner.
Results: A total of 252 students participated in the study. Knowledge attainment of students using MR was comparable with the other two methods. Participants reported higher enjoyment and engagement (p< 0.001) for the case vignette method, compared with MR and video-based teaching. There was no difference in enjoyment or engagement ratings between MR and the video-based methods.
Conclusion: This study demonstrated that the implementation of MR is an effective, acceptable, and feasible way of teaching clinical medicine to undergraduate students on a large scale. However, case-based tutorials were found to be favoured most by students. Future work could further explore the best uses for MR teaching within the medical curriculum.

Keywords: medical education, augmented reality, online learning


Over the past decade, remote learning has gained increased prominence in higher education.1 Undergraduate medicine, which has gradually been evolving from lecture-based curricula towards more integrated, collaborative approaches,2,3 has also followed this trend. The COVID-19 pandemic has accelerated the pace of change and necessitated a faster transformation to remote learning in order to meet complex learning objectives.4

One important method of delivering remote medical teaching has been the use of case-based learning. Students are presented with a short text-based clinical vignette and discuss the case with a tutor remotely. This approach uses authentic patient cases to provide a relevant and realistic experience to students, aiming to connect existing theory to clinical practice.5 Studies have demonstrated its benefits, which include improving knowledge attainment, enhancing patient care outcomes, and yielding student satisfaction.6,7

As an extension to this, medical educational multimedia such as video have also become a prominent method of delivering case-based teaching, as opposed to using written vignettes alone. Video-based case materials have been shown to be an engaging way of learning, although some studies have identified limitations, such as a reduction in deep thinking compared with written vignettes.8,9

A more novel and less developed form of remote learning is the use of Extended Reality (XR). This is an umbrella term encompassing virtual reality (VR), augmented reality (AR) and mixed reality (MR). In contrast to VR, which involves a completely digital environment, AR is a form of technology which superimposes digital interfaces upon the physical world. MR, an extension to this, enables the user to directly interact with digital holograms placed in the real environment. Advances in visual technology over recent years have resulted in an increase in the use of XR in medical education. The fields of anatomy, surgical simulation and telementoring have started to integrate XR into their training programmes.10–12

One pioneering XR device is Microsoft HoloLens 2, released in November 2019. This mixed reality technology creates an immersive environment using head-mounted glasses with a built-in camera. Previous research has demonstrated its feasibility, acceptability, and effectiveness in delivering remote-access bedside teaching, with the use of patient-based cases, to clinical-year medical students.13 The headset, worn by the tutor, livestreams their environment to students remotely, allowing them to view and interact with both the tutor and the patient. Moreover, the tutor can superimpose holograms, such as scans, onto the live environment as an adjunct to the session. Although the processes of delivering remote clinical teaching using MR have been described, adequate work has yet to demonstrate its utility on a large scale and compare its effectiveness directly with other forms of remote learning.

This study aimed to directly compare three forms of remote learning: clinical vignette, video-based case and MR teaching, via the Microsoft Hololens 2, delivered remotely to a cohort of third year medical students at one UK medical school. We set out to evaluate the feasibility of conducting MR teaching on a large scale, and to establish how this form of teaching compares to case-based learning and video-based learning with regards to student knowledge attainment and engagement.

Materials and Methods

Study Population

Medical students in Year 3 at Imperial College London participated in three online teaching sessions, one in each format. All students were expected to attend the teaching sessions and to complete the formative assessment, as the session was part of their scheduled learning. The topics taught aligned directly with the students’ curriculum. Inclusion of their data used as part of the research trial was optional.

The cohort were divided into a morning and afternoon session based on their current placement locations in order to minimise disruption to their clinical attachments. Students were randomised in three equal subgroups using the RANK-L method on Microsoft Excel.

All participants received an information sheet prior to the study with detailed information about the study (Appendix 1). At the beginning of the session, a researcher explained the information to participants and allowed time for answering any questions. Consent to use answers from the formative assessment was gained via Mentimeter at the beginning of the formative assessment. Students were informed they would be able to participate in all aspects regardless of whether they had provided consent for the study. All participants were able to withdraw their consent up to 2 weeks after the completion of data collection, after which all answers were anonymised.

Patients were consented to partake in the MR sessions using pre-approved Imperial College GDPR protocol and were informed that sessions would be recorded. Curtains were pulled around patients and signs placed in surrounding areas to ensure privacy for the patient and prevent other ward patients or staff being inadvertently filmed.


Three standardised scripts for each of the tutorial topics were devised; Painful Hands, Chest Pain and Breathlessness. Each script was tailored to the requirements of the specific teaching method, but contained the same framework, depth and detail to ensure congruence between the material being taught. Each script was rehearsed with individual tutors on two occasions to ensure homogeneity in teaching styles across all three tutorials.

Three simultaneous online teaching sessions were delivered; one HoloLens tutorial, one video-based tutorial, and one vignette-based tutorial – all delivered via Microsoft Teams. The tutorials were on the topics of chest pain, breathlessness, and painful hands. Each tutorial lasted 25 minutes and students rotated from one to the next, with 5-minute breaks between each tutorial, until they had experienced all three styles. There were approximately 42 students in each session.

The MR session featured a live video stream of a tutor on a hospital medical ward, interacting with and examining a patient, whilst also displaying holograms on the screen such as investigation results. Students were able to watch this session directly on their electronic devices and interact in real time with the tutor and patient using the chat and microphone facilities on Microsoft Teams.

The video-based tutorial consisted of students watching a 2–3 minute pre-recorded patient-testimony video taken from the learning platform Speaking Clinically, a national resource provided by the Medical Schools Council of video archives of real patient encounters. The tutor then guided students through a teaching session related to the clinical condition that the patient described in the video.

The vignette tutorial featured a written case, which the students read in the first 2–3 minutes of the session, and then analysed with the tutor.

All three tutorials were designed to be interactive in nature, with questions and discussions actively encouraged.

A formative assessment was then developed which featured very short answer (VSA) questions.14,15 There were 60 questions in total, with 20 on each of the three topics being taught. The questions were written by the researchers with guidance from expert question-writers at Imperial College London. Questions were standard-set by a select group of six medical educators using the Ebel method. An inclusion threshold of 50% of the items per topic was set; students who had answered at least 10 out of 20 items in a particular topic were included. This was done in order to prevent bias caused by dropout. If the inclusion criteria were met, blank answers were marked as wrong.

A questionnaire was also developed to assess engagement and enjoyment of the tutorials. Participants were asked to rate their enjoyment and engagement for each teaching method using a 5-point Likert scale.


Immediately after completion of the three teaching sessions, all students joined a single Microsoft Teams call and participated in the online formative assessment and enjoyment and engagement questionnaire via the quiz platform Mentimeter. The formative assessment was approximately 1 hour in length and students completed these questions remotely under usual exam-conditions and time-constraints.

All students were asked to rate their enjoyment of and engagement with each form of online session on a Likert scale from 1 to 5: the exact questions are outlined in Appendix 2.

Marking and Feedback

VSAs were marked electronically and verified by two researchers.14 A third clinician was available to arbitrate any queries.

Once all students had completed the assessment, the formative paper with answers and explanations was made available through direct email to the students.

Power Analysis

We performed an a priori power analysis to determine the appropriate sample size for the study. Our primary outcome measure was performance on the formative assessment. The expected mean (SD) for the formative assessment was 65.7% (16.5%). We performed the power analysis for four levels of effect size, as calculated by Cohen’s d (0.3, 0.4, 0.5 and 0.7) for a one-way ANOVA comparing the means of the three groups, with one group performing better and the other two groups performing equal to the expected mean. We aimed for a minimum power value of 0.8. Table 1 shows the results of the power analysis for the four levels of effect size.

Table 1 Results of Power Analysis for the Four Levels of Effect Size, Alpha of X

We opted for a target sample size of 100 per group, which gave an over 80% power to detect differences in the mean scores on the formative assessment of effect sizes of 0.4 or above (Cohen’s d).

Statistical Analysis

The study administration team produced an Excel file containing answers and scores for each student for each question. Each student was allocated a numerical code to ensure anonymity. The data were transferred into R version 4.1.0 for analysis. Comparison between performances on the formative assessment by tutorial method were investigated using a repeated measures two-way ANOVA with factors “Topic” (chest pain, breathlessness, painful hands) and “Tutorial Method” (“MR”, “video-based” or “case vignette”) and their interactions. Where the ANOVA main effects were significant, we performed pairwise comparisons with paired t-tests. We used the Holm correction to adjust p-values for multiple comparisons in the ANOVAs and in post hoc tests. Engagement and enjoyment were also analysed in the same manner.

Patient and Public Involvement

This study was conducted with medical students as the participants. Patients were involved only for the bedside teaching sessions themselves. We thoroughly briefed patients beforehand and discussed the nature of our study, as well as providing them with an information sheet and the opportunity to ask any questions. We complied and adhered to all GDPR policies of Imperial College London School of Medicine, and Imperial College London NHS Trust.

Results will be disseminated to study participants (medical students) via Professor Amir Sam, Head of Imperial College School of Medicine.


A total of 324 students participated in the teaching sessions; of those, 252 students (78%) consented to participate in the study. Students in each session were randomised into three subgroups (see Table 2 for combinations of tutorial methods and topics per group). A total of 84, 82 and 86 participants from groups A, B and C, respectively, entered at least one answer on the formative assessment.

Table 2 Means and Standard Deviations of Scores on the Formative Assessment for Each Topic and Tutorial Method Combination

Clinical Knowledge Attainment and Performance

Table 2 shows the combinations of tutorial methods and topics per group and the means and standard deviations on the assessment of each tutorial by topic combination. Participants with over 50% missing data on any topic were removed from the analysis. In Groups A, B and C, respectively, 60 (71.4%), 62 (78%) and 60 (69.8%) had complete data for at least 50% of the assessments and were included in the analyses, totaling 182 participants. For these participants, missing answers were treated as incorrect. Cronbach’s alpha was calculated for each topic, collapsed over tutorial method. Cronbach’s alpha for Chest Pain was 0.57, for Breathlessness was 0.64 and for Painful Hands was 0.71.

Mean knowledge attainment scores differed across the three tutorial methods, shown by a significant main effect of Tutorial Method (F(2, 2) = 6.31, p = 0.004). When we compared each pair of tutorial methods using post hoc two-tailed t-tests, there was a trend towards higher scores on the video-based tutorial than the other two methods (p < 0.1), which did not survive correction for multiple comparisons.

Student Perceptions, Engagement and Enjoyment

Table 3 shows the means and standard deviations and medians and interquartile ranges of ratings of enjoyment and engagement for each tutorial by topic combination).

Table 3 Means, Standard Deviations and Medians and Interquartile Ranges of Engagement and Enjoyment Ratings (from 1 to 5, Where Higher Scores Indicate Higher Engagement/Enjoyment) for Each Combination of Topic and Method

Mean enjoyment ratings differed across the three tutorial methods, shown by a significant main effect of Tutorial Method (F(2, 2) = 15.68, p < 0.001). The main effect of Topic was not significant. When we compared each pair of tutorial methods using post hoc two-tailed t-tests, participants reported higher enjoyment for the case vignette method, compared to the other two (p< 0.001). We found no difference in enjoyment ratings between MR and the video-based methods. Of the included participants, 31 did not enter an enjoyment rating for one or more of the three methods.

Mean engagement ratings differed across the three tutorial methods, shown by a significant main effect of Tutorial Method (F(2, 2) = 13.49, p < 0.001). After adjustment for multiple comparisons, there was a trend towards a main effect of Topic (F(2,2) = 3.61, p = 0.056). When we compared each pair of tutorial methods, using post hoc two-tailed t-tests, participants reported higher engagement ratings for the case vignette method, compared to the other two (p<0.001). We found no difference in engagement ratings between MR and the video-based methods. Of the included participants, 40 did not enter an engagement rating for one or more of the three methods.

Participants indicated which of the tutorial methods was their favourite. A total of 79 participants (43.4%) preferred the case vignette, 57 participants (31.3%) preferred the MR session and 28 participants (15.4%) preferred the video-based tutorial; 28 of the included participants (15.4%) did not enter an answer.


The trend towards remote learning, increased by the COVID-19 pandemic, has driven medical schools around the world to meet students’ learning requirements using methods different from traditional classroom-based teaching approaches. Owing to advancements in visual technologies, interest in the use of augmented reality in medical education has started to increase, though research into this topic is currently limited.

Previous knowledge-attainment studies into the use of augmented and mixed reality in medical education have mainly focused on physical anatomy or neuroanatomy.16 Many studies have also sought to ascertain its effectiveness as a tool to teach technical skills, such as suturing.17 Our research focused on core medical conditions in the undergraduate medical curriculum, and examined the use of this technology as a virtual ward round and bedside teaching experience, taking it beyond a skills-based teaching tool and into a knowledge delivery tool.

This study directly compared knowledge attainment and student engagement between three forms of online learning, one of which being the use of Microsoft Hololens 2, a mixed reality headset, at the bedside. Moreover, we aimed to explore the feasibility of applying MR as a teaching tool on a large scale, following previous research demonstrating its proof-of-concept. Our research demonstrated that the implementation of MR is an effective, acceptable and feasible way of teaching clinical medicine to undergraduate students on a large scale. Our results revealed that knowledge attainment of students using the Microsoft HoloLens was equivalent compared with students taught via live online case-based and video-based tutorials.

There exists extensive research into the effectiveness of case-based and video-based learning as knowledge-imparting tools.18,19 Our results are important as they demonstrate the potential of MR as a valid scalable method of teaching undergraduate medical students where face-to-face exposure to certain patients is logistically challenging.

Our results also show that the case vignette tutorial was regarded by students as more engaging and enjoyable than the video-based or MR formats. Furthermore, the vignette-style was the least resource-intensive form of learning. However, both video-based tutorials and MR tutorials are authentic, requiring the student to actively listen to real patients. The use of these real patients poses the challenge of information being presented in piecemeal and tangential ways (as it is in clinical practice). Information is not given to students in the concisely presented way of a case-vignette, creating more uncertainty and the need for higher cognitive processing. The information is tidily presented to students in case vignettes and does not require any further information gathering, thus may be perceived as more enjoyable to students, despite being not as authentic an experience.

This study had a large sample size, used random allocation, and is the first large-scale study of its kind to directly compare MR technology with other forms of remote learning with regards knowledge attainment. Limitations included the fact that invigilation of a remote formative examination was not possible. However, we mitigated for this by having a strict time control per question, and ensuring that the questions were constructed in such a way that they were not easy to search online. The nature of the online assessment made it possible for students to drop-out part way through. We therefore used an inclusion threshold of 50% complete data on each of the formative assessments. The sample size for analysis was lower than intended in the power analysis, due to missing data and drop-out of students during the session.

Another limitation was assessment of knowledge attainment, rather than assessment of clinical and professional skills after the sessions. One could postulate that sessions involving live patient consultations, streamed by the HoloLens, might be associated with better performance in assessments of clinical and professional skills, and this should be explored in future studies.

Our study evaluated the effectiveness of three online teaching modalities at two levels of the Kirkpatrick model: reaction (engagement) and learning (knowledge attainment).20 Future studies could explore the impact of these teaching modalities on the trainees’ behaviour and the organisation for which they work. Moreover, more work needs to be done into whether XR is most effective using smaller groups, rather than large cohorts of students. With regards to engagement, further work could be conducted to explore the reasons for differing student engagement and enjoyment ratings among the three styles of teaching.


Our study demonstrated that short-term knowledge attainment was comparable with remote online video-based, case-vignette and MR teaching. We were able to successfully upscale MR to teach an entire year group over the course of the day. Student engagement and enjoyment was greater with simple case-vignette teaching. This may be explained by its comparatively more structured presentation and lack of information-gathering which is required in real-time and clinically authentic consultations.

Ethical Approval

The study was approved by the Education Ethics Review Process at Imperial College London. Ethics approval number: EERP2021-015a.


We would like to thank the Faculty of Education Office (Medicine) at Imperial College London for their involvement in the organisation and delivery of the teaching sessions as part of the study, and all students who participated. We also thank Health Education England for their support in conducting this study.

Author Contributions

All authors made substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; took part in drafting the article or revising it critically for important intellectual content; agreed to submit to the current journal; gave final approval of the version to be published; and agree to be accountable for all aspects of the work.


There is no funding to report.


Professor Celia Brown reports grants from NIHR ARC West Midlands, during the conduct of the study. Dr James Kinross holds grants from NIHR i4i, Innovate UK, H2020, CRUK and EPSRC. He is a Co-founder of the MR company Medical iSight. He is a shareholder of Surgease Ltd and Concentric Ltd. He has received Consultancy fees from Ethicon. The authors report no other conflicts of interest in this work.


1. Pei L, Wu H. Does online learning work better than offline learning in undergraduate medical education? A systematic review and meta-analysis. Med Educ Online. 2019;24(1):1666538. doi:10.1080/10872981.2019.1666538

2. Khogali SE, Davies DA, Donnan PT, et al. Integration of e-learning resources into a medical school curriculum. Med Teach. 2011;33(4):311–318. doi:10.3109/0142159X.2011.540270

3. Taveira-Gomes T, Ferreira P, Taveira-Gomes I, Severo M, Ferreira MA. What are we looking for in computer-based learning interventions in medical education? A systematic review. J Med Internet Res. 2016;18(8):e204. doi:10.2196/jmir.5461

4. Sam AH, Millar KR, Lupton MG. Digital clinical placement for medical students in response to COVID-19. Acad Med. 2020;95:1126. doi:10.1097/ACM.0000000000003431

5. Thistlethwaite JE, Davies D, Ekeocha S, et al. The effectiveness of case-based learning in health professional education. A BEME systematic review: BEME Guide No. 23. Med Teach. 2012;34(6):e421–44. doi:10.3109/0142159X.2012.680939

6. McLean SF. Case-based learning and its application in medical and health-care fields: a review of worldwide literature. J Med Educ Curric Dev. 2016;3:JMECD–S20377. doi:10.4137/JMECD.S20377

7. Nicklen P, Keating JL, Maloney S. Student response to remote-online case-based learning: a qualitative study. JMIR Med Educ. 2016;2(1):e5025.

8. Basu Roy R, McMahon GT. Video‐based cases disrupt deep critical thinking in problem‐based learning. Med Edu. 2012;46(4):426–435. doi:10.1111/j.1365-2923.2011.04197.x

9. De Leng BA, Dolmans DH, Van de Wiel MW, Muijtjens AM, Van Der Vleuten CP. How video cases should be used as authentic stimuli in problem‐based medical education. Med Edu. 2007;41(2):181–188. doi:10.1111/j.1365-2929.2006.02671.x

10. Borgersen NJ, Naur TM, Sørensen SM, et al. Gathering validity evidence for surgical simulation: a systematic review. Ann Surg. 2018;267(6):1063–1068. doi:10.1097/SLA.0000000000002652

11. Ekstrand C, Jamal A, Nguyen R, Kudryk A, Mann J, Mendez I. Immersive and interactive virtual reality to improve learning and retention of neuroanatomy in medical students: a randomized controlled study. CMAJ Open. 2018;6(1):E103. doi:10.9778/cmajo.20170110

12. Wang S, Parsons M, Stone-McLean J, et al. Augmented reality as a telemedicine platform for remote procedural training. Sensors. 2017;17(10):2294. doi:10.3390/s17102294

13. Bala L, Kinross J, Martin G, et al. A remote access mixed reality teaching ward round. Clin Teach. 2021;18:386–390. doi:10.1111/tct.13338

14. Sam AH, Field SM, Collares CF, et al. Very‐short‐answer questions: reliability, discrimination and acceptability. Med Edu. 2018;52(4):447–455. doi:10.1111/medu.13504

15. Sam AH, Hameed S, Harris J, Meeran K. Validity of very short answer versus single best answer questions for undergraduate assessment. BMC Med Educ. 2016;16(1):1–4. doi:10.1186/s12909-016-0793-z

16. Stepan K, Zeiger J, Hanchuk S, et al. Immersive virtual reality as a teaching tool for neuroanatomy. Int Forum Allergy Rhinol. 2017;7(10):1006–1013. doi:10.1002/alr.21986

17. Rojo A, Raya L, Sanchez A. A novel mixed reality solution based on learning environment for sutures in minor surgery. Appl Sci. 2021;11(5):2335. doi:10.3390/app11052335

18. Hurtubise L, Martin B, Gilliland A, Mahan J. To play or not to play: leveraging video in medical education. J Grad Med Educ. 2013;5(1):13. doi:10.4300/JGME-05-01-32

19. Bi M, Zhao Z, Yang J, Wang Y. Comparison of case-based learning and traditional method in teaching postgraduate students of medical oncology. Med Teach. 2019;41(10):1124–1128. doi:10.1080/0142159X.2019.1617414

20. Yardley S, Dornan T. Kirkpatrick’s levels and education ‘evidence’. Med Edu. 2012;46(1):97–106. doi:10.1111/j.1365-2923.2011.04076.x

Creative Commons License © 2023 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.