The Feasibility and Effectiveness of an Online Curriculum for Virtual Onboarding/Orientation of Graduate Medical Education Learners During the COVID-19 Pandemic

Background

The education, training, and integration of new learners into Graduate Medical Education (GME) sponsoring institutions is critical to learner performance and wellbeing, the provision of high-quality patient care, research output, and patient experience. The United States Accreditation Council for Graduate Medical Education (ACGME) requires that sponsoring institutions perform orientation sessions to educate residents and fellows about ACGME common program requirements, and institution-specific resources, policies and practices, to facilitate learner integration. Historically, onboarding learners into GME sponsoring institutions has been an in-person activity.^1–3 The COVID-19 pandemic, and the need for safe social distancing (reduced human contact), caused an abrupt termination of face-to-face GME orientation globally and necessitated pivoting to online approaches to onboarding.^4,5

Residency and fellowship programs encountered significant educational, strategic, and operational challenges in the setting of the COVID-19 global misfortune.^6–13 Sponsoring institutions had limited time to pivot to online education, training, and integration of learners. Further, the dearth of best practices, guidelines, recommendations, and institutional experiential data to guide online GME learner onboarding into sponsoring institutions imposed additional challenges.

This cross-sectional study uses an electronic survey to determine the usefulness of an online GME orientation curriculum for learner onboarding during the COVID-19 pandemic. Specifically, using this survey, we sought to i) examine the satisfaction of learners who virtually integrated into our GME programs and ii) to better understand the barriers or challenges to implementing such a curriculum for future GME orientation.

Methods

We created an online onboarding curriculum from our pre-existing in-person GME orientation curriculum to facilitate the integration of our learners into our medical center. Per our standard protocol for face-to-face GME orientation, we conducted a cross-sectional survey of residents and fellow learners who underwent virtual GME orientation at our institution between June 1, 2020, and August 31, 2020.

Declaration

The Ohio State University Wexner Medical Center Institutional Review Board ethically approved the study protocol (IRB #2020E0868), which involved a retrospective analysis of the GME virtual orientation process. The Ohio State University Wexner Medical Center Institutional Review Board waived informed consent for all subjects per routine for retrospective studies. We executed all methods under our institutional guidelines and regulations for such studies.

Survey Sampling Frame

We administered our survey to all incoming residents and fellows. Our institution has representation from various training programs, including 75 ACGME residencies and fellowships; and 80 non-ACGME accredited programs. Our programs span the gamut of specialties and subspecialties, including medical, surgical, hospital-based specialties, dentistry, podiatry, and psychology.

Orientation Curriculum Development

We conducted remote orientation sessions for incoming residents and fellows using CarmenCanvas, our institution’s version of the Canvas^® Learning Management System (LMS) platform. The online orientation curriculum comprised recorded lectures that covered topics deemed essential for incoming learner onboarding using our pre-existing in-person orientation curriculum, which has undergone rigorous refinement year-over-year, as a framework. Specifically, the ACGME common program requirements,³ The Ohio State University College of Medicine, and The Ohio State University Wexner Medical Center policies and procedures served as the basis for our online orientation curriculum content. The following personnel partnered in creating the curriculum and selecting content creators/presenters: the designated institutional official, the assistant designated institutional official, the GME office director of operations management, two GME program coordinators, the GME office quality program outcome manager, the GME office finance manager, the associate director of medical education at The Ohio State University College of Medicine, and a chief resident. For reference, the collective GME experience of the curriculum design team exceeded 40 years.

We identified faculty members with substantial expertise on the selected topics to partner with GME staff to create the lectures, and we subsequently uploaded them onto CarmenCanvas. The lectures centered on themes we considered critical for trainee integration from a global GME perspective³ and our local institutional requirements, such as professionalism, crew resource management, point-of-care ultrasonography, effective patient handoffs, patient safety, quality improvement, medical information management, GME policies, clinical learning environment review, epidemiology in the COVID-19 era, and telemedicine. We have provided a complete description of each lecture’s goals and objectives in Appendix A. The lectures were pre-recorded PowerPoint presentations, ranging from ten to sixty minutes. The overall length of the curriculum was 8 hours. It was obligatory for trainees matriculating into residency or fellowship programs at The Ohio State University Wexner Medical Center to complete the orientation curriculum over ten days. Upon completing the online orientation curriculum, we invited the learners to complete a survey developed to assess the curriculum’s ability to facilitate onboarding into our sponsoring institution, although this was not mandatory.

Survey Development

The GME Office and our College of Medicine’s Office of Curriculum and Scholarship collaborated to create the survey to assess learner satisfaction, challenges, and barriers to online GME onboarding. Items on the questionnaire covered seven broad domains, including: the usefulness of the orientation curriculum to performance of trainee duties, overall quality of the GME orientation curriculum, the efficacy of the virtual orientation process for trainee learning, ease of accessing the curriculum, ease of navigation of the CarmenCanvas orientation material, organization of the orientation process and whether the lectures supported trainee learning. The survey comprised only closed-ended questions with categorical responses on a Likert scale. Specifically, the responses ranged from: “rarely” to “frequently” for the projected use of the orientation content (4-point Likert scale); “very poor” to “very good” when rating the overall quality of the virtual GME orientation (5-point Likert scale); “very ineffective” to “very effective” when ranking individual learning (6-point Likert scale). The ease of access and navigation, level of organization, and the learning support provided by the orientation material, were rated from “strongly disagree” to “strongly agree” (5-point Likert scale). We performed a reliability analysis of the survey for the following items: the Carmen course was easy to access, the Carmen course was easy to navigate, the orientation was well organized, and the lectures supported my learning. The result was a Cronbach’s alpha of 0.80, which is within an acceptable range.¹⁴ The initial survey was piloted and pretested amongst GME staff members to ensure that the included questions adequately captured the overall themes of satisfaction, challenges, and barriers to virtual orientation. The staff members also commented on the clarity and appropriateness of the survey questions. Educational and methodological experts from our institutional office of curriculum and scholarship also reviewed the survey items to ensure the questions were written under survey best practices, as described by Dillman et al¹⁵ and had face validity. We have provided a complete list of the questions and response choices in Appendix B.

Survey Administration

We invited all incoming residents and fellows to complete the survey — the only prerequisite was to complete all lectures before taking the survey. We embedded the survey into the CarmenCanvas LMS platform. Learners received up to five email reminders to complete the survey. The orientation course was available for learner access for ten days, and the learners could meet their orientation expectations on their timeline. We emailed instructions to all learners to help them navigate the system and access the lectures. The CarmenCanvas LMS platform has a built-in communication system that facilitates direct communication between learners and GME personnel to solve technical difficulties rapidly.

Statistical Analysis

CarmenCanvas recorded all online survey responses. Once completed, raw data de-identification and data management were performed using Microsoft Excel (2016). All survey respondents’ descriptive statistics, including gender, race, ethnicity, and medical specialty, were analyzed. Likewise, we analyzed the frequency of responses for each question overall. Finally, we performed multiple linear regression analysis to identify factors that may predict satisfaction with virtual GME orientation, including gender, race, and ethnicity. All analyses were 2-tailed, and we considered the results significant at p < 0.05. We performed data analyses using IBM SPSS Statistics, Version 26.0 (Armonk, NY, USA).

Results

Of 353 learners enrolled in the virtual GME orientation, 272 learners, including 191 residents and 81 fellows, responded to the survey with a response rate of 77% (Figure 1). Completing the survey was not mandatory, and the survey was available for completion after the trainees completed their orientation lectures.

Figure 1 Study flow diagram.

Survey Demographics

Characteristics of Learner Respondents

Of the 272 learners that completed the survey, 54% (n=147) were male, and 46% (n=125) were female. Most learners were incoming resident physicians at 70% (n=191), with fellows only comprising 30% (n=81) of survey respondents. Regarding ethnicity, most respondents were White at 66% (n=181); in contrast, 19% (n=52) were Asian, 5% (n=14) were Black or African American, and 4% (n=10) were Hispanic or Latino. As expected, participation was best represented in the survey by our largest program, Internal Medicine, at 33% (n=89). In contrast, smaller programs such as Neurological Surgery and Emergency Medicine / Internal Medicine were the least represented at 1% (n=2) each (Table 1).

Table 1 Demographic Characteristics of the GME Learners That Participated in the Study

Outcomes of the Survey

Usefulness of Orientation Content for Performance of Trainee Duties

At 51% (n=140), most respondents reported they would frequently use the content from the orientation to perform their clinical duties. Only a small proportion, 3% (n=8), felt it would minimally affect their performance of clinical duties (Table 2).

Table 2 Survey Questions and Responses Provided by the Learner Participants

Overall Quality of the GME Orientation

The online GME orientation lectures were described to be of “good” or “very good” quality by 77% (n=210) of respondents. Only 2% (n=4) of respondents felt that the quality was “poor” or “very poor” (Table 2).

Efficacy and Support of the Virtual Orientation for Trainee Learning

Another critical parameter of the survey was the effectiveness of our novel online approach to onboarding for trainee learning. Two hundred forty-seven trainees (89%) reported that the remote presentation of the GME orientation curriculum was “effective” for learning the orientation materials. Substantially, few learners preferred an in-person approach, which was not achievable because of the pandemic. Specifically, only 5% (n=11) of respondents felt that the virtual orientation was “ineffective” or “very ineffective” (Table 2).

Ease of Access to CarmenCanvas

The majority of learners reported accessing the lectures quickly during the orientation period — 93% (n=254) said the lectures were straight forward to access(Table 2).

Ease of Navigation of CarmenCanvas LMS

The majority of respondents felt that navigation of the curriculum was easy, and learners could complete it without difficulty, at 92% (n=250). A few learners shared some challenges, but this was a small proportion of the respondents at 2% (n=5) (Table 2).

Organization of the Orientation Process

At 91% (n=246), most respondents reported that the virtual orientation process was well organized, with no identifiable barriers to curriculum and survey access or completion (Table 2).

Support of Learning

At 87% (n=235), most respondents felt that the online GME curriculum supported their learning. Only 4% (n=10) of respondents thought the lectures did not benefit their education and training (Table 2).

Impact of Race and Gender on Survey Responses

We stratified survey responses according to gender (male and female), ethnicity (Asian, White, Black or African American, Hispanic or Latino, or “other”), and medical sub-specialty. Multiple linear regression analyses did not reveal significant differences in all survey responses based on gender, ethnicity, or medical sub-specialty.

Discussion

The integration of learners into GME sponsoring institutions is critical to the training, education, and wellbeing of learners and impacts the quality of patient care. The COVID-19 pandemic caused unprecedented disruptions, made face-to-face GME learner orientation impossible, and necessitated pivoting to online approaches to orientation curriculum delivery.¹⁶ Although other aspects of GME, such as residency education, have adopted online strategies and programs to educate learners, data describing online GME orientation curricula delivery is sparse. We report on online GME learner integration to guide GME programs globally.^17–22

Main Findings

We successfully conducted an online GME orientation and discovered this strategy to support learner integration into our sponsoring institution. Our analysis of learner feedback data indicates that virtual GME orientation using CarmenCanvas LMS is useful and provides residents and fellows with the education, training, and information needed to adjust to their new clinical learning environment. Specifically, our learners overwhelmingly appreciated the ability to complete the lectures remotely at their own pace. The learners described the orientation curriculum as beneficial to their learning and training (87%), quickly accessing the curriculum (93%), and easily navigating relevant content (92%). They appreciated the organization of the lectures (91%) and the flexibility that the virtual platform provided them with onboarding. Moreover, gender, race, ethnicity, or medical sub-specialty did not influence the survey responses. Essentially, asynchronous online learning facilitated GME orientation at The Ohio State University Wexner Medical Center at a time when face-to-face orientation was prohibited.

Online education is becoming widespread in GME and provides educators and learners with a flexible approach to teaching and training during the COVID-19 pandemic and beyond.^17–25 GME educators have leveraged various virtual educational technologies (e.g. Zoom, Microsoft Teams, CarmenCanvas LMS, etc.) to educate and train learners, although the best platform remains to be determined. Since online learning was a drastic change for GME programs during the COVID-19 pandemic, GME educators have employed surveys to evaluate the effectiveness of online learning and opportunities for improvement.²² For example. Ellison et al electronically surveyed educational leaders in US surgical training programs and discovered that less hands-on learning and increased online learning during the pandemic adversely affected the training, education (from a significant reduction in operations), and wellbeing of all learners (medical students and surgical learners).²¹ Another survey study of internal medicine and surgery residents investigating the impact of COVID-19 on resident education and wellbeing at Allegheny General Hospital reported positive outcomes such as improved learner experience with telemedicine and crisis management; however, less hands-on clinical experience and increased stress and anxiety were common drawbacks.¹⁸ Although comprehensive virtual education, including courses and conferences, has been implemented across GME training programs for residents and fellow education and training, there are no reports of using online learning for GME orientation.^17–26 We, therefore, surveyed our learners to determine the usefulness of our proprietary online orientation curriculum and opportunities for improvement — our findings demonstrate that our online curriculum and CarmenCanvas LMS are useful tools for orientation.

At The Ohio State University Wexner Medical Center, learners entering our GME training programs participate in an institutional GME orientation to acclimate them to the unfamiliar learning environment. Before the COVID-19 pandemic, we performed GME onboarding in an in-person, small-group station-based, active learning fashion, to disseminate critical principles and procedures applicable to all specialties in GME. However, the COVID-19 pandemic prevented our standard in-person orientation process because of the absolute need for social distancing.^7–13 As a result, we creatively pivoted to a virtual GME orientation process in 2020, and transitioned our refined pre-existing in-person orientation material into an online curriculum, and delivered the content to our learners using our CarmenCanvas LMS platform. We ensured alignment of the orientation curriculum with institutional and ACGME requirements. We maintained the in-person curriculum, even around procedural safety techniques and understanding the clinical learning environment, yet successfully transitioned to an online delivery platform to comply with the CDC and World Health Organization recommendations to reduce the risk of COVID-19 transmission.

We find it concerning that limited data is available to guide virtual GME orientation curriculum creation and strategies for online curriculum delivery. With the rapid mutation of the COVID-19 virus, online approaches for learner orientation may become standard practice, and GME experiences must be shared to improve this process.²⁷ To our knowledge, our manuscript is the first in the GME literature which provides a context for virtual orientation curriculum creation and delivery, in addition to learner perception of online onboarding, at our large academic center. Our manuscript highlights the value of this unique approach to learner onboarding, and GME sponsoring institutions can develop online onboarding curricula with our approach as a guide.

Lessons Learned

Our strategy did not require changing our orientation materials’ content, but only the delivery platform. We did not encounter any barriers to implementing virtual GME orientation. The virtual CarmenCanvas platform enabled the provision of GME orientation to late-arriving residents and eliminated the scheduling of makeup lectures as occurs with the in-person format. Based on survey responses, we found this change to be very successful. Of note, specific hands-on lectures (ie, invasive procedures) were presented as recorded lectures with integrated video recordings to emphasize the procedures’ key aspects, which may not be the best approach to teach hands-on procedures and techniques.

Our study helped us identify areas for improvement using this novel virtual orientation format from trainee feedback. A few trainees expressed that despite the lecture format’s informative nature (PowerPoint slides with voiceover recordings and integrated videos), an interactive design such as videoconferencing (e.g. Zoom, Webex, and Microsoft Teams), which has been reported to be effective for GME education, would be more helpful, especially for questions and answers.^22,23,25 We intend to use this approach in the future. In addition, there was significant variability in the lectures’ duration. The respondents requested standardization of the length of lectures (the majority preferred shorter, concise lectures of ~ 30 minutes duration) to improve trainee engagement. Finally, we failed to include captioning.

Study Limitations

Our study has limitations inherent to its cross-sectional, retrospective nature. However, the data presented were prospectively collected at a large GME sponsoring institution and provided insight into GME virtual onboarding. Germane to survey studies, there is a risk of response bias, but our high response rate of 77%, and the educational background of our learners, makes this unlikely. Another limitation was the lack of hands-on experiential training for specific procedures, such as airway management and invasive vascular access. We should note that these procedural training were re-incorporated using small group active learning sessions in the subsequent weeks after the vaccination of our learners — a reflection of the flexible nature of our orientation process. Using a virtual format may not provide trainees with the best skills to perform such procedures. We also did not account for trainees with learning disabilities, and we intend to determine how best to support such learners in future virtual orientations. Our study also failed to capture granular data to determine why a small subset of learners did not benefit from online GME orientation, and our future studies will address this limitation for process improvement. Finally, we failed to add captions to the recorded lectures. Our quick adaptation to online orientation did not allow us to learn and implement this best practice of online education.

Conclusions

Our study demonstrates that virtual delivery of a GME orientation curriculum is a useful method for trainee integration into GME sponsoring hospitals with minimal implementation barriers. Undoubtedly, the inability to perform face-to-face orientation was a significant roadblock, but our ability to pivot to a virtual platform made orientation achievable and safe from an infectious perspective. Virtual onboarding of GME trainees must be performed thoughtfully by aligning ACGME program requirements and local institutional guidelines, and may become standard practice as contagious diseases such as COVID-19 remain prevalent. GME programs need to share their unique virtual onboarding processes as we navigate this unprecedented time, to facilitate the creation of best practices for virtual trainee integration into GME sponsoring institutions.