Incorporation of Comprehensive Examinations for a Time-Efficient Remediation Method

Introduction

The preclinical curriculum is considered the foundation of medical education. In this portion of the curriculum, students learn basic sciences and imperative skills that prepare these future doctors to apply scientific knowledge in patient healthcare.¹ Medical programs have a duty to concentrate efforts on developing processes for the identification and assessment of student performance.^2,3 This duty includes monitoring metrics that forecast future successes and failures. Early use of this information is important because it allows faculty to address learning deficiencies and apply an intervention.^4,5 Furthermore, it ultimately increases the opportunity of favorable United States Licensing Exam (USMLE) Step 1 and Step 2 CK outcomes.⁶

Remediation in medical education should be a focused, time-limited, and highly structured process designed to address student deficiencies and to allow students to prove content competency before progressing to the next phase in the curriculum.^3,7 It is also institution-specific; A wide variability exists in the methods used to affect remediation in different medical schools across the country. Limited research has been conducted in this field and no consensus remains on a standard method.⁸

Literature suggests that remediation should include assessment tools for identifying deficiencies, individualized instruction, and feedback followed by reassessment.^9,10 The protocol implemented at the University of Colorado School of Medicine (UCSOM) is some of the most clearly described.^11,12 Underperforming students were identified and referred to the remediation program. A learner specific ten-step remediation plan focusing on clinical and reasoning skills was developed and implemented. The outcome of this plan was then assessed independently by either another faculty or a senior trainee.

Though the UCSOM remediation process showed benefit to students, the hours spent dedicated to execute this remediation is found to be excessive.¹² A commonly shared sentiment among medical educators involves the intensive amount of time spent helping struggling learners.⁷ Additionally, a disproportionate amount of time is being spent on a relatively small number of students. These time demands have potential impact on protocol sustainability, leading to the re-evaluation of the system used at Kirk Kerkorian School of Medicine - UNLV (KSOM).

Formerly, the KSOM remediation process was a paper-based approach, where an individual faculty-written assignment was administered and student results were analyzed to evaluate for competency. A new approach is to use comprehensive final examinations for remediation, which are relatively uncommon in medical education.¹³ KSOM implemented this change as a way to preserve faculty time and provide students an opportunity to demonstrate competency of previously failed content.

In this study, we analyzed the faculty time utilized with the current remediation process used at KSOM and compared this data with the former process. We hypothesized that the transition to the comprehensive end-of-semester final examination is a more time-efficient method of student remediation than the previous method.

Materials and Methods

At KSOM preclinical students are required to demonstrate content competency via supervised, web-based National Board of Medical Examiners (NBME) examinations compiled through the NBME Customized Assessment Service. These exams include faculty-selected multiple choice questions from the NBME question bank. For the Class of 2021, KSOM preclinical students were able to demonstrate content competency by exceeding a 70% passing threshold on individual system-based exams throughout the curriculum. The passing threshold for Class of 2024 was raised to 75% to better identify students at-risk of poor future exam performance. Failure to meet the passing standard requires individual students to remediate.

Previously, the remediation process was performed through paper-based assignments completed during intersession weeks between curricular blocks that could be used for rest or remediation (Figure 1). In this setting, faculty would review individual student exam performances and collaborate with them to address content and/or test-taking deficiencies. Written assignments were then created based on the identified areas of weakness for student completion under faculty supervision, and feedback was provided based on the results. Faculty time estimates were provided by faculty who were involved in this remediation process.

Figure 1 Schematic of the first two years of the Doctorate of Medicine program for the Class of 2021 at the Kirk Kerkorian School of Medicine at UNLV.

KSOM implemented a new approach for the Class of 2023, using end-of-semester comprehensive final examinations for remediation. These are supervised, web-based examinations administered to all students at the end of each semester, consisting of a sufficient number of faculty-selected NBME questions relating to each organ system covered in the semester (Figure 2). Students who under-perform on individual system examinations are given a second opportunity to demonstrate content competency on the comprehensive exam by successfully answering 75% of the system-specific questions. Students who still did not exhibit competency through the comprehensive end-of-semester final examination were then analyzed to identify areas of deficiency, and an additional, individualized NBME examination was administered. Total faculty time estimates for creating additional exams and working with students were retrospectively provided by faculty who do most of the remediation.

Figure 2 Schematic of the first two years of the Doctorate of Medicine program for the Class of 2024 at the Kirk Kerkorian School of Medicine at UNLV.

Deidentified student performance data was obtained from institutional databases at KSOM with an approved Institutional Review Board (IRB) protocol. The first two years of the preclinical curriculum for the graduating class of 2021 and 2024 were the focus of this study. Data from Class of 2022 and 2023 were excluded from the analysis secondary to inconsistent implementation of a combination of both remediation approaches and curricular limitations secondary to COVID-19, respectively.

Results

For the class of 2021, three transitional weeks were held during the fall semester of 2017 and two during the spring of 2018. Each transitional week required two faculty members, necessitating 40 hours from each member per transitional week. In total, 400 hours of faculty time were dedicated to remediating all students who failed at least one examination (Table 1).

Table 1 Remediation Time Allocation Using the Paper-Based Method

For the class of 2024 there were two initial remediation opportunities through comprehensive finals: one in fall 2020, and one in spring 2021. During the fall semester of 2020, 26 students did not meet the passing threshold for one or more of the system examinations covered. 22 students successfully demonstrated proficiency in their previously deficient subject using the end-of-semester comprehensive examination while four students needed additional remediation through an additional individual NBME custom examination. An average of three hours per student was dedicated by a faculty member for exam performance analysis and individualized coaching. Four individual remediation examinations were needed to be constructed with an average time of three hours per exam creation totaling 12 hours. During this semester, the cumulative number of faculty hours utilized to remediate students was 27 hours.

Sixteen students yielded unsatisfactory performance in at least one system examination in the spring semester of 2021. Fourteen of those met the competency standard after taking the end-of-semester comprehensive exams, while the remaining two needed further remediation. A sum of six hours was dedicated to examination performance analysis and individualized coaching. Additionally, the course director spent nine hours formulating three individual examinations. An aggregate of 18 faculty hours was used to remediate students during this semester.

The total amount of faculty time committed to remediating students for the Class of 2024 was 45 hours. Between the 42 students who failed an examination at least once across the span of both semesters, an average of 1.07 hours of faculty time was dedicated to each student to completely remediate them. Transitioning from paper-based remediation to the utilization of end-of-semester comprehensive examinations also yielded 355 hours less faculty time dedicated to remediating students (Table 2).

Table 2 Remediation Time Allocation with Inclusion of the Comprehensive Examinations

Discussion

KSOM used cumulative final exams as another attempt to demonstrate competency on basic science NBME examination in which a student previously underperformed. Those who demonstrated proficiency on the component of the final exam pertaining to the subject area that was previously failed avoided the need for a more formal and time-consuming remediation process. The KSOM Class of 2024 results showed that students were able to avoid additional remediation for 36 of 42 failed exams. This yielded significant savings in terms of faculty time spent on remediation. Faculty spent 45 total remediation hours for the Class of 2024, compared to 400 hours remediating using the paper-based assignment method for the Class of 2021. Based on these findings, we were able to show a significant decrease in faculty time spent on remediation. With the transition to comprehensive end-of-semester final examinations, 355 hours were saved. Results also showed that faculty used an average 1.07 hours/student with end-of-semester comprehensive examinations. For external comparison, a mean of 10 faculty hours/student is utilized at UCSOM.⁶

Saving faculty time is beneficial in many ways.¹⁴ Importantly, increased time can be spent improving the overall curriculum for all students rather than on a limited number of students, addressing a common complaint amongst administrative members.⁷ Some of the areas of focus include researching, student mentoring, creating practice test questions, and offering guidance to students before sitting for examinations. These opportunities could potentially contribute to curriculum efficiency, as well as the experience and outcomes of medical education.

In addition to saving faculty time, implementation of this testing method created a potentially more sustainable and efficient remediation model for KKSOM due to its replicability and ability to target underperforming students. End-of-semester comprehensive final examinations serve as an opportunity for faculty to analyze longitudinal performance factors to screen students at high-risk for poor performance on future assessments.¹⁵ Furthermore, identifying these longitudinal trends allows students to collaborate closely with trained faculty to improve areas of weakness and creation of individualized plans to reach a competency standard.^16,17

We recognize several limitations in this study. First, the recent implementation of end-of-semester final examinations limited our data to the analysis of a single cohort for comparison to a previous cohort. Furthermore, due to the limited research in this field and lack of consensus on the most effective remediation protocol, we were only able to compare our data with studies from UCSOM.^11,12 This lack of program comparison could impact generalizability as curriculum and protocols vary vastly across medical schools.

While significant time is saved by testing for remediation, further evaluation is required to evaluate long-term effectiveness on content competency and would further be strengthened by a multi-institutional comparison. It would also be beneficial to evaluate the amount of student time utilized in remediation compared across institutions. Finally, a future comparison of faculty hours to student hours should be conducted to determine the effects of saved faculty time on student time. It would also be of value to explore mental health concerns raised by administration of comprehensive final exams.

Conclusion

With preclinical education serving as the foundation for medical education, it is important to consistently reevaluate the ways we assess and remediate students. This process is often very time intensive. By adopting an end-of-semester final examination, KSOM has been able to save considerable faculty time. This time saved allows them to focus on improving the overall curriculum for all students with the goal of improving individual student outcomes.