Perceptions of the Emergency Medicine Resident Selection Process by Program Directors Following the Transition to a Pass/Fail USMLE Step 1

Introduction

Beginning January 26th, 2022, the National Board of Medical Examiners (NBME) transitioned scoring of Step 1 of the United States Medical Licensing Examination (USMLE) from a 3-digit score to pass/fail.¹ Historically in emergency medicine (EM), the Step 1 score has been weighted heavily by program directors (PDs) as one of the most important metrics in assessing a medical student’s competitiveness for EM residency programs, alongside standardized letters of evaluation (SLOE).² In the United States (US), the selection process for residency programs focuses on locally determined competitiveness to compare applicants such as past academic records, standardized testing scores, resumes, and personal statements. Other countries, such as the United Kingdom and Australia, have adapted a more holistic selection criteria which encompasses a wide range of written and observed formats such as multiple mini-interviews, situational judgement tests, and clinical problem-solving tests to better assess an applicant’s competitiveness.³ The NBME proposed the transition to a pass/fail Step 1 with the hopes of reducing the emphasis on Step 1 performance in residency screening, accelerating research regarding USMLE performance in predicting a resident’s clinical and board examination performance, and minimizing racial demographic differences within medical boards performance.⁴ While the US has not yet adapted the methods of holistic selection criteria as seen in other countries, the transition to a pass/fail Step 1 is the first step in adapting a more holistic approach to the residency application process.

In the past, Step 1 has been an important milestone in the medical education pathway but has been known to create a significant amount of physical, mental, and financial stress for students during their dedicated preparation time. One study suggested that the pressure to perform well on Step 1 led to 77% of surveyed students feeling “burned out” and 72% of surveyed students reporting a decrease in their overall quality of life following the preparation process.⁵ Other studies report that, on average, students spend more than 10 hours per day for over 4 weeks preparing for the Step 1 examination.⁶ Despite this time spent studying, Step 1 scores have not been shown to predict a resident’s performance in EM residency.⁷

Demographic differences have also been observed to play a part in USMLE performance. In the past, the emphasis on Step 1 scores during the residency selection process has been argued as being an objective, standardized measure. However, Step 1 scores have been shown to disadvantage female and underrepresented minority applicants.⁸ With a transition to pass/fail Step 1 scoring, the residency selection processes may adopt a more holistic selection process, enhancing the diversity within the physician workforce.⁹

Many hope that the transition to a pass/fail Step 1 will lead to decreased mental and physical stress on students. Despite the mitigation of student stress, PDs may be concerned at the prospect of having one less piece of objective data to compare prospective applicants. Patel et al analyzed the differences between procedural and nonprocedural specialties and how they ranked attributes before and after the implementation of Step 1 pass/fail. The study concluded that while there were individual differences placed on residency applicant characteristics by procedural and non-procedural specialty PDs, there was little change in rank order of applicant attributes when compared before and after the transition to pass/fail Step 1. Nonprocedural specialties considered clerkship grades and a dean’s letter most important when compared to procedural specialties who considered Step 1 score and letters of recommendation most important prior to transition to a pass/fail Step 1.¹⁰ It is unclear how this transition to a pass/fail Step 1 exam will affect how EM, a nonprocedural specialty, and how EM PDs will assess applicants’ competitiveness to residency programs. The purpose of this study was to evaluate the perceptions of EM PDs on the transition to a pass/fail USMLE Step 1 exam. We predicted that other metrics such as USMLE Step 2 Clinical Knowledge (CK) score, extracurricular activities, standardized letters of evaluation (SLOEs), research involvement, and community service would become more important in the EM resident selection process following the transition to a pass/fail USMLE Step 1. Furthermore, we hoped to elicit the opinions of EM PDs on the ability of USMLE examinations to predict resident clinical and board examination performance.

Methods

An IRB-approved secure survey consisting of multiple choice and ranking questions was created using Qualtrics software and sent out to the PDs of all 282 EM residency programs accredited by the Accreditation Council for Graduate Medical Education in the United States. The IRB was approved through the University of Alabama at Birmingham Institutional Review Board. The survey was sent to PDs via email. This PD email list was compiled by using the American Medical Association’s Residency and Fellowship Database. The initial email to residency PDs was followed with two reminders, one sent two weeks after the original email was sent and another four weeks after the original email was sent. Each PD consented to participate in this study before completing the survey. The ranking questions focused on assessing each program’s current resident selection practices and then asked them to compare their current practices with how they expected their selection criteria to change for future applicants following the transition to pass/fail Step 1. The ranking questions were comprised of 16 attributes that each PD ranked in importance based on their own resident selection process. PDs were also asked a series of multiple-choice questions to determine their level of confidence in the ability of USMLE Step 1 and Step 2 CK to predict a student’s clinical skills and ability to succeed in residency. Following the data collection period, R studio and MATLAB were used for statistical analysis and a p-value <0.05 was considered significant. Chi squared tests were used to analyze the results of the multiple-choice questions in R Studio using Tidyverse and knitr packages, and two-tailed paired T-tests were used to analyze ranking questions in MATLAB (version 9.8).

Results

The multiple-choice portion of the survey was completed by 57 (20.21%) EM PDs (Table 1). Thirty-four of the responses came from academic programs, 14 were academic-affiliated, and 9 were community-based programs. Of the 57 program directors who completed the multiple-choice portion of the survey, 6 (10.53%) of the responses came from a PD affiliated with top 15 NIH funded program. PDs agreed that both USMLE Step 1 and Step 2 CK adequately predicted a resident’s ability to pass the EM board certification exam at the end of residency (52.6% and 73.7% respectively, p<0.01). When asked if Step 1 and Step 2 CK are accurate predictors of a resident’s ability to perform clinically within EM, only 10.5% of PDs answered “yes” to Step 1 being predictive (p<0.01), compared to 31.6% for Step 2 CK (p=0.33). When asked if a medical student’s class rank will be considered more following the transition to Step 1 Pass/fail, most respondents (52.6%, p<0.01) believe that a medical student’s rank will be considered more. With regards to the transition to pass/fail Step 1, 68.4% (p<0.01) of the respondents reported that students would not be as prepared for residency compared to students who took a graded Step 1.

Table 1 Emergency Medicine Program Directors’ Responses Concerning the Predictive Value of Step 1 and Step 2 on Resident Performance

43 (15.2%) of PDs completed the entirety of the survey section evaluating applicant attributes. PDs ranked 14 of the 16 attributes higher following the transition to a pass/fail Step 1 (Table 2). Although the change was minimal, two attributes (SLOE and Step 2 CK score) remained in the top quartile of attributes compared to previous years. Were ranked lower following the transition to a pass/fail Step 1. However, analysis using a paired T-test comparing PDs perceived rankings pre-transition and post-transition to a pass/fail Step 1 revealed that these two changes were statistically insignificant, with p-values equal to 0.24 and 0.92, respectively. The changes did not affect the overall order of the top quartile of attributes compared to previous years. The first quartile of attributes pre-transition and post-transition are as follows: SLOEs [1st], away rotations [2nd], clerkship grades [3rd] and Step 2 CK score [4th]. Previously ranked the 5th highest attribute, the Step 1 numerical score is expected to be replaced by increased emphasis on extracurricular involvement, leadership experiences, class rank, and volunteering.

Table 2 Perceived Relative Rankings by 43 Emergency Medicine Program Directors of Various Residency Applicant Attributes Before and After Implementation of a Pass/Fail Step 1

Discussion

Our results indicate that EM PDs may not place more emphasis on the Step 2 CK score following the transition to a pass/fail Step 1 despite most PDs agreeing that Step 2 CK is a better predictor of a resident’s board examination performance. With the removal of a scored Step 1, EM PDs are likely to continue to consider SLOEs, away rotations, clerkship grades, and Step 2 CK score as being the most important attributes when assessing a medical student’s competitiveness to their residency program. According to the AAMC, EM programs around the country received an average of 861.59 residency applications in 2022.¹¹ As programs will no longer have a numerical Step 1 to initially screen applications, this transition may encourage a more holistic review of an applicant’s entire application.

The AAMC defines a holistic review as a flexible and individualized approach to assessing an applicant’s competitiveness through a balanced consideration given to experiences, attributes, and scholarly metrics.¹² The transition to a more holistic review in the medical resident selection process will promote a more diverse applicant pool to which programs are able to find applicants that align with the institution’s mission and goals. A more diverse physician workforce ensures all backgrounds, beliefs, and perspectives are represented in the medical field.

Resident performance: Our results suggest that PDs believe the Step 2 CK score to be a better predictor for EM specialty board examination performance compared to Step 1 score. PDs agreed that Step 1 score is not a strong predictor for residency clinical performance (p<0.01), but remained relatively neutral when asked if Step 2 CK was a strong predictor in residency clinical performance (p=0.33). However, when asked if students will be better prepared clinically after Step 1 transitions to pass/fail, 68% of respondents reported that students will not be as prepared and 32% remained neutral, even though the Step 1 exam places greater emphasis on basic science knowledge as compared to clinical knowledge. One possible explanation for this is that PDs may believe that intensive studying for a scored examination may help develop the discipline needed to perform well clinically. Medical students agree that the Step 1 exam falls short on focusing on the clinical knowledge required to successfully treat patients.¹³ The Step 2 CK exam, on the other hand, focuses more on testing a student’s ability to apply medical knowledge to clinical scenarios they will likely encounter in the future. With less emphasis placed on preparation for the Step 1 exam, medical students responded to a survey that they would be more likely to focus on other factors that will prepare them better for residency, including the Step 2 CK exam and clinical research.¹⁴

Limitations

There were several limitations to this study. The response rate was low for this survey. Given the low response rate and incomplete responses, there is a chance that this sample size might not be representative of all EM PDs. On the Qualtrics survey, PDs initially completed the multiple-choice portion of the survey. However, 14 of the PDs did not complete the second half of the survey consisting of the ranking questions. Therefore, the total number of responses for the multiple-choice questions did not equal the number of responses for the ranking questions. The purpose of this study was to develop an initial understanding of PD perceptions of a pass/fail USMLE Step 1 and to assess which components of the application may become more important following this change. Research in the coming years should be geared towards assessing the actual observed change. Follow-up data can then be compared to the initial data to determine if PDs will place more emphasis on attributes that were not accounted for in this study.

Conclusion

Without a 3-digit score on the USMLE Step 1 examination, our results indicate that SLOEs, away rotations, clerkship grades, and Step 2 CK scores will continue to be in the highest quartile of attributes considered by PDs when assessing a medical student’s competitiveness for residency programs. PDs indicate that they may not place a greater emphasis on Step 2 CK scores following the transition despite most PDs agreeing that the Step 2 CK score is a better predictor of EM board certification exam performance compared to the Step 1 score. More emphasis may be placed on leadership experiences, extracurricular involvement, class rank, and volunteering. Further research is required in the coming years to determine how the transition to a pass/fail USMLE Step 1 will impact the resident selection process.