Perspectives on Transitioning Step 1 of the United States Medical Licensing Examination to a Pass/Fail Scoring Model: Defining New Frameworks for Medical Students Applying for Residency
Authors Blamoun J, Hakemi A, Armstead T
Received 8 December 2020
Accepted for publication 23 January 2021
Published 11 February 2021 Volume 2021:12 Pages 149—154
Checked for plagiarism Yes
Review by Single anonymous peer review
Peer reviewer comments 2
Editor who approved publication: Dr Md Anwarul Azim Majumder
John Blamoun,1,2 Ahmad Hakemi,3 Teresa Armstead4
1Central Michigan University, College of Medicine, Mt Pleasant, MI, 48859, USA; 2Physician Assistant Program, MidMichigan Health, Midland, MI, 48670, USA; 3Central Michigan University, College of Health Professions, Mt Pleasant, MI, 48859, USA; 4Central Michigan University, College of Education and Human Services, Mt Pleasant, MI, 48859, USA
Correspondence: Ahmad Hakemi
Central Michigan University, College of Health Professions, 1280 East Campus Drive, Mt Pleasant, MI, 48859, USA
Tel +1 989-774-1273
Abstract: For years, the USMLE Step 1 has acted as an unofficial “concours” for medical students applying to residency positions in the United States. The three-digit numeric score has been used to rank thousands of applicants without any evidence of validity. The USMLE will soon change score reporting to a pass/fail outcome. The main reason given was to address the concerns about its effects on the well-being of the students and medical education. It is argued that time for change has come. The authors discuss the various viewpoints of the stakeholders and the effects of this change on applicants and potential changes on the undergraduate medical curriculum. Furthermore, this article discusses several metrics that can be utilized in the application process in lieu of the USMLE Step 1. Additionally, some novel key metrics in the application process are identified, and their unique dynamic and adaptive characteristics are deliberated. Finally, the benefits of a transparent and holistic process are strongly advocated.
Keywords: USMLE step 1 residency, multiple mini interviews, holistic graduate medical education admissions
On February 12, 2020, the United States Medical Licensing Examination (USMLE) made the following charged announcement on its website:
The USMLE program will change score reporting for Step 1 from a three-digit numeric score to reporting only a pass/fail outcome. A numeric score will continue to be reported for Step 2 Clinical Knowledge (CK) and Step 3.1
This decision was supported by the American Medical Association and the Educational Commission for Foreign Medical Graduates.1 Ostensibly, the change was made to “address concerns about Step 1 scores impacting student wellbeing and medical education.”1 Moving to a pass/fail system essentially stops residency programs from using USMLE Step 1 for screening/evaluating purposes. For years, the USMLE Step 1 has been repurposed as a “concours” for entrance to prestigious US residency programs, despite the fact that there is a paucity of literature supporting the correlation of Step 1 scores and clinical competence.2 With this shift, the landscape for many US and International medical students has changed: a pass/fail system means that Step 1 cannot be used as an applicant ranking tool. However, it is helpful to note that the USMLE is only one of many standardized tests that students take prior to graduation from medical school. Standardized tests originated in early 20th century in the US.3
Transitioning to a Pass/Fail System
The USMLE Step 1 is a seven-hour examination taken during the second year of medical school. It is prepared by the National Board of Medical Examiners. It is the first step of a three-part examination for medical licensure in the United States (Step 2 has two parts, namely: Clinical Knowledge and Clinical Skills). Step 3 assesses unsupervised patient management in ambulatory settings.
The USMLE was originally designed as one of the four examinations necessary for licensure as a physician in the United States. For the purposes of licensing, the USMLE historically employed a pass/fail system, with licensing boards disregarding numerical scores. That being said, over the years, examinations have been morphed for other purposes. For example, at many medical schools, passing Step 1 is a requirement for starting clinical clerkships. The fact that Step 1 is part of the licensure examination may seem odd to medical students inculcated into obtaining high scores to earn residency positions; nevertheless, it has become a competitive imperative for faculty to teach to this test. Important to note here is the tension between the work of Step 1 and its applicability: while Step 1 measures the learner applicant’s general understanding of basic science concepts and their application in the practice of medicine, residency programs expect superior clinical performance.
Nevertheless, the high volumes of residency applications that must be processed have compelled nearly all residency programs to use a quantitative measure for screening and stratifying applicants. By default, the Step 1 score has acted as a “screener.” This has essentially re-purposed the USMLE Step 1 as a gateway for US medical students and International Medical Graduates into residency programs. Along these lines, the conclusions of the 2018 National Resident Matching Program’s “Program Director Survey” demonstrated that 94% of residency programs specified that the USMLE Step 1 score was a significant factor in selecting applicants to interview.4 The global competition for limited residency slots in the United States has made it a competitive imperative for International Medical Graduates to start early and study hard for Step 1. In fact, the majority of Caribbean medical schools that cater to American students stress success in Step 1.
Anxiety over scores as a selection for residency programs has been taking an extraordinary toll on medical students’ well-being.5 Changing the general grading system to the pass/fail system in the pre-clinical years of US medical schools has improved psychological well-being in medical students.5 That being said, third- and fourth-year medical students and residents prefer the ongoing use of numerical scoring because they believe that scores are important in residency selection, that residency applicants are advantaged by examination scores, and that scores provide an important impetus to review and solidify medical knowledge.2 On the other hand, Chen6 and colleagues argue that the current Step 1 milieu attenuates learning and adds to workforce inequality. Over the years, potential residents have noted a non-validated narrative with the establishment of the cult predictive ability of the USMLE Step 1 for the screening/selection of first-year residents, rather than a nuanced approach.7 A recent survey of surgical residency program directors found that approximately 20% of program directors believe that this change will improve student well-being, and that 89% believe it will increase the importance of the USMLE Step 2 for applications.8
Many US medical students do not attend lectures (which are live-streamed) and, in many instances, ignore the established curriculum and place emphasis on self-purchased commercial board review content for success in USMLE Step 1. This milieu emphasizes Step 1 basic sciences content and underrates student learning. Courses such as clinical skills have poor attendance and laboratory practice time is minimized. Helpful to note here is that at least 9 companies offer commercial content for US medical students, and many of the authors of such content are medical school faculty.6 Medical students have argued that commercial companies are aggressively encouraging them to take a what-you-need-to-know approach to studying to optimize learning under time-constraints by producing quality content.
However, the ultimate goal of a residency training program is to train professionals to become clinically competent and USMLE Step 1 has low to moderate predictive ability in this regard.7 That being said, it is important to recall, as noted above, that program directors have historically relied heavily on USMLE Step 1 for resident selection. Residency programs have adopted the Step 1 score as a “screener”’ over the years instead of using other objective measures of competency for various reasons. Because most courses during the preclinical years of study are pass/fail, there is no index of discrimination. However, in the future, the development of new assessment models for Entrustable Professional Activities (EPAs) in the clinical years may be a validated viable option. Due to a paucity of time in medical school, lack of focus, and continuity, student volunteer research with faculty is rarely useful. Important to note is that overreliance on a single standardized metric can be specifically detrimental to minority students because it puts disproportionate weight on a single measure and thus create barriers to entry. Underrepresented students in medicine (eg, self-identified Black students) have historically underperformed white students in USMLE Step 1.9
International Medical Graduates
International Medical Graduates (IMGs) have an arduous pathway; they must pass Step 1, Step 2 CK, and Step 2 CS of the USMLE in order to be certified by the Educational Commission for Foreign Medical Graduates (ECFMG) before applying to residency programs in the USA. IMGs are divided into two categories: US citizens studying in the Caribbean and non-US citizens studying in the rest of the world. Caribbean medical schools have multimillion-dollar affiliation agreements with US hospitals, and many of their medical students obtain residencies in the hospitals in which they perform their clerkships. Thus, this change should not create difficulties for them. Meanwhile, non-US IMGs from the rest of the world compete fiercely for residency positions in the US; the numerical score system has provided them with a “level playing field”10 indeed, there are rarely judged by other criteria. This change necessitates ingenuity in finding other suitable criteria with which to assess residency candidates, such as the USMLE Step 2 CK, which continues to use a three-digit numerical score. Because IMGs have traditionally used Step 1 scores to differentiate themselves, the elimination of the three-digit numeric score will have negative repercussions. As gestured to above, traditionally, a high Step 1 score has single-handedly been the determining force as to whether a candidate is awarded an interview.11 It is anticipated that Caribbean medical schools catering to US citizens will swiftly reorganize their curriculum, moving from a heavy emphasis on basic sciences to focus instead on clinical sciences. This change may disadvantage non-US citizen medical students familiar with different models of medical education; however, it may also present them with new opportunities.
For the foreseeable future, it appears that non-US citizen IMGs would do well to seek a high numeric score in the USMLE Step 2 CK to enhance their probability of landing a favorable residency.
Stakeholders of importance are medical students, medical schools, the American Association of Medical Colleges (AAMC), and residency programs. The presidents of the National Board of Medical Examiners (NBME) and Federation of State Medical Boards (FSMB) are not stakeholders per se, but they have opined on this topic and expressed various ideas on improving the process in an invited commentary.12 More specifically, they suggested numerous metrics by which to rank applications while at once indirectly affirming the past use of USMLE Step 1 as a default. Solid recommendations from other stakeholders’ medical schools, residency programs, and the AAMC have yet to gel.
Summary of Selection Metrics
The purpose of passing Step 1 is to meet the partial requirements of permanent licensure in all 50 states and territories of the US. Step 1 was created to evaluate the applicant’s understanding and application of basic sciences relevant to medical practice.13 With Step 1 being out of the equation, a transparent holistic approach needs to be established. Because this will take time, we must ask: what other metrics may residency programs use in the meantime?
Suggestions have been made for using Step 2 CK scores for screening; recall, as noted above, that applicants can take the Step 2 examination after applying to a residency program and then submit its three-digit numeric score—recall that program directors typically heavily weigh “numbers” in screening applicants. Step 2 CK has more clinical relevance, but, unlike Step 1, its validity has not been established for residency programs. A study published by Sharma et al,14 noted that the USMLE Step 2 CK was the best predictor of residency performance on standardized testing during and after residency, as well as clinical performance from multiple perspectives during a yearlong block continuity experience. This study has many limitations, and it would be challenging to apply it to a holistic selection process. We hypothesize that providing scores for Step 2 CS may be a reliable and valid option. Step 2 CS is essentially a series of objective structured clinical evolution (OSCE) stations with well-established reliability and validity that assess knowledge, skills, and attitudes. Step 2 CS has superior psychometric properties and is the gold standard for the assessment of clinical skills in a controlled and reproducible clinicomimetic environment. Providing numeric scores for Step 2 CS may satisfy program directors’ curiosity for “numbers.” Other options include the use of the “shelf exams” taken at the end of each clerkship; these subject examinations, prepared by the National Board of Medical Examiners, may improve objectivity.15 What duties do residency programs have to inform applicants of the “real” admission criteria they use and the profiles of successful applicants? Surely, they use criteria other than the USMLE Step 1. Notably, this change is unlikely to affect those applicants from “prestigious” medical schools who are generally favored by program directors.16
A former president of the National Resident Matching Program (NRMP) has eloquently written on this issue and urged residency programs to be more transparent and publish the profiles of the 3–5 cohorts they admit.17 According to these former director programs, they closely guard their admission characteristics.17 Residency programs have not been transparent in publishing their “traffic rules.”18 A mindset change among residency program directors is necessary. A 2018 survey of the National Resident Matching Program rated the Step 1 score the most frequently used factor of 33 factors in screening applicants for interviews.4 It is interesting to note that they considered Step 2 CS as well, which assesses those skills for which program directors are actually looking.18
The deeper point here is that the time has come to transition to a holistic, transparent, and fair process. Accordingly, residency program directors should also consider setting up multiple mini interviews (MMIs) and situational awareness tests in addition to reviewing personal statements, audition clerkship evaluations, letters of reference, Medical Student Performance Evaluations (MSPE), and other aforementioned metrics. The MMI is powerful tools that medical schools frequently use as a metric in selecting students (notably, validated computerized versions of the MMI exists). Research has shown that MMI performances can predict clerkship and licensing examination performance.19 Most importantly, the prestige of a medical school should not be a metric, even though it is functioning as such presently nationwide and remains a bragging point. Program directors need to integrate diverse perspectives and attributes into an inclusive holistic selection process and validate it. A holistic framework that includes the review of non-cognitive capabilities will complement other cognitive metrics. Above all, residency programs should be transparent about what types of applicants they seek.
A Timely Proposal
The dominance and disproportionate importance of Step 1 as a powerful screening and selection tool has created a “Step 1 climate” or milieu to the detriment of medical students.6 With program directors’ insistent use of numerical scores in screening applicants, a shift to Step 2 CK will require evolving validated metrics. Will we have a “Step 2 climate”? It is anticipated that pass/fail grading will require curricular infrastructure changes to the M1 and M2 years, including increased emphasis on clinical context and early introduction to patient care to facilitate an earlier preparation for Step 2 CK. The relentless emphasis on training medical students for high Step 1 scores in medical schools will hopefully push the curriculum in a new direction, providing students with new learning experiences, such as early introduction to patient care. Important to remember is that substitution of one number for another is not the intent of the stakeholders who initiated this change. Program directors presently use the USMLE Step 1 as an assessment tool for the classification of applicants; however, USMLE Step 1 measures the wrong metric. USMLE’s transition to a pass/fail system for Step 1 presents a good opportunity for residency programs to innovate in ways that provide stakeholders with alternative metrics beyond a three-digit score. To be sure, programs need to develop dependable measures for the assessment of applicants, and multiple assessment methods are necessary to effectively measure candidate knowledge, skills, and attitudes. A more robust system will enable programs to select applicants who best fit their objectives. As noted earlier, scaling up to a numeric score for the Step 2 CS and an admission MMI will upgrade the assessment process to a higher level of Miller’s pyramid. In addition to clerkship grades, measures such as meaningful research, volunteering, community service, letters of reference, and work experience should also be considered. They can provide an in-depth understanding of the applicant’s accomplishments.
Most importantly, a well-defined and transparent holistic admissions policy for residency program applicants should be normalized. The advent of the COVID-19 pandemic has promulgated this approach. A recent study by Hammond encourages this approach.20 A broad holistic review and selection framework can evaluate applicants based on several aspects compared to the metrics criteria; furthermore, it can be goal and mission centric. Other interventions can include standardizing the interview experience and transparency in the review of USMLE scores. A residency program utilizing a holistic process may sidestep or lower the cut-off USMLE scores.21 Marginal scores can have more balanced reviews by considering the entire context in an application. Other considerations can include socioeconomic class, adversities, cumulative meaningful life/work experiences, and unique contributions. Including MMIs as an integral part of the holistic process will increase the reliability of the assessment of non-cognitive attributes associated with success in a residency program.22 MMI station designers need to align scenarios with the institutional mission to increase validity. All assessments are samples, and in a high-stake interview for a residency position, a ten station MMI is needed to achieve a reliability of 0.65.23,24 As with any assessment, it is important to design stations that assess non-cognitive variables that are important to the program and not just what is easy.24 The paucity of residency programs following a holistic admission process is concerning, and efforts should be undertaken by stakeholders to develop standardized metrics similar to AAMC recommended strategies.2,25,26
All authors made a significant contribution to the work reported, whether that is in the conception, study design, execution, acquisition of data, analysis and interpretation, or in all these areas; took part in drafting, revising or critically reviewing the article; gave final approval of the version to be published; have agreed on the journal to which the article has been submitted; and agree to be accountable for all aspects of the work.
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
The authors declare that there is no conflict of interest.
2. McGaghie WC, Cohen ER, Wayne DB. Are United States medical licensing exam step 1 and 2 scores valid measures for postgraduate medical residency selection decisions? Acad Med. 2011;86:48–52. doi:10.1097/ACM.0b013e3181ffacdb
3. Harvard: from college to university, recruiting a superior faculty, the elective system. Education Encyclopedia. Available from: https://education.stateuniversity.com/pages/1951/Eliot-Charles-1834-1926.html.
4. National resident matching program, data release and research committee. Results of the 2018 NRMP program director survey. Washington, DC: National Resident Matching Program; 2018. Available from: https://www.nrmp.org/wp-content/uploads/2018/07/NRMP-2018-Program-Director-Survey-for-WWW.pdf.
5. Bloodgood R, Short J, Jackson J, Martindale J. A change to pass/fail grading in the first two years at one medical school results in improved psychological well-being. Acad Med. 2009;84(5):655–662. doi:10.1097/ACM.0b013e31819f6d78
6. Chen DR, Priest KC, Batten JN, et al. Student perspectives on the “Step 1 Climate” in preclinical medical education. Acad Med. 2019;94:302–304. doi:10.1097/ACM.0000000000002565
7. Berner E, Brooks M, Erdmann JB. Use of the USMLE to select residents. Acad Med. 1993;68(10):753–759. doi:10.1097/00001888-199310000-00005
8. Pontell ME, Makhaul AT, Kumar GN, Drolet BC. The change of USMLE Step 1 to pass/fail: perspectives of the surgery program director. J Surg Educ. 2020;1–8. doi:10.1016/j.surg.2020.06.034
9. Rubright JD, Jodoin M, Barone MA. Examining demographics, prior academic performance, and United States medical licensing examination scores. Acad Med. 2019;94(3):364–370. doi:10.1097/ACM.0000000000002366
10. Salari S, Deng F. A stepping stone toward necessary changes: how the new USMLE Step1 scoring system could affect the residency application process. Acad Med. 2020;95:1312–1314. doi:10.1097/ACM.0000000000003501
11. Boulet JR, Pinsky WW. Reporting a pass/fail outcome for USMLE Step 1: consequences and challenges for international medical graduates. Acad Med. 2020;95(9):1322–1324. doi:10.1097/ACM.0000000000003534
12. Katsufrakis PJ, Chaudhry MS. Improving residency selection requires close study and better understanding of stakeholder needs. Acad Med. 2019;94(3):305–308. doi:10.1097/ACM.0000000000002559
14. Sharma A, Schauer DP, Kelleher M, Kinnear B, Sall D, Warm W. USMLE Step 2 CK: best predictor of multimodal performance in an internal medicine residency. J Grad Med Educ. 2019;11:412–419. doi:10.4300/JGME-D-19-00099.1
15. Crane MA, Chang HA, Azamfirei R. Medical education takes a step in the right direction: where does that leave students? JAMA. 2020;323(20):2013–2014. doi:10.1001/jama.2020.2950
16. Hammond MA, Andrews J, Skochelak SE. Improving the residency application and selection process: an optional early result acceptance program. JAMA. 2020;323:503–504. doi:10.1001/jama.2019.21212
17. Blacklow AB. Re: improving the residency application and selection process. JAMA Network. https://jamanetwork.com/journals/jama/fullarticle/2759409.
18. Cuddy MM, Winward ML, Johnson MM, Lipner RS, Clauser BE. Evaluating validity evidence for USMLE Step 2 clinical skills data gathering and data performance predicts history-taking and physical examination ratings for first year internal medicine residency? Acad Med. 2016;91:133–139. doi:10.1097/ACM.0000000000000908
19. Better HJ, Eva KW, Rosenfeld J, Norma G. Multiple mini interviews predict clerkship and licensing examination performance. Med Educ. 2007;41:378–384. doi:10.1111/j.1365-2929.2007.02709.x
20. Hammoud MM, Standiford T, Carmody JB. Potential implications of COVID-19 for the 2020–2021 residency application cycle. JAMA. 2020;324(1):29–30. doi:10.1001/jama.2020.8911
21. Aibana O, Swails JL, Flores RJ, Love L. Bridging the gap: holistic review to increase diversity in graduate medical education. Acad Med. 2019;94(8):1137–1141. doi:10.1097/ACM.0000000000002779
22. Bates T, Mutha S, Coffman J Practicing holistic review in medical education. Healthforce Center at UCSF. Available from: https://healthforce.ucsf.edu/sites/healthforce.ucsf.edu/files/publication-pdf/Practicing%20Holistic%20Review%20in%20Medical%20Education_102220.pdf.
23. Eva KW, Rosenfeld J, Reiter HI, Norman GR. An admissions OSCE: the multiple mini-interview. Med Educ. 2004;38(3):314–326. doi:10.1046/j.1365-2923.2004.01776.x
24. Humphrey-Murto S, Wood TJ, Ross S, et al. Assessment pearls for competency-based medical education. J Grad Med Educ. 2017;9(6):688–691. doi:10.4300/JGME-D-17-00365.1
25. Moynahan KF. The current use of United States medical licensing examination Step 1 scores: holistic admissions and student well-being are in the balance. Acad Med. 2018;93(7):963–965. doi:10.1097/ACM.0000000000002101
26. Addams AN, Bletzinger RB, Sondheimer HM, White SE, Johnson LM. Washington, DC: association of American Medical Colleges; 2010. Roadmap to diversity: integrating holistic review practices into medical school admission processes. Acad Med. 2016;91:1472–1474. doi:10.1097/ACM.0000000000001403
This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.Download Article [PDF]