Predicting performance on the United States Medical Licensing Examination Step 1 and Step 2 Clinical Knowledge using results from previous examinations

Introduction

The United States Medical Licensing Examination (USMLE) Step 1 and Step 2 Clinical Knowledge (CK) offered by the National Board of Medical Examiners (NBME) are often regarded as two of the most challenging milestones in medical school. In first place, in most schools, students who do not approve the USMLE Step 1 and Step 2 CK are not candidates for graduation. In addition to that, lower scores in the USMLE Step 1 and Step 2 CK might be predictive of lower means during the residency in-training examinations.¹ Likewise, several studies have shown that lower scores in the USMLE Step 1 and/or in the Step 2 CK might be associated with subsequent failure during the specialty board examinations.^2,3 Conversely, other investigators have found that higher scores during the USMLE Step 1 and/or the Step 2 CK might be associated with better performance during the specialty boards.^4–6 As a result, many residency programs use the scores from the USMLE Step 1 and/or Step 2 CK as criteria for accepting candidates.^7,8 Furthermore, the individual school’s USMLE passing rates among first time takers can have a great impact in the accreditation process or in the general perception about the school.

Finding early specific indicators of failure in the USMLE Step 1 and the Step 2 CK would be of benefit for medical schools and also for medical students, because it could be used to develop early interventions that could solve deficiencies and help at-risk students to ultimately attain a passing score. Therefore, there is a growing interest in finding adequate predictors of failure in the USMLE. Several investigators have postulated the use of objective data, such as the scores from one or several NBME Basic Science Subject Examinations⁹ and the performance on NBME Customized Assessment Service questions,¹⁰ to identify at-risk students. The NBME Comprehensive Basic Science Examination (CBSE) is administered by many medical schools to identify students performing below expectations¹¹ or to test student progress,¹² but more studies regarding its use as predictive tool for success in the USMLE are needed. Furthermore, specific data on this subject in Hispanic or Latino populations are currently scarce in the literature.

The University of Puerto Rico School of Medicine (UPR SoM), located in San Juan, PR (an insular commonwealth of the USA), was founded in the year 1950 and has been accredited by the Liaison Committee on Medical Education since 1954. According to the graduation questionnaire from the Association of American Medical Colleges, 98% or more of the students from the UPR SoM self-identify as Hispanic, Latino, or of Spanish origin. In our institution, second-level medical students must take and approve the USMLE Step 1 in order to be promoted to the third level. Likewise, students must approve the USMLE Step 2 CK prior to graduation. While searching for indicators of failure in the USMLE among our group of students in order to provide targeted interventions, we hypothesized that the NBME CBSE could be used as a predictor of outcome in the USMLE Step 1, because it is offered in the UPR SoM at the end of the second level (about 4–8 weeks before most students take the USMLE Step 1 for the first time) and because it tests topics from courses given in first and second medical levels.

Therefore, this study was aimed at determining if the NBME CBSE could predict a passing score during the USMLE Step 1. As a secondary outcome, we aimed at assessing if the USMLE Step 1 could predict a passing score during the USMLE Step 2 CK, as opposed to other possible predictors of passing the USMLE Step 2 CK, such as the NBME Medicine Clinical Science Subject Examination (CSSE).

Methods

For all students from the UPR SoM who took the USMLE Step 1 for the first time from June 1, 2011, to December 31, 2017, the Office of Evaluation of the UPR SoM provided the investigators with the scores in that test and also with demographic data (gender) and the scores of the NBME CBSE, the NBME Medicine CSSE, and the USMLE Step 2 CK. Inclusion criteria included being 21 years of age or older, having studied at the UPR SoM, and having taken the USMLE Step 1 for the first time in the described period. Scores from the four examinations were linked using the student number. Central tendency measures and dispersion measures were used to assess the raw distribution of all study variables. Independent samples’ t-test, Pearson correlation matrices, and chi-squared test for independence served as the inferential approach. Statistical significance was set to a P≤0.05, using a two-tailed approach. IBM-SPSS was used for all calculations.

The study was approved by the Institutional Review Board of the University of Puerto Rico Medical Sciences Campus.

Results

Data were gathered from 724 consecutive students, comprising all second-level students from 2010–2011 to 2016–2017 academic years. From these, five (0.7%) students had not taken the USMLE Step 1 at the time of analysis, for which they were excluded. The final number of students included in the analysis is 719 (99.3%). For the USMLE Step 2 CK analysis, a total of 564 (77.9%) consecutive students from the total sample were included, comprising all students who had taken the NBME CBSE between 2010–2011 and 2016–2017 academic years and who had already taken the USMLE Step 2 CK. This difference in the sample number is explained by the fact that the USMLE Step 1 and Step 2 CK examinations are taken at different times during the student’s education.

USMLE Step 1

As described in Table 1, of the 719 students who took the USMLE Step 1 within the study period, 345 (48.0%) students were male and 374 (52.0%) students were female. A total of 671 (93.3%) students approved the USMLE Step 1 on their first attempt. The mean score for all students was 220.5±9.2 (225.2±18.9 for males and 218.7±19.2 for females). Of the 48 (6.7%) students who did not approve the test on their first attempt, 17 (4.9%) students were male and 31 (8.3%) students were female. The difference in mean scores between male and female students was statistically significant (P≤0.0001), but the difference in passing rates between genders was not (P=0.075).

Table 1 Gender distribution, mean scores, and SD of students during USMLE Step 1, NBME CBSE, USMLE Step 2 CK, and NBME Medicine CSSE Notes: aP≤0.0001. bP=0.075. cP≤0.0001. dP=0.5. eP=0.393. fP=0.331. Abbreviations: CBSE, Comprehensive Basic Science Examination; CK, Clinical Knowledge; CSSE, Clinical Science Subject Examination; NBME, National Board of Medical Examiners; USMLE, United States Medical Licensing Examination.

NBME CBSE as a predictor of performance in the USMLE Step 1

A total of 718 (99.9%) students who took the USMLE Step 1 within the study period also took the NBME CBSE. Only one student who took the USMLE Step 1 during the study period had previously taken the NBME CBSE during an academic year that was out of the study sample and thus was not included in this analysis. The mean score in this test was 63.8±9.2 (66.5±9.3 for males and 61.6±8.4 for females) (Table 1). There was a statistically significant difference in mean scores between male and female students (P≤0.0001). A significant correlation (r=0.730, P≤0.001) between the score in the NBME CBSE and the score in the USMLE Step 1 was found in this study group (Figure 1).

Figure 1 Score in the NBME CBSE vs score in the USMLE Step 1. Notes: r=0.730, P≤0.001. Abbreviations: CBSE, Comprehensive Basic Science Examination; NBME, National Board of Medical Examiners; USMLE, United States Medical Licensing Examination.

We then aimed at testing a score of 66 in the NBME CBSE as a cutoff to predict outcome in the USMLE Step 1 among our group of students. A score of 66 in the NBME CBSE was selected because it is detailed as an approximate equivalent to a score of 190 in the USMLE Step 1 (data from the NBME CBSE score report).¹³ Since the minimum passing score for the USMLE Step 1 during our study period ranged from 188 (until December 31, 2013) to 192 (from January 1, 2014 to December 31, 2017) (data from the NBME), a midpoint of 190 was chosen. Of the 48 students who did not approve the USMLE Step 1 on their first attempt, 47 (97.9%) students had a score <66 in the NBME CBSE. Conversely, of those who achieved a score of 66 or greater on this examination (293, 40.8%), only one (0.3%) student did not approve the USMLE Step 1 on his/her first attempt (data not shown). Therefore, a score <66 in the NBME CBSE was significantly associated to failure in the USMLE Step 1 (P≤0.00001). This was even more relevant in female students. When students with a score <66 in the NBME CBSE were adjusted by gender, female students were 55.5% less likely to approve the USMLE Step 1 on their first attempt (OR =0.445, 95% CI =0.329–0.603, P≤0.001)

USMLE Step 2 CK

For the USMLE Step 2 CK, of the 564 students, 258 (45.7%) students were male and 306 (54.3%) students were female (Table 1). A total of 541 (95.9%) students approved the USMLE Step 2 CK on their first attempt; 20 (3.5%) students did not approve the test on their first attempt, and 3 (0.6%) students had not yet received their scores at the time of analysis (data not shown). The mean score was 235.5±17.5 (236.2±17.7 for males and 235.2±17.1 for females) (Table 1). Of the 23 (4.1%) students who did not approve the test on their first attempt, 13 (5.0%) students were male and 10 (3.3%) students were female. There were no statistically significant differences in mean scores (P=0.500) and in passing rates (P=0.393) between male and female students.

NBME Medicine CSSE as predictor of performance in the USMLE Step 2 CK

A total of 546 students took the NBME Medicine CSSE within the study period; 248 (45.4%) students were male and 298 (54.6%) students were female (Table 1). The mean score in this test was 76.8±8.3 (77.8±8.3 for males and 77.1±7.7 for females). There was no statistically significant difference between the mean scores of male and female students (P=0.331). A significant correlation (r=0.572, P≤0.001) was found between the score in the NBME Medicine CSSE and the score in the USMLE Step 2 CK (Figure 2). Of the 20 students who did not approve the USMLE Step 2 CK on their first attempt, 14 (70.0%) students had a score <74 in the NBME Medicine CSSE. Conversely, of those who achieved a score of 74 or higher on this examination (370, 66.7%), only six students (1.6%) did not approve the USMLE Step 2 CK on their first attempt (data not shown). Although this association was significant (P≤0.000232) for the entire student sample, it lost its statistical significance when adjusted for gender (P=0.306).

Figure 2 Score in the NBME Medicine CSSE vs score in the USMLE Step 2 CK. Notes: r=0.572, P≤0.001. Abbreviations: CK, Clinical Knowledge; CSSE, Clinical Science Subject Examination; NBME, National Board of Medical Examiners; USMLE, United States Medical Licensing Examination.

USMLE Step 1 as predictor of performance in the USMLE Step 2 CK

Of the 541 students who approved the USMLE Step 2 CK on their first attempt, 509 (94.1%) students also approved the USMLE Step 1 on their first attempt (data not shown). There was a significant correlation (r=0.698, P≤0.001) between the scores in the USMLE Step 1 and the USMLE Step 2 CK (Figure 3). Of the 20 students who did not approve the USMLE Step 2 CK on their first attempt, 15 (75.0%) students had a score of <208 on the USMLE Step 1. Conversely, only 5 of the 351 (1.4%) students who obtained a score of 208 or higher on the USMLE Step 1 did not approve the USMLE Step 2 CK on their first attempt (data not shown). There was a significant correlation (r=0.684, 95% CI =636–0.730, P≤0.0001) between obtaining a score of 208 or higher in the USMLE Step 1 and subsequently attaining a passing grade on the first take of the USMLE Step 2 CK.

Figure 3 Score in the USMLE Step 1 vs score in the USMLE Step 2 CK. Notes: r=0.698, P≤0.0001. Abbreviations: CK, Clinical Knowledge; USMLE, United States Medical Licensing Examination.

Discussion

In our group of students, the mean score was higher in males during the USMLE Step 1 and during the USMLE Step 2 CK, but the difference in mean USMLE score between genders was only significant for the Step 1. Differences in the USMLE Step 1 mean score favoring male test takers have been documented for decades: in 1973 by Weinberg and Rooney,¹⁴ in 1993 by Case et al,¹⁵ and in 2018 by Gauer and Jackson.¹⁶ In some instances, the differences in mean scores among genders have also been found to be statistically significant.^14,16 In our population, there were also statistically significant higher mean scores for males in the NBME CBSE. Most of our students took the USMLE Step 1 shortly (within 6–8 weeks) after the NBME CBSE. This relatively short period of time between the two examinations could explain the persistence of higher mean scores for male students in both tests.

The reasons for differences in mean in the USMLE Step 1 among genders are unclear and poorly described in the literature. To our knowledge, this is the first time that this pattern is also described in a Hispanic/Latino population. However, given the observational nature of our study, it was not possible to assess factors contributing to this difference. More studies on this subject are certainly needed to explore the factors that could be contributing to it. Since female students who obtained a score <66 in the NBME CBSE were less likely to approve the USMLE Step 1, it might be relevant to consider gender differences when designing interventions for students with lower performance while in preparation for the USMLE Step 1. Interestingly, there were no statistically significant differences in the mean between males and females during the NBME Medicine CBSE and during the USMLE Step 2 CK, suggesting that female students tend to catch up over time with their male counterparts.

We found a significant correlation between the score in the NBME CBSE and the subsequent score in the USMLE Step 1. In 1997, Glew et al¹⁷ found a correlation between the first NBME CBSE administration and eventual USMLE Step 1 score that was identical to ours (r=0.73), but in their study, the first NBME CBSE occurred almost 2 years prior to the USMLE Step 1 take. Also, they analyzed the correlation to the USMLE Step 1 using a second and third NBME CBSE administration.¹⁷ In contrast, in our study, the students took the NBME CBSE only once and, in most cases, within 6–8 weeks of taking the USMLE Step 1.

Similarly, we also found a significant positive correlation between the score during the NBME Medicine CSSE and the score in the USMLE Step 2 CK. There was also a significant correlation between the scores in the USMLE Step 1 and USMLE Step 2 CK. This correlation supports prior work done by Monteiro et al,¹⁸ who reported that the USMLE Step 1 is a significant predictor of the USMLE Step 2 CK score.

In our group, students who obtained a score <66 in the NBME CBSE were less likely to approve the USMLE Step 1 on their first attempt. Similarly, most of the students who did not approve the USMLE Step 2 CK on their first attempt had a score <74 in the NBME Medicine CSSE or a score of <208 in the USMLE Step 1. Therefore, we postulate that these scores might be used as indicators to identify students who are at higher risk of failing. However, it is important to realize that student performance on standardized tests such as the USMLE Step 1 and USMLE Step 2 CK depends on numerous factors and it might be impossible to find a single predictor of a passing score. Although generalization of results might not be possible given the specific population being studied, our findings could guide medical educators to target interventions aimed at helping at-risk students to pass the USMLE Step 1 and USMLE Step 2 CK based on objective indicators. Nonetheless, ultimately deciding how many students will be considered for early interventions might also be matter of institutional capacity, support availability, and financial resources. Many institutions may need to select more than one indicator in order to determine which students are going to benefit the most from targeted interventions.

Future studies on this topic could further assess if other socio-demographic characteristics (such as exact age upon taking each examination, marital status, background education, income, and household details) or other objective academic measures (such as specific course grades and other NBME subject examinations) could also serve as indicators of success or failure during the USMLE Step 1 and Step 2 CK.

Limitations

This is a single-center study in a medical school with a vast majority of Hispanic/Latino students. Therefore, the results might not be generalized to all medical students in USA. Other limitations of this study are inherent to its observational design, including that we were unable to consider important variables such as total study time after taking the NBME CBSE prior to taking the USMLE Step 1, or time elapsed between the first attempt at taking the USMLE Step 1 vs the USMLE Step 2 CK or between NBME Medicine CSSE and the USMLE Step 2 CK.

Conclusion

There are significant correlations between the scores of the NBME CBSE and the USMLE Step 1, between the scores of the NBME Medicine CSSE and the USMLE Step 2 CK, and between the scores of the USMLE Step 1 and USMLE Step 2 CK. A score <66 in the NBME CBSE is significantly associated with failure in the USMLE Step 1 first take among our students. Similarly, a score <208 in the USMLE Step 1 is significantly associated with failure during the USMLE Step 2 CK. A score <74 in the NBME Medicine CSSE might also indicate risk of failure during the USMLE Step 2 CK.

Acknowledgments

The authors would like to thank 1) Gerónimo Maldonado-Martínez, RPT, MPH, PhD, for his advice on statistical analysis; 2) Irma L. Rivera-Colón, EdD, MS, and her staff in the Office of Evaluation for providing the investigators with the data; and 3) Karen G. Martínez, MD, MSc, for her meaningful insight about the manuscript. This project was partially supported by the National Institute on Minority Health and Health Disparities (NIMHD) and the National Institute of Allergy and Infectious Diseases (NIAID) of the National Institutes of Health under the Award Number U54MD007587 for the Puerto Rico Clinical and Translational Research Consortium. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Disclosure

The authors report no conflicts of interest in this work.

References

1.		Thundiyil JG, Modica RF, Silvestri S, Papa L. Do United States Medical Licensing Examination (USMLE) scores predict in-training test performance for emergency medicine residents? J Emerg Med. 2010;38(1):65–69.
2.		de Virgilio C, Yaghoubian A, Kaji A, et al. Predicting performance on the American Board of Surgery qualifying and certifying examinations: a multi-institutional study. Arch Surg. 2010;145(9):852–856.
3.		Swanson DB, Sawhill A, Holtzman KZ, et al. Relationship between performance on part I of the American Board of Orthopaedic Surgery Certifying Examination and Scores on USMLE Steps 1 and 2. Acad Med. 2009;84(10 Suppl):S21–S24.
4.		Harmouche E, Goyal N, Pinawin A, Nagarwala J, Bhat R. USMLE scores predict success in ABEM initial certification: a multicenter study. West J Emerg Med. 2017;18(3):544–549.
5.		Dillon GF, Swanson DB, McClintock JC, Gravlee GP. The relationship between the American board of anesthesiology part 1 certification examination and the United States medical licensing examination. J Grad Med Educ. 2013;5(2):276–283.
6.		Sutton E, Richardson JD, Ziegler C, Bond J, Burke-Poole M, McMasters KM. Is USMLE Step 1 score a valid predictor of success in surgical residency? Am J Surg. 2014;208(6):1029–1034.
7.		Berner ES, Brooks CM, Erdmann JB. Use of the USMLE to select residents. Acad Med. 1993;68(10):753–759.
8.		Gauer JL, Jackson JB. The association of USMLE Step 1 and Step 2 CK scores with residency match specialty and location. Med Educ Online. 2017;22(1):1358579.
9.		Holtman MC, Swanson DB, Ripkey DR, Case SM. Using basic science subject tests to identify students at risk for failing step 1. Acad Med. 2001;76(10 Suppl):S48–S51.
10.		Brenner JM, Bird JB, Willey JM. Formative assessment in an integrated curriculum: identifying at-risk students for poor performance on USMLE Step 1 using NBME custom exam questions. Acad Med. 2017;92(11S Association of American Medical Colleges Learn Serve Lead: Proceedings of the 56th Annual Research in Medical Education Sessions):S21–S25.
11.		Wright WS, Baston K. Use of the National Board of Medical Examiners® Comprehensive Basic Science Exam: survey results of US medical schools. Adv Med Educ Pract. 2017;8:377–383.
12.		Johnson TR, Khalil MK, Peppler RD, Davey DD, Kibble JD. Use of the NBME Comprehensive Basic Science Examination as a progress test in the preclerkship curriculum of a new medical school. Adv Physiol Educ. 2014;38(4):315–320.
13.		National Board of Medical Examiners. Subject Examination Program Comprehensive Basic Science Examination Score Interpretation Guide. Available from: http://www.nbme.org/pdf/SampleScoreReports/ComprehensiveBasicScience/CBSE%20Score%20Report.pdf. Accessed June 27, 2018.
14.		Weinberg E, Rooney JF. The academic performance of women students in medical school. J Med Educ. 1973;48(3):240–247.
15.		Case SM, Becker DF, Swanson DB. Performances of men and women on NBME part I and part II: the more things change.... Acad Med. 1993;68(10 Suppl):S25–S27.
16.		Gauer JL, Jackson JB. Relationships of demographic variables to USMLE physician licensing exam scores: a statistical analysis on five years of medical student data. Adv Med Educ Pract. 2018;9:39–44.
17.		Glew RH, Ripkey DR, Swanson DB. Relationship between students’ performances on the NBME Comprehensive Basic Science Examination and the USMLE Step 1: a longitudinal investigation at one school. Acad Med. 1997;72(12):1097–1102.
18.		Monteiro KA, George P, Dollase R, Dumenco L. Predicting United States Medical Licensure Examination Step 2 clinical knowledge scores from previous academic indicators. Adv Med Educ Pract. 2017;8:385–391.