Pre-matriculation clinical experience positively correlates with Step 1 and Step 2 scores

Introduction

Despite advances in a more holistic approach to medical school admissions, the selection of applicants into residency is determined, in large part, by their United States Medical Licensing Exam (USMLE) Step 1 and Step 2 scores.^1–3 Residency program directors note that this is often a pivotal factor when deciding whether or not to offer an applicant an interview.^2,4 Despite evidence against this practice and pleas to the contrary, the USMLE Step 1 and Step 2 will continue to play a large role in determining the potential choices future that residency applicants have in choosing their medical specialty.^1,3

The Medical College Admissions Test (MCAT) and undergraduate grade point average (U-GPA) are the two pre-admission variables that have received the most attention in the literature and have been shown to positively correlate with USMLE Step 1 and Step 2 scores.^5,6 However, other non-academic factors have received very little attention as to how well they predict performance on these two high stakes exams. Admissions committees view self-reported clinical experience positively.⁷ However, despite this expectation, there exists a paucity of information focusing on the benefit of this experience in terms of USMLE performance.^7,8

We noted anecdotally that registered nurses (RNs) attending medical school at our institution did extremely well in the USMLE Step 1 and Step 2 in recent years, which stimulated interest in the correlation between formal pre-matriculation medical experience and these two exams. The purpose of this study was to determine whether pre-matriculation formal clinical experience of students was a positive predictor of USMLE Step 1 and Step 2.

Methods

This research investigation was a retrospective, cohort study conducted at the University of Arizona College of Medicine (UACOM) in Tucson, Arizona, USA, and analyzed students in the Class of 2017 and 2018. The UACOM has a 4-year medical school curriculum, with the first 2 years consisting of a pre-clinical, systems-based, basic science curriculum. Like most present-day curriculum, instruction is interactive, small group, and clinically case-based. The USMLE Step 1 exam is typically taken at the end of the students’ second year prior to beginning clinical clerkships, while the USMLE Step 2 exam is generally taken after the students have rotated through all required clinical clerkships, which include 6 weeks of Family Medicine, Pediatrics, Surgery, Obstetrics and Gynecology, and Psychiatry, 12 weeks of Internal Medicine, 3 weeks of Neurology, and a 3-week elective of the student’s choice.

Data were recorded from two primary programs: Docuware and records from the UACOM admissions office regarding students’ American Medical College Application Service (AMCAS) application information. Docuware is an online program used by the UACOM that stores Medical Student Performance Evaluations, which were used to record student’s USMLE Step 1 and Step 2 scores. First-time scores were used for both examinations regardless of pass or fail status. Variables recorded from AMCAS included demographic information including classification as an underrepresented minority (URM) or as having a disadvantaged background, MCAT score, U-GPA, and formal and non-formal pre-matriculation clinical experience. Formal clinical experience was defined as RN, physician assistant, nurse practitioner, paramedic, emergency medical technician (EMT), or licensed practical nurse. Applicants with at least 6 months of scribing or patient care technician experience prior to matriculation were also considered under this group. Students with shadowing experience for any amount of time were considered to have non-formal clinical experience.

All student data were immediately de-identified after extraction from the records. Upon completion of data retrieval, we standardized the data using the definitions already described for pre-matriculation clinical experience. We summarized student’s characteristics by clinical experience using mean ± SD for continuous variables and using frequency and the associated percentage for categorical variables. Two-sample t-tests were performed to compare continuous variables and Fisher’s exact tests were performed to compare categorical variables. Of the 227 students in both classes, nine (3.96%) students were not assessed in USMLE Step 1 calculation, and 61 (26.9%) students were not assessed in USMLE Step 2 calculations due to an absence of recorded USMLE scores. Two-sample t-tests were performed to compare USMLE Step 1 and Step 2 scores between students with and with no formal pre-matriculation clinical experience. Multiple linear regression was conducted to control for student’s characteristics (including ethnicity, gender, U-GPA, MCAT score, URM, and disadvantaged background) while assessing the effects of the clinical experience. MCAT and U-GPA were included in the multiple regression analyses due to theoretical and previously reported associations with USMLE Step 1 and Step 2 score outcomes. The URM and disadvantaged background classification were included in the multiple regression analyses due to former published associations between these demographic factors and USMLE exam performance.⁹ Statistical significance was defined as P<0.05. All statistical analyses were performed using SAS 9.4.

Ethics approval and informed consent

This research was approved by the UACOM Institutional Review Board (IRB# 13 – 0418), and deemed exempt for obtaining consent from the participants.

Results

Our study had a total of 227 students from the two classes, with 40 (17.6%) having formal pre-matriculation clinical experience, as already defined. Of those 40, 25 (62.5%) were scribes, 9 (22.5%) nurses, 3 (7.5%) EMTs, 2 (5%) paramedics, and 1 (2.5%) pharmacist. Table 1 displays the summary of U-GPA, MCAT score, and other demographic data between students with formal pre-matriculation clinical experience and those without. There were no statistical differences between students with formal pre-matriculation clinical experience and those without. In the latter group, shadowing was the main clinical experience listed. All students in our study had some form of pre-matriculation clinical experience, either formal or non-formal, as is strongly recommended in the UACOM website. Table 2 displays the summary of USMLE Step 1 and Step 2 scores between students with formal pre-matriculation clinical experience and those without. After controlling for ethnicity, gender, U-GPA, MCAT scores, URM, and the disadvantaged status, the students with formal pre-matriculation clinical experience had a statistically significantly higher Step 1 and Step 2 scores than those with no formal pre-matriculation clinical experience with an adjusted P-value of 0.03 and <0.01, respectively. In addition to the formal clinical experience, based on the multiple linear regression models, we also found that U-GPA and MCAT scores were significant predictors for USMLE Step 1 score (P<0.01, P<0.0001, respectively) and USMLE Step 2 score (P=0.01, P<0.01, respectively). Specifically, students with higher U-GPA and MCAT scores had significantly higher Step 1 and Step 2 scores. Other demographic factors included in the regression (URM and disadvantaged student classification) did not significantly correlate with Step 1 or Step 2 score.

Table 1 Summary of student characteristics by clinical experience (Yes vs No) Notes: aDerived from Fisher’s exact test for categorical variables and two-sample t-test for continuous variables. bMean±SD. Abbreviations: U-GPA, undergraduate grade point average; MCAT, Medical College Admissions Test; URM, underrepresented minority.

Table 2 Summary of Step 1 and Step 2 scores by clinical experience (Yes vs No) Notes: aDerived from two-sample t-test. bDerived from linear regression adjusting for student characteristics, including U-GPA, MCAT score, ethnicity, gender, URM, and disadvantaged status. cMean±SD. Abbreviations: U-GPA, undergraduate grade point average; MCAT, Medical College Admissions Test; URM, underrepresented minority.

Discussion

Numerous variables are considered by an admissions committee during the medical school application process. Some of these, such as U-GPA and MCAT score, have been extensively investigated, and they not only predict academic success in medical school, but also correlate positively with USMLE Step 1 and Step 2.^5,6,9 Our findings are consistent with these prior findings. However, a more holistic review also considers other variables such as resilience, distance traveled, cultural diversity, and prior medical experience. These qualities are intuitively important in future physicians but are hard to define and difficult to study.^10,11 It seems rational that prior to committing extensive time, energy, and cost to a career in medicine, one would expect a student to gain exposure to the field. Despite this expectation, there exists a paucity of information focusing on the benefit of this experience in terms of academic performance.^7,8

Our study demonstrated a statistically significant correlation between formal medical experience prior to matriculation and USMLE Step 1 and Step 2 scores (Table 2). A formal, structured, clinical experience that would result in a higher USMLE Step 1 score is not intuitive as Step 1 tests basic science knowledge while Step 2 is a more clinically based exam. However, the Cognitive Load Theory (CLT) may help explain why a structured clinical experience might result in a higher Step 1 and Step 2 score than students without said experience.^12–16

Effective learning, according to CLT, is a product of the complexity of the material (the intrinsic load) and the tools the learner possesses to interpret and synthesize this information (the germane load). Ideally, the student converts this material into long-term memory (LTM) to propagate learning and future problem solving. Intrinsic load is dependent on the number of unknowns in a problem and thus is inherent to the problem itself. As unknown items in a problem increase, so does the intrinsic load and the difficulty of transferring knowledge into their LTM for the novice student lacking experience. Thus, medical students exposed to an interactive, medical position prior to matriculation might be expected to have a more robust germane load compared to the student lacking such experience.

All USMLE Step 1 and Step 2 questions contain a clinical vignette for delivery and require an understanding of the relationship between basic science and clinical medicine. Medical schools, recognizing the importance of USMLE exam performance, have followed suit, changing their exams during the basic science years to parallel the USMLE format.¹ Clinically experienced students also have the advantage in this situation, due to their increased germane load. Thus, one might argue that a clinically experienced group may benefit in test performance, especially USMLE Step 1 and Step 2.

Artino et al noted that admissions committees view self-reported clinical experience positively and are more likely to select students with such experience.⁷ However, they found that students with self-reported clinical experience performed no better than their counterparts in the USMLE Step 1 and Step 2.⁷ One criticism of that study is that clinical experience was neither defined nor was the duration reported. Paolino et al investigated clinical experience and performance with Step 1 and Step 2 and found very small differences between groups without clinical experience vs those with varied clinical experiences. However, the authors failed to control for MCAT scores and included clinical experiences that we do not consider patient interactive.⁸ We only included positions that require interactive, direct patient care in our formal clinically experienced group. Shadowing was not included in this group since it is a more passive experience in which the student follows the physician in their practice for an undefined period of time. This is an observational role for most participants with no direct patient care experience. This is in sharp contrast to a scribe, a paid position in which the scribe and physician interact directly concerning patient care.

There are significant limitations to our study. It represents a single-institution study of a small number of students over two years. The correlation could be strengthened via the addition of more students from our institution and other institutions nationwide. In addition, we employed a holistic approach to admissions, relying less on MCAT and U-GPA. Thus, our mean MCAT and U-GPA score for accepted applicants may be lower than at other schools. It might be that at a higher MCAT and/or U-GPA score, the difference between our two groups may disappear. Furthermore, we failed to record and control for age in our analysis. It is theorized that older students might have performed better when compared to their younger classmates. Although it would have been interesting to compare the two groups based on their academic performance during the basic science instruction of the first two years, this was not performed secondary to our pass/fail system. Finally, participants’ undergraduate major was not studied. Based on CLT, it might be expected that the science vs non-science majors might perform better in Step 1 based on the former’s exposure to terminology and similar format as medical school.

Conclusion

In summary, our study demonstrates a statistically significant correlation between higher USMLE Step 1 and Step 2 scores in medical students with formal pre-matriculation clinical experience. If our findings are reproduced in future studies, then we will have one more variable besides the traditional MCAT and U-GPA to predict future success in USMLE Step 1 and Step 2 exams, allowing for a more holistic admission process while upholding the academic excellence of each institution.

Acknowledgment

The authors would like to thank the UACOM admissions office for their generosity and time in acquiring the data for this study.

Author contributions

RS made substantial contributions to the study conception and design, data acquisition, and the first and final draft of the article.

CJ was responsible for data analysis and interpretation and contributed to the first and final draft of the article.

DR made substantial contributions to study conception and design as well as the first and final draft of the article.

LAB made substantial contributions to data interpretation and to the first and final draft of the article.

WA-R was the senior author of this project and made substantial contributions to the study conception and design, data interpretation, and the first and final draft of the article.

All authors contributed toward data analysis, drafting and critically revising the paper and agree to be accountable for all aspects of the work.

Disclosure

The authors report no conflicts of interest in this work.

References

1.		Prober CG, Kolars JC, First LR, Melnick DE. A plea to reassess the role of United States medical licensing examination step 1 scores in residency selection. Acad Med. 2016;91(1):12–15.
2.		National Resident Matching Program. Data Release and Research Committee: Results of the 2014 NRMP Program Director Survey. Washington, DC: National Resident Matching Program; 2014. Available from:http://www.nrmp.org/wp-content/uploads/2014/09/PD-Survey-Report-2014.pdf. Accessed September 13, 2018.
3.		Katsufrakis PJ, Uhler TA, Jones LD. The Residency Application Process: Pursuing Improved Outcomes Through Better Understanding of the Issues. Acad Med. 2016;91(11):1483–1487.
4.		Gliatto P, Leitman IM, Muller D. Scylla and Charybdis. Academic Medicine. 2016;91(11):1:1498–1500.
5.		Mitchell K, Haynes R, Koenig J. Assessing the validity of the updated Medical College Admission Test. Acad Med. 1994;69(5):394–401.
6.		Wiley A, Koenig JA. The validity of the Medical College Admission Test for predicting performance in the first two years of medical school. Acad Med. 1996;71(10 Suppl):S83–85.
7.		Artino AR, Gilliland WR, Waechter DM, Cruess D, Calloway M, Durning SJ. Does self-reported clinical experience predict performance in medical school and internship? Med Educ. 2012;46(2):172–178.
8.		Paolino ND, Artino AR, Saguil A, Dong T, Durning SJ, Dezee KJ. Predicting medical school and internship success: does the quality of the research and clinical experience matter? Mil Med. 2015;180(4 Suppl):12–17.
9.		Julian ER. Validity of the Medical College Admission Test for predicting medical school performance. Acad Med. 2005;80(10):910–917.
10.		Witzburg RA, Sondheimer HM. Holistic review--shaping the medical profession one applicant at a time. N Engl J Med. 2013;368(17):1565–1567.
11.		Mahon KE, Henderson MK, Kirch DG. Selecting tomorrow’s physicians: the key to the future health care workforce. 2013;88(12):1806–1811Acad Med.
12.		Albanese MA, Mitchell S. Problem-based learning: a review of literature on its outcomes and implementation issues. Acad Med. 1993;68(1):52–81.
13.		Neville AJ. Problem-based learning and medical education forty years on. A review of its effects on knowledge and clinical performance. Med Princ Pract. 2009;18(1):1–9.
14.		Young JQ, van Merrienboer J, Durning S, Ten Cate O. Cognitive Load Theory: implications for medical education: AMEE Guide No. 86. Med Teach. 2014;36(5):371–384.
15.		Kirschner PA, Sweller J, Clark RE. Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educ Psychol. 2006;41(2):75–86.
16.		Sweller J. Cognitive load during problem solving: Effects on learning. Cogn Sci. 1988;12(2):257–285.