Back to Journals » Advances in Medical Education and Practice » Volume 14

Student Performance on Course Objectives in a First-Year Medical School Foundational Science Course Correlate with USMLE Step 1 Scores

Authors Mathew RE, Riehs M, Blenda AV, Chosed RJ , Wright WS 

Received 14 September 2022

Accepted for publication 14 January 2023

Published 28 January 2023 Volume 2023:14 Pages 61—69

DOI https://doi.org/10.2147/AMEP.S389785

Checked for plagiarism Yes

Review by Single anonymous peer review

Peer reviewer comments 2

Editor who approved publication: Professor Balakrishnan Nair



Rhea E Mathew,1 Molly Riehs,1 Anna V Blenda,2 Renee J Chosed,2 William S Wright2

1School of Medicine, University of South Carolina School of Medicine Greenville, Greenville, SC, USA; 2Department of Biomedical Sciences; University of South Carolina School of Medicine Greenville, Greenville, SC, USA

Correspondence: William S Wright, University of South Carolina School of Medicine Greenville, Department of Biomedical Sciences, 701 Grove Road, Greenville, SC, 29605, USA, Tel +1 864 455 9865, Fax +1 864 455 8404, Email [email protected]

Purpose: Early identification of students at risk for poor United States Medical Licensing Examination® (USMLE) Step 1 examination (Step 1) performance allows medical schools to provide targeted intervention for those students. Therefore, determination of metrics that identify struggling students is necessary for proper intervention. We hypothesize that; 1) student performance on pre-matriculation metrics will correlate with their Molecular and Cellular Foundations of Medicine (FDNS) course performance and 2) student performance in the FDNS course and on specific FDNS course objectives will correlate with their Step 1 performance.
Methods: This was a retrospective cohort study analyzing data for students matriculating to the University of South Carolina School of Medicine Greenville in 2018 and 2019. Linear regression analysis was conducted to assess the correlation between pre-matriculation metrics, performance in the FDNS course, performance on FDNS objectives, and USMLE Step 1 performance. Adjusted R-squared (adjusted r2) values were compared with a p-value at < 0.05.
Results: The FDNS course grade correlated with pre-matriculation metrics of science undergraduate grade point average (uGPA), total uGPA, and the Medical College Admission Test (MCAT), adjusted r2 of 0.139, 0.121, 0.223, respectively. The FDNS course grade showed a stronger correlation to USMLE Step 1 performance (adjusted r2 = 0.257) than pre-matriculation metrics. USMLE Step 1 performance strongly correlated with FDNS course performance when two objectives, pertaining to anabolic and catabolic processes, regulation of cell cycle, and DNA replication and repair, were combined, adjusted r2 of 0.357.
Conclusion: The FDNS course grade and performance on specific course objectives could serve as a predictor for USMLE Step 1 performance and provides a more targeted and concise approach to identification of low-performing students and subsequent intervention.

Keywords: medical school curriculum, course objectives, course modifications, medical school performance, USMLE step 1

Introduction

It is critical for admissions committees to understand the importance of pre-matriculation criteria to ensure retention and timely graduation of medical students. Furthermore, identifying poor student performance early in medical school can allow for expedited academic intervention. Numerous studies have assessed predictive factors for medical school success. It has been reported that pre-matriculation metrics such as undergraduate grade point average (uGPA) and the Medical College Admissions Test (MCAT) correlate with first year medical school (MS1) performance, second year medical school (MS2) performance, and the United States Medical Licensure Examination® (USMLE) Step 1 examination (Step 1) score.1–4 In addition, several academic institutions have correlated the 2015 MCAT exam with quantitative metrics of MS1 GPA, MS2 GPA, average summative exam scores, and USMLE Step 1 performance and both the MCAT and uGPA served as better indicators for MS1 average summative examination performance than either metric alone.1 A recent study assessed the correlation between combined MCAT score and uGPA for 2016 and 2017 medical school matriculants and their performance in pre-clerkship and clerkship years as well as USMLE Step 1 and Step 2 Clinical Knowledge (CK) exams.2 They reported medium to large correlations between combined MCAT and uGPA for predicting student performance and progress in medical school. MS2 GPA closely correlated with MS1 GPA as well as uGPA and MCAT.3 Similar findings were noted for pre-matriculation metrics highly correlating with MS1 GPA, MS2 GPA, and licensure examinations in Doctor of Osteopathic Medicine (DO) programs.4

The impact of qualitative factors like undergraduate major or underrepresented minority status have been correlated with medical student performance. There was no difference in overall medical school performance between science and nonscience undergraduate majors.4,5 However, having a strong undergraduate science background could contribute to a better MCAT score.6 The uGPA and MCAT score may not be as effective at predicting MS1 GPA in underrepresented minorities in medicine. One multisite study found statistically insignificant differences in MCAT predictive capability for MS1 performance between students of different races, ethnicity, parental education, and gender, while another study noted that in minority students, uGPA seems to be more correlated with MS1 performance than MCAT score.1,3 In general, a negative correlation between underrepresented minority status and MS1 performance was found.7 Using both uGPA and MCAT provides stronger correlations and allows for higher predictive value for MS1 performance and assessing both statistics could account for variances noted due to sociodemographic differences among students.1,3

Previous studies have largely focused on assessing whether the MCAT and uGPA served as predictors of medical school success. While MCAT and uGPA correlate with MS1 and MS2 performance, the MCAT and preclinical scores are more highly correlated with success on future standardized examinations. MCAT has a high predictive value for further success in medical school including USMLE Step 1 performance and the National Board of Medical Examiners® (NBME) Subject Exams.8 MCAT shows less predictive value for Objective Structured Clinical Examination (OSCE) and post-graduation success including residency evaluations as indicated by a military medicine study.9 MS1 GPA and MS2 GPA were highly correlated with USMLE Step 1 and further clinical performance.3 The first medical school exam grade was more strongly correlated with USMLE Step 1 score when compared to pre-matriculation factors alone.10 One study assessed the predictive value of NBME Comprehensive Basic Science Self-Assessment (CBSSA) as part of the Foundation in Medicine and Musculoskeletal modules in the first 5 months of medical school. This study showed that low scores on NBME CBSSA serves as an indicator for poor academic performance and low USMLE Step 1 scores.11

Predicting how a student will perform once in medical school remains a challenge as many factors appear to play a role. Additionally, the identification of a metric to distinguish struggling students before matriculation or early in medical school is essential so that learners can be offered academic assistance prior to USMLE Step 1. A meta-analysis published in 2007 revealed that the MCAT alone has a fairly high predictive value for the USMLE Step 1 (r = 0.66) however, the MCAT does not provide a very good predictive value for performance in pre-clinical course work (r= 0.43).12 Another study utilized a different outcome to assess medical student success across schools that they termed “unimpeded progress toward graduation” which reported that the combination of MCAT and uGPA were the best predictors for student success.13 A very recent study took a longitudinal approach to access student success on NBME tests during the first year of medical school.14 The authors found that the students who showed “growth” or continued improvement in these assessment scores during the first year were more likely to perform better on the USMLE Step 1 than their classmates who did not show the same pattern of improvement.

There have been a few studies indicating the utility in using early exam grades as a predictor of medical school performance. However, there are no known studies that have correlated USMLE Step 1 performance with performance on course assessment questions mapped to course level objectives. Therefore, the specific aim of this study was to determine if student performance on Molecular and Cellular Foundations of Medicine (FDNS) course objectives correlates with FDNS course performance and USMLE Step 1 performance. We hypothesize that; 1) student performance on pre-matriculation metrics will correlate with their FDNS course performance and 2) student performance in the FDNS course and on specific FDNS course objectives will correlate with their Step 1 performance. Knowing early in medical school whether specific course objectives correlate with medical student performance allows for more directed course improvement as well as early targeted intervention for struggling students.

Methods

Data Collection

This was a retrospective cohort study that analyzed performance data for students matriculating to the University of South Carolina School of Medicine Greenville in 2018 (N = 110) and 2019 (N = 106). The following deidentified data were collected: science uGPA, total uGPA, MCAT score, student performance in the FDNS course (the first biomedical science course in the first year of the curriculum) and test item performance on FDNS course objectives. Test items are developed and mapped to FDNS course objectives by the course faculty (ie course content experts teaching the material) in ExamSoft®, the assessment platform used to administer FDNS exams. Test items may be mapped to multiple course objectives.

Molecular and Cellular Foundations of Medicine Course Description and Grading

The FDNS course is the first biomedical science module in the pre-clerkship curriculum for the University of South Carolina School of Medicine Greenville which has been previously described.15 The FDNS course covers the basic science content for biochemistry, genetics, cell biology, molecular biology, and introduction to pharmacology. The overall grade for the FDNS course consists of points earned through weekly team-based learning assessments that account for 5% of the course grade (individual readiness assurance test and the group readiness assurance test) and a final exam consisting of multiple-choice exam questions that accounts for 95% of the course grade. Test items are written by content experts teaching the course, and NBME format is followed in most cases.

Data Analysis

Data were analyzed using the statistics program International Business Machines (IBM) Statistical Package of the Social Science (SPSS) software, version 27 (IBM Corporation, Armonk, NY). Pearson product-moment correlation analysis was performed to determine if there was a correlation with pre-matriculation data (ie science uGPA, total uGPA, and MCAT score) and/or FDNS course objectives with the FDNS course grade. In addition, a Pearson product-moment correlation analysis was performed to determine if there was a correlation with pre-matriculation data, FDNS course objectives, or FDNS course performance with the USMLE Step 1 score (3-digit-score). A multivariate regression analysis was performed to determine which independent variable combinations might provide a stronger correlation to FDNS course performance. Statistical significance was set at a p-value <0.05. The sample analyzed included two cohorts of medical students from two consecutive years (matriculation years 2018 and 2019) who had completed the FDNS course and taken the USMLE Step 1.

Ethical Approval

This study was reviewed and considered exempt by the University of South Carolina Institutional Review Board (IRB).

Results

Figure 1 provides the grade distribution for the final exam in FDNS and the overall grade distribution for the course. The FDNS course grade was weakly-moderately correlated with pre-matriculation metrics of science uGPA, total uGPA, and MCAT, adjusted r2 of 0.139, 0.121, 0.223, respectively (Table 1). USMLE Step 1 performance was also weakly correlated with science uGPA, total uGPA, and MCAT, adjusted r2 of 0.095, 0.07, and 0.153, respectively (Table 1). The FDNS course grade showed a moderate correlation to USMLE Step 1 scores, adjusted r2 = 0.257 (Table 1).

Table 1 Correlation Between Pre-Matriculation Metrics and Molecular and Cellular Foundations of Medicine Course Objective Performance with Molecular and Cellular Foundations of Medicine Course Grade and USMLE® Step 1 Score

Figure 1 Student performance on summative assessments in the molecular and cellular foundations of medicine course. Summative course exam and final course grade. Number of students = 211.

Specific FDNS course objectives are listed in Table 2 along with the number of course exam items mapped to each objective. Questions were mapped to one or more objective(s) by course faculty (ie course content experts teaching the material). The FDNS course grade was more strongly correlated with performance on course objectives pertaining to cellular structure and growth (ie objective 1), pharmacokinetics (ie objective 9), and lifestyle medicine (ie objective 10) compared to the other course objectives, adjusted r2 of 0.178, 0.158, and 0.232, respectively (Table 1). USMLE Step 1 performance was more strongly correlated with FDNS course performance when objectives pertaining to catabolic and anabolic process (ie objective 4) and regulation of cell cycle and DNA replication and repair (ie objective 7) were combined, r(190) = 0.598, p = <0.001 (adjusted r2 of 0.357).

Table 2 Molecular and Cellular Foundations of Medicine Course Objectives for Academic Years 2018–2019 and 2019–2020

Figure 2A presents the student performance on all questions mapped to a single objective. FDNS course objectives pertaining to cellular structure and growth (ie objective 1), evaluation of pedigrees (ie objective 6), and pharmacokinetics (ie objective 9) had >20% of students perform <69.5% on all items mapped to that objective while objectives pertaining to regulation of cell cycle and DNA replication and repair (ie objective 7) and clinical decision making uses biochemical and molecular testing (ie objective 8) had <6% of students perform <69.5% on all items mapped to that objective. Figure 2B presents student performance across multiple course objectives. That is, how many students performed >69.45 on all ten objectives, on nine objectives, etc. Twenty-eight percent of students scored >69.45 on all questions mapped to each objective while 3.3%, 1.4%, and 2.4% of students scored >69.45 on all questions mapped to either 5, 4, or 3 objectives, respectively.

Figure 2 Student performance on molecular and cellular foundations of medicine course objectives. (A) Student performance on all questions mapped to molecular and cellular foundations of medicine course objectives. Obj 1: N = 15 (7.28); Obj 2: N = 15 (7.28); Obj 3: N = 18 (8.74); Obj 4: N = 55 (26.70); Obj 5: N = 14 (6.80); Obj 6: N = 28 (13.59); Obj 7: N = 15 (7.28); Obj 8: N = 31 (15.05); Obj 9: N = 6 (2.91); Obj 10: N = 9 (4.37). Obj. = Molecular and cellular foundations of medicine objective. N(%) = number of questions (% of questions) mapped to each molecular and cellular foundations of medicine objective. A question can be mapped to more than one FDNS objective. Number of students = 211. (B) Percent of students scoring ≥69.5% on molecular and cellular foundations of medicine course objectives. Number of students = 211.

Discussion

Our data presents correlations of student performance outcomes based on course objectives in a medical school FDNS course to the USMLE Step 1 performance. Providing such insight will help identify struggling students for early targeted intervention which has been shown to improve future performance.10 This study may also inform other medical school foundational science course directors as they work to improve their own courses as this process of analyzing course objective outcomes provides targeted opportunities for course adjustment in content time allocation, additional review, etc.

In addition, we provide further analysis of student performance on individual objectives within a course and how those objectives correlate with FDNS course grades. Student performance on FDNS course objectives and its correlation with course performance allow for targeted improvements for course design. Specifically, performance on cellular structure and growth, pharmacokinetics, and lifestyle medicine more strongly correlated with the overall FDNS grade compared to other course objectives. The strongest correlations with student performance mapped to course objectives that represented only a small portion of the assessed material in the course, specifically objectives 1, 9, and 10 (3 out of 10 module objectives; 27 out of 203 assessment mappings). Ideally, we would want all course objectives to have strong correlations, especially those that represent the larger portions of assessed materials (objectives 4, 6, and 8). This finding suggests the need to further examine how items were mapped for each objective as well as further examine the content taught for each objective. We could consider mapping each assessed item to the single best objective to more accurately assess student performance on each objective in the course. This is also an interesting finding given that cellular structure, signaling, and growth (ie objective 1) are often discussed in numerous required undergraduate courses while pharmacokinetics and lifestyle medicine (ie objectives 9 and 10, respectively) are generally new topics for incoming students.

Within our curriculum, pre-matriculation variables have a weak correlation with performance in the FDNS module and on USMLE Step 1 performance. The FDNS module grade had a significantly higher correlation with USMLE Step 1 performance compared to pre-matriculation variables which is consistent with other studies.10,11 We also show an even stronger correlation with USMLE Step 1 performance when FDNS course performance and two FDNS objectives (4 and 7), pertaining to anabolic and catabolic processes, regulation of cell cycle, and DNA replication and repair, were combined. While no single course objective alone revealed a strong correlation with USMLE Step 1, this could be due to the narrow focus of the objectives within this first-year basic science course. The manner in which this content appears on USMLE Step 1 is typically embedded within a vignette that includes a specific pathology description. At the stage when this course is taught (the first basic science module of the first year) the students have not been exposed to disease pathology. Therefore, our finding that no single FDNS course objective had a strong correlation with Step 1 performance may reflect the aggregation of knowledge throughout our first and second year curriculum is what is influencing Step 1 success.

Greater than forty-eight percent of students averaged >89.5% on exam items mapped to course objectives 2, 3, 5, 7, and 8 (Figure 2A). This is not a surprising finding given that these objectives, except objective 8, cover areas that would require some level of background knowledge from undergraduate training (Table 2). However, good performance on objectives that are novel to the incoming student might be more reflective of effective study strategies and time management skills. One study found that study skills and time management ability were often more predictive than MCAT or uGPA.16 It is critical to establish foundational knowledge upon entering medical school. In addition, development of study strategies and time management skills are essential to future medical school success. This can be further addressed through qualitative studies assessing student learning styles correlated with performance on specific module objectives or module performance.

Future directions for this research could include tracking performance of specific course objectives throughout pre-clinical years which could help identify specific content areas for targeted improvements on USMLE Step 1 performance, clinical performance, and provide specific information for targeted course improvements. Further study will analyze the course objectives as these topics are further reinforced throughout the MS1 and MS2 curriculum to determine if performance on reinforced topics provide a stronger correlation to Step 1.

This type of analysis with assessing course objectives with Step 1 performance can be an especially useful metric for schools with truly integrated curricula where foundational content is spread over multiple course modules. Course directors and teaching faculty typically rely on student feedback, peer evaluation and average course grades to gauge the success of a given course and then determine if course modifications should be made. However, the targeting of course changes could be better informed if student performance on specific course objectives was known to correlate with Step 1 performance. In this situation, course directors and teaching faculty could modify how a given objective content was delivered, add additional time for specific topics within “impactful” objectives, or allow for review hours for these objectives.

One limitation of the study is that the analysis on USMLE Step 1 performance is from one institution. A second limitation of the study is that results are dependent on stated objectives where each assessed item can be mapped to multiple objectives. Additionally, mapping of assessment items is done by several assessment item authors, due to multiple faculty teaching in module, which introduces some level of variations in mapping to objectives. A third limitation of the study is the change to USMLE Step 1 to pass/fail. How the change in USMLE Step 1 scoring will impact predictions of student performance has yet to be analyzed and reported.

Conclusion

The FDNS course grade and performance on specific course objectives could serve as a predictor of USMLE Step 1 performance. Poor performance in the course or on identified course objectives could allow for early student identification and targeted intervention. Analysis of the relationship between specific module objectives and USMLE Step 1 performance provides a more targeted and concise approach to identification and intervention of low-performing students. In addition, the findings of this study could be used to improve the foundations course design, course content, assessment item mapping and refinement of course objectives.

Ethics Approval

This study received exemption from Human Research Subjects on 6/4/2020 by the University of South Carolina institutional review board. The research was exempted in accordance with 45 CFR 46.104(d)(4). The research is compliant with the Declaration of Helsinki. The data was provided to researchers in a deidentified file and data were stored in a password protected electronic file. The reference number is Pro00100807.

Acknowledgments

We are thankful for the data support provided by Kathryn Johnson, Alex Tharpe, and Amanda Piekutowski. The data/abstract of this paper was presented at the ABE/APHMG 2021 virtual meeting as a poster presentation with interim findings. The poster’s abstract and poster are presented on ResearchGate. Link: https://www.researchgate.net/publication/353601029_Student_Performance_on_Specific_Course_Objectives_in_a_First-Year_Foundational_Science_Course_Correlate_with_Step_1_Scores.

Funding

There is no funding to report.

Disclosure

The authors report no conflicts of interest in this work.

References

1. Busche K, Elks ML, Hanson JT, et al. The validity of scores from the new MCAT exam in predicting student performance: results from a multisite study. Acad Med. 2020;95(3):387–395. doi:10.1097/ACM.0000000000002942

2. Hanson JT, Busche K, Elks ML, et al. The validity of MCAT scores in predicting students’ performance and progress in medical school: results from a multisite study. Acad Med. 2022;97(9):1374–1384. doi:10.1097/ACM.0000000000004754

3. White CB, Dey EL, Fantone JC. Analysis of factors that predict clinical performance in medical school. Adv Health Sci Educ Theory Pract. 2009;14(4):455–464. doi:10.1007/s10459-007-9088-9

4. Dixon D. Prediction of osteopathic medical school performance on the basis of MCAT score, GPA, sex, undergraduate major, and undergraduate institution. J Am Osteopath Assoc. 2012;112(4):175–181.

5. Smith SR. Effect of undergraduate college major on performance in medical school. Acad Med. 1998;73(9):1006–1008. doi:10.1097/00001888-199809000-00023

6. Cuddy MM, Swanson DB, Clauser BE. A multilevel analysis of examinee gender and USMLE step 1 performance. Acad Med. 2008;83(10 Suppl):S58–S62. doi:10.1097/ACM.0b013e318183cd65

7. Schneid SD, Apperson A, Laiken N, Mandel J, Kelly CJ, Brandl K. A summer prematriculation program to help students succeed in medical school. Adv Health Sci Educ Theory Pract. 2018;23(3):499–511. doi:10.1007/s10459-017-9808-8

8. Hu Y, Martindale JR, LeGallo RD, White CB, McGahren ED, Schroen AT. Relationships between preclinical course grades and standardized exam performance. Adv Health Sci Educ Theory Pract. 2016;21(2):389–399. doi:10.1007/s10459-015-9637-6

9. Saguil A, Dong T, Gingerich RJ, et al. Does the MCAT predict medical school and PGY-1 performance? Mil Med. 2015;180(4 Suppl):4–11. doi:10.7205/MILMED-D-14-00550

10. Bennion LD, Torre D, Durning SJ, et al. Early identification of struggling learners: using prematriculation and early academic performance data. Perspect Med Educ. 2019;8(5):298–304.

11. Gullo CA, Shapiro JI, Miller BL. Predicting medical student success on licensure exams. Med Sci Educ. 2015;25:447–453. doi:10.1007/s40670-015-0179-6

12. Donnon T, Paolucci EO, Violato C. The predictive validity of the MCAT for medical school performance and medical board licensing examinations: a meta-analysis of the published research. Acad Med. 2007;82(1):100–106. doi:10.1097/01.ACM.0000249878.25186.b7

13. Dunleavy DM, Kroopnick MH, Dowd KW, Searcy CA, Zhao X. The predictive validity of the MCAT exam in relation to academic performance through medical school: a national cohort study of 2001-2004 matriculants. Acad Med. 2013;88(5):666–671. doi:10.1097/ACM.0b013e3182864299

14. Wang L, Laird-Fick H, Parker C, Liao Z, Solomon D. Growth in basic science knowledge in first-year medical school and USMLE Step 1 results: a longitudinal investigation at one school. Adv Health Sci Educ Theory Pract. 2022;27(3):605–619. doi:10.1007/s10459-022-10104-y

15. Simpson R, Conrad H, Blackwell TH, Wright WS. A student survey: influence of emergency medical technician training on student’s application, matriculation, and transition into medical school. Adv Med Educ Pract. 2022;13:227–235. doi:10.2147/AMEP.S340553

16. West C, Sadoski M. Do study strategies predict academic performance in medical school? Med Educ. 2011;45(7):696–703. doi:10.1111/j.1365-2923.2011.03929.x

Creative Commons License © 2023 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.