Use of the National Board of Medical Examiners^® Comprehensive Basic Science Exam: survey results of US medical schools

Introduction

US medical schools vary greatly with respect to curriculum design and assessment methods, yet every student must complete the United States Medical Licensing Examination^® (USMLE). This examination process assesses the readiness of an individual to become a licensed physician. The USMLE is a three-step examination; this article focuses on Step 1, which is “designed to measure basic science knowledge”.¹ Regardless of structure or pedagogical approach, a medical school curricula must ensure that a medical student is prepared for the USMLE Step 1 as this exam is considered as one of the top factors in selecting applicants to interview and in ranking applicants for the National Resident Match Program^®2 and is a requirement for medical licensure by most state authorities in the US.³

An externally validated measure of performance is one method schools use to determine the preparedness of each student for the USMLE Step 1. The National Board of Medical Examiners^® (NBME) offers a subject examination program that provides validated assessment tools for medical education. One such subject exam is the Comprehensive Basic Science Exam (CBSE). The NBME CBSE is strongly correlated with USMLE Step 1 performance^4–7 and has been shown to identify students “at increased risk for poor performance on the USMLE Step 1”.⁷ The NBME CBSE has also been used to provide external validation of internal assessments within new US medical schools^5,8 and effectiveness of revised medical education curricula at an established medical school.⁹ Although there have been reports of how the NBME CBSE has been used at individual medical schools,^4–6,8,9 its utilization across US medical schools, to our knowledge, has not been reported.

The aim of the current study was to determine 1) the extent to which US medical schools use the NBME CBSE; 2) how many times a school uses the NBME CBSE prior to students taking the USMLE Step 1; and 3) whether schools administer the NBME CBSE for curriculum change and/or identification of students performing below expectations.

Methods

Survey methods

A list of schools was obtained from the Liaison Committee on Medical Education (LCME^®) website for accredited MD programs in the US. There were 145 schools listed on the LCME website as of February 29, 2016, with preliminary (six schools), provisional (four schools), or full (135 schools) accreditation. The Associate Dean for Curriculum, or equivalent, was determined by review of each school’s website for those schools with provisional or full accreditation (n = 139). An email message was sent along with the link to the anonymous survey to the curriculum leader identified at each of these schools with provisional or full LCME accreditation as of February 29, 2016, for a total of 139 schools. This was a voluntary survey which was hosted at SurveyMonkey^®. An initial email request to complete the survey was sent on May 3, 2016. A follow-up reminder email to all potential respondents was sent on May 10, 2016. The survey closed on May 16, 2016. The survey asked the initial question “Does your institution administer the NBME CBSE prior to the USMLE Step 1?”. Respondents were asked to complete a series of follow-up questions that were different if the answer to the initial question was “yes” or “no”. A review by the institutional review board of University of South Carolina was conducted, and it was determined that this was not human subject research.

Statistics

Data were expressed as mean (standard deviation of the mean). Mann–Whitney test was performed with GraphPad InStat version 3.10 software (GraphPad Software, Inc., San Diego, CA, USA), using a criterion of p < 0.05 as statistically significant. Percentages were rounded to the nearest whole number.

Ethical approval

A review by the institutional review board of the University of South Carolina was conducted, and it was determined that this was not human subject research and therefore approval was not required.

Results

There were 53 responses (38% response rate) to the first question in the survey regarding the use of the NBME CBSE with the number of responses to follow-up questions slightly lower. The NBME CBSE was administered by 37 of the schools (70%) responding, while 16 schools (30%) did not administer the NBME CBSE. Of the schools that responded “yes” to administering the NBME CBSE, all except one reported that the exam was paid for by the institution (n = 36, 97%) and the other school did not respond.

Among the schools responding to the question “Which of the following are used for the basic science courses at your institution?” (n = 48), 34 (71%) used pass/fail or pass/fail/honors for grading basic science courses, while 14 (29%) used a letter grade. There were five schools that did not respond to the question. The percentage of schools using each grading method for basic sciences courses was different than those reported on the LCME Annual Medical School Questionnaire Part II, 2012–2013 through 2015–2016.¹⁰ Figure 1 presents the grading methods used by schools that administered the NBME CBSE for basic science courses.

Figure 1 Grading methods for the basic science courses broken down by schools that did and did not administer the NBME CBSE (n = 48). Abbreviations: NBME, National Board of Medical Examiners®; CBSE, Comprehensive Basic Science Exam.

Purposes and timing of administration of the NBME CBSE

Data presented cover only schools that administered the NBME CBSE. In all, 28 schools (78%) out of 36 respondents (one school did not respond) that administered the NBME CBSE prior to the USMLE Step 1 identified students performing below expectations and provided targeted intervention strategies. Forms of intervention provided were online coursework (n = 3, 11%), strategies for studying and/or test taking (n = 26, 93%), additional coursework provided by faculty (n = 3, 11%), exam preparation course (n = 11, 39%), and others (n = 11, 39%). Percentages did not equal 100% as respondents could select as many intervention strategies as appropriate. Table 1A provides a breakdown of the combination of intervention strategies used by schools, while Table 1B provides intervention strategies schools listed as other.

Table 1 Breakdown of intervention strategies used by schools that utilized the NBME CBSE to identify students performing below expectations (A), n = 28, and specific responses provided for the intervention strategy “Other” (B), n = 11 Abbreviations: NBME, National Board of Medical Examiners®; CBSE, Comprehensive Basic Science Exam; USMLE, United States Medical Licensing Examination®.

In all, 13 schools (36%) out of 36 respondents (one school did not respond) used the data from the NBME CBSE to make changes to the curriculum. Six schools (17%) used the exam as part of the formative assessment for a course, while five schools (14%) used the exam as part of the summative assessment for a course.

Eight schools (23%) out of 35 respondents (two schools did not respond) reported that students are required to make a minimum score (as determined by the institution) on the NBME CBSE prior to attempting the USMLE Step 1. Schools that require a minimum score on the NBME^® CBSE trend higher (P=0.1189; although not statistically significant) on the number of exams administered (2.00±1.3) compared to the number of exams administered (1.37±0.84) for schools that do not require students to make a minimum score on the exam.

Seven schools (20%) out of 35 respondents (two schools did not respond) reported that students are instructed to study for the NBME CBSE, while eight schools (23%) reported that students are instructed “not” to study for the NBME CBSE and 20 schools (57%) reported that they are not given instructions with regard to studying. Only four of the seven schools that instructed students to study for the exam provided study time. The amount of study time for students varied between schools. For example, one school reported that student study time was “3–5 days”, while another school stated that “sophomores have most Fridays free to study from March through May to study for Step 1 (and the CBSE)”. A third school stated that they provided 6 weeks to study with no instructions for students.

A total of 24 schools (67%) out of 36 respondents (one school did not respond) administered the exam one time, while six schools (17%) administered the exam two times. Four schools (11%) administered the NBME CBSE four times prior to students taking the USMLE Step 1. Two schools (6%) administered the NBME CBSE one time with additional exams being administered based on student performance. Percentages did not equal 100% due to rounding.

Timing of administrations of exams varied among schools. For example, of the schools that administered the NBME CBSE one time, four administered the exam once in year 2 semester 1, 18 administered the exam once in year 2 semester 2, one administered the exam “4–5 months prior to them taking Step 1”, and one allowed students to choose when to take the exam. Table 2 provides timing of the administration of the NBME CBSE for schools that administered the exam two and four times. Only two schools (6%) out of 36 respondents (one school did not respond) reported administering the NBME CBSE in the first semester of year 1, and four schools (11%) reported administering the NBME CBSE in the second semester of year 1 (at the end of the first year) with the two schools that administered the exam in the first semester of year 1 also administering the exam in the second semester of year 1. Seven schools (19%) administered the NBME CBSE during the third semester of school (first semester of year 2) with four of these schools administering the exam one time. Five schools (14%) offered the exam multiple times in the second semester of year 2 prior to administration of the USMLE Step 1 (Table 2).

Table 2 Semester of Administration of the NBME CBSE for schools that administer the exam two and four times Notes: n, number of respondents to question. #Reported only one time for NBME CBSE administration although stated administering NBME CBSE two times. Abbreviation: NBME, National Board of Medical Examiners®; CBSE, Comprehensive Basic Science Exam.

In all, 38 schools (83%) out of 46 respondents (seven schools did not respond) required students to take the USMLE Step 1 at the end of the second year, while other schools required students to take the USMLE Step 1 midway through the second year (n = 3, 7%), during the third year (n = 3, 7%), or end of the second year through middle of the third year (n = 2, 4%). Percentages do not add up to 100% due to rounding.

A total of 17 schools (50%) out of 34 respondents (three schools did not respond) had administered the NBME CBSE ≤5 years, while seven schools (21%) had administered the exam between 6 and 10 years. Eight schools (24%) had administered the exam >10 years. Figure 2 presents how many times a school administered the NBME CBSE grouped by the number of years a school had administered the NBME CBSE.

Figure 2 Number of times the NBME CBSE is administered to an entering class broken down by the number of years a school has been administering the NBME CBSE (n = 36). Note: “Other” = depends on student performance on first administration. Abbreviations: NBME, National Board of Medical Examiners®; CBSE, Comprehensive Basic Science Exam.

In all, 13 schools (59%) out of 22 respondents (four schools did not respond) that offered the NBME CBSE one time reported a graduating class size of ≤120 students, two schools (9%) reported a graduating class size of 121–160 students, five schools (23%) reported a graduating class size of 161–200 students, two schools (9%) reported a graduating class size of >200 students, and four of the 26 schools declined to answer. Figure 3 presents how many times a school administered the NBME CBSE compared to the number of students who graduated in each class.

Figure 3 Number of times the NBME CBSE is administered broken down by the number of students in each graduating class (n = 53). Note: “Other” = school responded “yes” to administering the NBME CBSE but did not provide the number of times the exam is administered or the number of students graduating in each class. Abbreviations: NBME, National Board of Medical Examiners®; CBSE, Comprehensive Basic Science Exam.

Schools that do not administer the NBME CBSE

Data presented cover only schools that did not administer the NBME CBSE. Of the 16 schools that did not administer the NBME CBSE, four schools (25%) did administer the exam in the past, while 10 schools (63%) did not administer the exam in the past. Two schools (13%) were unsure if the exam was administered in the past. Nine schools (75%) out of 12 respondents (four schools did not respond) that did not offer the NBME CBSE reported a graduating class size of ≥121 students (Figure 3).

Of those four schools that administered the exam in the past, one school was unsure of the reason for discontinuing the exam. A second school felt that the exam did not correlate with the curriculum and thought that the questions were of “poor quality”. A third school said that the NBME CBSE was “not part of promotion”, and therefore, some of the students “did not take it seriously”; however, they do require “low performers” to take the exam “as part of academic probation.” The fourth school that discontinued the NBME CBSE only gave the exam one time and did not use the data; however, they are reinstituting the exam “as a progress test and plan to give it three times to a class prior to Step 1” with the desire to “assess students readiness for Step 1 and also as a means of evaluating the effectiveness of our curriculum”.

Discussion

To our knowledge, this is the first report describing the scope with which US medical schools use the NBME CBSE. In addition, we provide timing and rationale for administration of the NBME CBSE by US medical schools. Of the schools responding to the survey, most schools administered the NBME CBSE, and they did so prior to the students taking the USMLE Step 1. In addition, most schools that administered the NBME CBSE only administered the NBME CBSE one time. Some schools administered the NBME CBSE for curriculum improvement or course assessments, but the majority of schools administered the NBME CBSE to identify students who were performing below expectations.

The finding that schools use the NBME CBSE to identify students performing below expectations is not surprising given the fact that the NBME CBSE has been reported to be a tool capable of identifying “students at risk for failing the USMLE Step 1” in established medical schools⁴ and predicting performance on the USMLE Step 1 in new medical schools.^5,8 It is interesting that eight schools out of the total 36 that administered the NBME CBSE did not use this NBME CBSE to identify students performing below expectations and provide intervention strategies. It would be of value to determine reasons why the NBME CBSE is not used to identify such students. It would also be of value to determine what, if any, metrics are used to identify students performing below expectations in the 16 schools that did not administer the NBME CBSE.

We also reported intervention methods, following identification of students performing below expectations, utilized by US medical schools (Table 1). In all, 26 schools (93%) that used the NBME CBSE to identify students performing below expectations provided strategies for studying and/or test taking as an intervention method. However, the effectiveness of this strategy may be limited as it has recently been reported that a test-taking strategy course improved test anxiety for students preparing to take the USMLE Step 1 but did not improve performance on the USMLE Step 1.¹¹ In addition, 11 schools (39%) that used the NBME CBSE to identify students performing below expectations provided an exam preparation course for students who had been identified as performing below expectations. The effectiveness of an exam preparation course intervention strategy also may be limited. Werner and Bull¹² reported that students taking a live commercial coaching course for USMLE Step 1 preparation do not show improved scores based on a predicted score using the NBME CBSE. Others also reported that coaching courses^13,14 or commercially available material¹⁴ do not improve performance on the USMLE Step 1. Although the literature regarding intervention strategies for improvement on USMLE Step 1 is variable, the number of days studied may be of benefit to students who are performing below expectations. Giordano et al⁶ reported that students who did not receive straight A’s during the first year of school improved scores with more days studied. Further studies on intervention strategies and timing of implementation for intervention strategies for students performing below expectations are needed as current methods used by many schools may not produce the desired outcome of improved USMLE Step 1 scores.

An interesting finding is that 13 schools (36%) administering the NBME CBSE used the data to make changes to the curriculum. Further research into how schools use these data for curriculum reform and the limitations with which these data can be used for curriculum reform is needed. Our findings show that of the 13 schools that used the NBME CBSE to make curricular changes, eight schools (62%) administered the exam one time, two schools (15%) administered the exam two times, and three schools (23%) administered the exam four times. In addition, only 2% of schools that administered the NBME CBSE instructed students to study for the exam, which could limit interpretation of results. The limited interpretation of the NBME CBSE scores is, in part, due to a high performance in content that has recently been taught.¹⁵ It is worth noting that the NBME CBSE is designed for students who have completed all basic science knowledge required for the USMLE Step 1, which would add to the limited utilization of the data from the NBME CBSE for schools that administer the exam prior to the end of basic science content.

Other reasons for the administration of the NBME CBSE are possible. For example, one new school reported using the NBME CBSE as a progress test in an effort to provide external validation of the internal assessments; however, now that validation of the internal measurements has been completed, students have recommended reducing the number of administrations from five to two.⁵ There is merit in using externally validated exams in all new and existing medical schools as it has been reported that internally developed examinations are of low quality and the quality of the exam can be improved by providing formal training to the writers of test items.¹⁶

Another reason for administration of the NBME CBSE might be to provide progress testing for schools. Progress testing has been reported to provide many benefits¹⁷ and has been used to monitor knowledge development in basic medical science courses at a medical school in Berlin.¹⁸ However, most US medical schools responding to the survey do not administer the NBME CBSE in a manner consistent with progress testing. We found that 8% of all schools responding to the survey (n = 53) administered the NBME CBSE four times and 11% of schools administered the NBME CBSE two times; however, the necessity of administering the NBME CBSE multiple times is unclear as the timing of exam administration for schools administering the NBME CBSE varies greatly (Table 2). All but one of the schools administering the NBME CBSE multiple times used the NBME CBSE to identify students performing below expectations. We are a young medical school that recently graduated our first class and we have decreased the number of NBME CBSE exams from four exams throughout the first two years to two exams (mid-M2 year and end-M2 year) for this entering class.

After 2 years, the best predictor of performance on the USMLE Step 1 was determined using our end-of-year-1 NBME CBSE, end-of-year-2 NBME CBSE, and our overall academic performance in year 1 and year 2 (R² = 0.81, standard error [SE] = 7.84).⁸ Another new medical school that offered the NBME CBSE five times before students took the USMLE Step 1 reported that the one score that best correlates with USMLE Step 1 performance is from the NBME CBSE administered at the end of year 2, following completion of basic science content; however, an exam score from a module within the first year had almost the same correlation with USMLE Step 1 scores as the end-of-year-2 NBME CBSE.⁵

A strength of this survey is that it is a cost-effective way to determine the scope and frequency with which US medical schools use the NBME CBSE within their curriculum. A limitation of this survey are the unknown characteristics of the schools that did not respond. A second limitation of the survey is that some respondents did not answer follow-up questions regarding a lead question. A third limitation is that some respondents either did not or were not a part of the institution when the NBME CBSE was first administered, limiting the knowledge for implementing the exam.

Conclusion

We report that in US medical schools, 70% of schools responding to the survey administered the NBME CBSE to medical students. Interestingly, out of the schools that do administer the NBME CBSE, only 78% percent used these data to identify students performing below expectations and provide targeted intervention strategies. In addition, for schools that administered the exam, most administered the exam one time during the second semester of year 2. The effectiveness of the intervention strategies provided for schools that administered the NBME CBSE as well as intervention strategies and timing of intervention for schools that do not administer the NBME CBSE should be investigated.

Acknowledgment

The authors would like to thank Michael Wiederman for his help with SurveyMonkey^® and expertise with the development and reporting of the survey.

Disclosure

The authors report no conflicts of interest in this work.

References

1.		USMLE. USMLE Step 1 Content Description and General Information. A Joint Program of the Federation of State Medical Boards of the United States, Inc., and the National Board of Medical Examiners®; 2016:1–7. Available from: http://www.usmle.org/pdfs/step-1/2016content_step1.pdf. Accessed February 9, 2017.
2.		National Resident Matching Program. National Resident Matching Program, Data Release and Research Committee: Results of the 2016 NRMP Program Director Survey. Washington, DC: National Resident Matching Program; 2016.
3.		Federation of State Medical Boards [webpage on the Internet]. State-specific requirements for initial medical licensure. Available from: http://www. fsmb.org/licensure/usmle-step-3/state_specific. Accessed April 12, 2017.
4.		Glew RH, Ripkey DR, Swanson DB. Relationship between students’ performances on the NBME comprehensive basic science examination and the USMLE step 1: a longitudinal investigation at one school. Acad Med. 1997;72(12):1097–1102.
5.		Johnson TR, Khalil MK, Peppler RD, Davey DD, Kibble JD. Use of the NBME comprehensive basic science examination as a progress test in the preclerkship curriculum of a new medical school. Adv Physiol Educ. 2014;38(4):315–320.
6.		Giordano C, Hutchinson D, Peppler R. A predictive model for USMLE step 1 scores. Cureus. 2016;8(9):e769.
7.		Holtman MC, Swanson DB, Ripkey DR, Case SM. Using basic science subject tests to identify students at risk for failing step 1. Acad Med. 2001;76(10 suppl):S48–S51.
8.		Wright WS, Williams SE, Hawkins HG, Otto RS. Predictive value of basic science content and NBME® comprehensive basic science exams in a new medical school. Association of American Medical Colleges Medical Education Meeting. Chicago, IL, USA. 2014.
9.		Conway D, Hanson J, Eddins-Folensbee F, Jackson J. Using admissions data and NBME exams to predict student success following curriculum renewal. 2016 SGEA: Rebooting Medical Education. Austin, TX, USA. 2016:S9.
10.		Association of American Medical Colleges [webpage on the Internet]. Number of medical schools using selected grading systems in pre-clerkship courses (excluding physical diagnosis/clinical skills). AAMC Curriculum Inventory, 2012-2016. Available from: https://www.aamc.org/initiatives/cir/406418/11.html. Accessed April 11, 2017.
11.		Green M, Angoff N, Encandela J. Test anxiety and United States Medical Licensing Examination Scores. Clin Teach. 2016;13(2):142–146.
12.		Werner LS, Bull BS. The effect of three commercial coaching courses on step one USMLE performance. Med Educ. 2003;37(6):527–531.
13.		McGaghie WC, Downing SM, Kubilius R. What is the impact of commercial test preparation courses on medical examination performance? Teach Learn Med. 2004;16(2):202–211.
14.		Thadani RA, Swanson DB, Galbraith RM. A preliminary analysis of different approaches to preparing for the USMLE step 1. Acad Med. 2000;75:S40–S42.
15.		Murre JM, Dros J. Replication and analysis of Ebbinghaus’ forgetting curve. PLoS One. 2015;10(7):e0120644.
16.		Jozefowicz RF, Koeppen BM, Case S, Galbraith R, Swanson D, Glew RH. The quality of in-house medical school examinations. Acad Med. 2002;77(2):156–161.
17.		Freeman A, Van Der Vleuten C, Nouns Z, Ricketts C. Progress testing internationally. Med Teach. 2010;32(6):451–455.
18.		Nouns Z, Schauber S, Witt C, Kingreen H, Schuttpelz-Brauns K. Development of knowledge in basic sciences: a comparison of two medical curricula. Med Educ. 2012;46(12):1206–1214.