Evaluation and enhancement of medical knowledge competency by monthly tests: a single institution experience

Introduction

The American Board of Internal Medicine (ABIM) since its establishment in 1936 has worked toward improvement of the US health care system. The ABIM certification implies that the physician has demonstrated adequate clinical knowledge, judgment, and skills to render appropriate patient care.¹ Board certification not only ensures better patient care^2,3 but is also preferred by the public from their health care providers.⁴

The performance at the ABIM certification examination is of particular importance to residency programs. It not only helps in maintenance of their American Council of Graduate Medical Education (ACGME) accreditation⁵ but also necessitates the programs to recruit residents of higher caliber. This enables programs to build upon the medical knowledge of their residents throughout training and allows for better performance on subsequent certification examinations.⁶

The ABIM pass rates for the certification examination has shown a declining trend in the 3-year rolling average recently, with pass rates being 91% in 2008 and 85% in 2012.¹ Hence, the residency programs want to identify residents who are at risk of being unsuccessful at the ABIM certification examination and need remediation. In-training examination (ITE) has been utilized as a predictor of performance of trainees in the ABIM examination in their first attempt.^7,8 This tool not only helps to provide feedback on resident performance but also helps to develop intervention strategies. However, ITE may not be the ideal modality of identification of at-risk individuals as it is held annually and represents snapshots of the performance as compared with a trend. The examination is too far apart to assess actual improvement and trends in performance. Hence, there is a need for a complementary method of assessment of a resident’s medical knowledge frequently over a period of time. This led us to develop a monthly test (MT) to enable the program to continually assess the performance of the residents throughout their residency.

Our curriculum is composed of two components. This test is based on the curriculum comprised of the ABIM core objectives known as the core curriculum. The other curriculum component is tailored appropriately to address areas of weakness identified by the residents’ performance in the previous year’s ITE; this is known as the dynamic curriculum. The dynamic curriculum is developed in a recurring cycle of 18 months.

The test was instituted in 2007 and is given in the third week of the month with a feedback session the following week.

The residents are tested once a month followed by a critique session and formative feedback of the multiple choice questions (Figure 1). The sources of the questions for the MT are standardized questions obtained by the program and are not a proprietary assessment tool to our program. The test session consists of 20 questions selected from an item pool of ~3,000 questions. The questions are selected by the program director and the chief resident based on the objective topics discussed in the didactic sessions of the previous month. Moreover, a semi-annual summative test is administered and consists of all objectives from the previous five MTs and consists of 40 questions per session, held in December and June of every year.

Figure 1 Flow diagram of the administration of the monthly tests. Abbreviations: ACGME, American Council of Graduate Medical Education; ABIM, American Board of Internal Medicine; ITE, in-training examination.

With the importance of performance in the internal medicine boards both for the residents and the program, there is need for identification of residents who may be at risk for failure at the ABIM examination. There is no study which has evaluated MTs as a tool to assess performance at the ABIM examinations.

Objective

The objectives of our study are 1) to determine the predictors of ABIM performance and 2) to assess whether the MTs administered by the program can be used along with the ITE examination to predict passing the ABIM examination.

Methods

The cohort consisted of 51 residents who successfully graduated from the internal medicine residency at the University of Toledo between 2010 and 2013. This consisted of consecutive batches after 2007, the year when the MTs were introduced. Data were collected from the records of the residency program and were de-identified. The study was approved by the local institutional review board.

Data were extracted on a predefined data collection form. Extracted data included the following: previous higher degree, United States Medical Licensing Examination (USMLE) step 1, 2, clinical knowledge, and step 3 scores. We did not have USMLE step 3 scores for all the residents as it is not a requirement to be reported to the program. In addition, information on ITE scores (percentage correct) in the postgraduate years (PGYs) 1, 2, and 3 and noon conference attendance were also collected.

The six ACGME core competencies were assessed for all the graduates by the faculty on a scale of 1–9 using New Innovations Evaluation System^®, and a cumulative score was developed for the entire training period. MTs were initiated at the University of Toledo from 2007. The yearly average of these test scores in the PGY1, PGY2, and PGY3 was calculated. A score based on the number of MTs missed was also calculated (monthly missed test score: number of tests missed/total number of tests). The score took into account that residents may miss MTs when they were on vacation or night float. The semi-annual summative examination scores were not included in the analysis because it may have been a confounder in the analysis as it included the questions from the earlier half of the year.

Analysis

Continuous variables were expressed as a mean ± standard deviation, and qualitative data were expressed as a percentage. Differences between groups were evaluated by unpaired t-tests or chi-square analysis, as appropriate. The effects of baseline parameters were assessed on ABIM pass rate. Univariate association between ITE and the MTs were assessed by Pearson’s correlation.

Multivariable logistic regression analysis was applied to identify the determinants of ABIM examination performance. In this model, the independent predictors used were ITE scores, yearly average of the MT scores, MT missed score, and the cumulative core competencies score. Forward stepwise selection was used to select the subset of predictors, which contribute most to the ABIM examination performance. Statistical significance was defined as P<0.05. Analysis was performed using SPSS Version 20.0 (IBM Corporation, Armonk, NY, USA).

Results

A total of 51 graduates appeared for the ABIM certification examination since the start of the MTs. Out of 51 graduates, 43 (84%) passed the ABIM examination on the first attempt and the remaining passed in their second attempt. All the residents had taken the annual ITE and the MT. There was no correlation found between features such as age, sex, time lag to start residency, and ABIM pass rate. The ITE and the MT scores improved for both the groups. The residents who passed the ABIM on the first attempt had higher evaluations by the faculty in terms of core competencies score as well as higher ITE scores (PGY1: 57.62±7.84 vs 49.85±6.12; PGY2: 65.65±8.21 vs 51.00±12.46; PGY3: 67.49±7.37 vs 58.32±10.17) and MT average scores (PGY1: 56.21±8.93 vs 47.80±4.55; PGY2: 64.16±9.26 vs 54.88±6.80; PGY3: 69.34±8.54 vs 65.63±5.28; Table 1). These residents also had higher USMLE scores as compared with the residents who failed to pass the ABIM examination in the first attempt.

Table 1 Faculty evaluation and resident’s performance stratified on the basis of American Board of Internal Medicine certification examination results Notes: P-value: *<0.05; **<0.01; ***<0.001. Data presented as mean ± standard deviation. Abbreviations: ABIM, American Board of Internal Medicine; ACGME, American Council of Graduate Medical Education; PC, patient care; MK, medical knowledge; PBLI, practice-based learning and improvement; ICS, inter-communication skills; Prof, professionalism; SBP, system-based practice; PGY1, postgraduate year 1; PGY2, postgraduate year 2; PGY3, postgraduate year 3; USMLE, United States Medical Licensing Examination; CK, clinical knowledge.

MTs had a good correlation with the ITE examination. The average of the first-year MTs showed a moderate correlation with second-year ITE (r=0.568, P<0.001; Table 2). Similarly, the second-year MTs were correlated with third year ITE (r=0.603, P<0.01; Table 2).

Table 2 Correlation between the monthly tests and the in-training examination scores Note: – Indicates the relevant correlation was not performed. Abbreviations: PGY1, postgraduate year 1; PGY2, postgraduate year 2; PGY3, postgraduate year 3.

The logistic regression model was developed to predict the result of the ABIM examination. The model included the subjective assessment of the ACGME core competencies by the faculty and the objective assessment of the medical knowledge by the ITE and the MT scores. The Hosmer–Lemeshow goodness of fit for the model was not significant (P>0.9) indicating that the model was well calibrated. The MT scores for the first year (odds ratio [OR] =1.302, CI =1.004–1.687, P=0.04) and second year (OR =1.125, CI =1.004–1.261, P=0.04) were independent predictors of ABIM performance along with the second-year ITE scores (OR =1.248, CI =1.096–1.420, P=0.001). Missed MT score was also independent predictor of ABIM certification (OR =1.237, CI =1.042–1.469, P=0.01; Table 3).

Table 3 Logistic regression analysis showing independent predictors of ABIM certification Abbreviations: ABIM, American Board of Internal Medicine; OR, odds ratio; PGY1, postgraduate year 1; PGY2, postgraduate year 2; ITE, in-training examination; MT, monthly test.

Discussion

The findings of our study indicate that MT can be used along with the ITE examination as a tool to assess medical knowledge and predictor of ABIM examination performance. The ACGME core competencies scores were not found to be predictive of the performance in the ABIM certification examination. Though studies have reported correlation between USMLE scores and performance during residency,⁹ we did not include USMLE scores as they reflect background medical knowledge rather than knowledge gained during residency. The number of tests missed had a negative association with ABIM pass rates with missing more tests indicative of being unsuccessful at the boards.

The results of the second-year ITE highly correlated with ABIM pass rates on the first attempt.¹⁰ Moreover, the ITE was primarily developed to assess the knowledge of residents in the middle of the residency. Hence, it was not developed to predict a resident’s performance based on the first-year ITE results. Moreover, the first-year ITE is taken too early in the residency to assess the knowledge or skills gained. By the time the second-year ITE results are known, little time may be left for intervention strategies to have the desired effect. MTs have an advantage as they help early identification of at-risk residents and consequently remedial measures could be instituted early during the course of their second year. These measures include, but are not limited to faculty supervised intensive board review sessions, dedicated topic reviews and attending board review courses. They also help to monitor the trend and effectiveness of the remedial measures instituted and to adapt them according to the needs of the focused group. It has been shown that the effects of repeated testing on learning are more than repeated studying.¹¹ It is because it involves retrieval of information from memory, which leads to long-term retention as compared with repeated studying.¹² The MT including the semi-annual cumulative testing along with the critique sessions are designed to help consolidate the information gained, which further strengthens the memory for long-term retrieval.¹²

The ITEs alone, which provide a snapshot at a time, may not be an adequate tool to assess the performance of residents who are at the borderline.⁸ The MT may be able to correctly identify these individuals, as a longitudinal trend would be better as compared with an annual exercise. These are the individuals who gain most from an intervention. The MT and subsequent critique sessions will allow the residents to evaluate their strengths and weaknesses and compare them to their peers on a monthly basis. This also gives the program an opportunity to assess its curriculum and didactic methods.

In our study, we did not find any correlation with the faculty assessment of the competencies and ABIM examination performance. MTs may also be a good tool to aid the faculty in assessment of the core competencies. This may play an important role in view of the new accreditation system implementation where emphasis is placed on innovation with tools to assess the six educational competencies, with clear performance-based endpoints and not numerical rating scales.¹³

Our study has several limitations. As the MTs were started relatively recently, our sample size was small, which may have affected our analysis. We evaluated the effect of this intervention in only one residency program. It may be helpful to validate this in other residency programs as well. We did not have the scores for ABIM examination available to us, and the results were taken as either pass or fail. It would be helpful to know the scores, especially those at the borderline, in order to further assess the remedial measures. We did not have complete data on the attendance at didactics. This would have shown some insight into the attitude and commitment of the residents toward education. We did not include USMLE scores, which may have given an insight on the medical knowledge of the residents. However, the scores would have been representative of the background medical knowledge instead of the residency training. Lastly, being a retrospective analysis, it has all the limitations associated with such an analysis.

Conclusion

The MT is a tool to enhance the formative feedback in medical knowledge and assess the performance of residents over a period of time. It may not only help in the evaluation of medical knowledge of the residents, but also supplement the ITE to help predict residents at risk of failing the ABIM. Looking at the usefulness of the monthly examination in conjunction with the annual ITE, it may be meaningful for other residency programs to replicate similar initiatives. To have a more definite conclusion, a comparative study to assess the impact of MTs and to study targeted optimal remediation regimens for residents with poor performance is needed.

Acknowledgments

We acknowledge the outstanding contributory efforts of Edith Reynolds, Tiffany Cantu, and Barbara Elliott in helping out with the data extraction from the Graduate Medical Education office. Part of this work has been accepted and presented as a poster finalist at the annual meeting of the American College of Physicians, ACP 2014, April 10–12, 2014, Orlando, FL, USA.

Disclosure

The authors report no conflicts of interest in this work.

---

References

1.	ABIM. First-Time Taker Pass Rates – Initial Certification; 2013. ABIM Available from: http://www.abim.org/about/examInfo/data-pass-rates.aspx. Accessed August 07, 2015.
2.	Curtis JP, Luebbert JJ, Wang Y, et al. Association of physician certification and outcomes among patients receiving an implantable cardioverter-defibrillator. JAMA. 2009;301:1661–1670.
3.	Turchin A, Shubina M, Chodos AH, Einbinder JS, Pendergrass ML. Effect of board certification on antihypertensive treatment intensification in patients with diabetes mellitus. Circulation. 2008;117:623–628.
4.	Freed GL, Dunham KM, Clark SJ, Davis MM; Research Advisory Committee of the American Board of Pediatrics. Perspectives and preferences among the general public regarding physician selection and board certification. J Pediatr. 2010;156(841–845):845e841.
5.	ACGME Program Requirements for Graduate Medical Education in Internal Medicine. Available from: http://www.acgme.org/acgmeweb/Portals/0/PFAssets/2013-PR-FAQ-PIF/140_internal_medicine_07012013.pdf.
6.	Norcini JJ, Grosso LJ, Shea JA, Webster GD. The relationship between features of residency training and ABIM certifying examination performance. J Gen Intern Med. 1987;2:330–336.
7.	Grossman RS, Fincher RM, Layne RD, Seelig CB, Berkowitz LR, Levine MA. Validity of the in-training examination for predicting American board of internal medicine certifying examination scores. J Gen Intern Med. 1992;7:63–67.
8.	Rollins LK, Martindale JR, Edmond M, Manser T, Scheld WM. Predicting pass rates on the American board of internal medicine certifying examination. J Gen Intern Med. 1998;13:414–416.
9.	Kanna B, Gu Y, Akhuetie J, Dimitrov V. Predicting performance using background characteristics of international medical graduates in an inner-city university-affiliated internal medicine residency training program. BMC Med Educ. 2009;9:42.
10.	Waxman H, Braunstein G, Dantzker D, et al. Performance on the internal medicine second-year residency in-training examination predicts the outcome of the ABIM certifying examination. J Gen Intern Med. 1994;9:692–694.
11.	Karpicke JD, Roediger HL 3rd. The critical importance of retrieval for learning. Science. 2008;319:966–968.
12.	Roediger HL, Karpicke JD. Test-enhanced learning: taking memory tests improves long-term retention. Psychol Sci. 2006;17:249–255.
13.	Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system – rationale and benefits. N Engl J Med. 2012;366(11):1051–1056.