Impact of inpatient caseload, emergency department duties, and online learning resource on General Medicine In-Training Examination scores in Japan
Received 30 January 2015
Accepted for publication 7 August 2015
Published 30 October 2015 Volume 2015:8 Pages 355—360
Checked for plagiarism Yes
Review by Single anonymous peer review
Peer reviewer comments 3
Editor who approved publication: Dr Scott Fraser
Kensuke Kinoshita,1 Yusuke Tsugawa,2 Taro Shimizu,3 Yusuke Tanoue,4 Ryota Konishi,5 Yuji Nishizaki,6 Toshiaki Shiojiri,7 Yasuharu Tokuda8
1Department of Medicine, Mito Kyodo General Hospital, University of Tsukuba, Mito City, Ibaraki, Japan; 2Department of Health Policy and Management, Harvard T.H. Chan School of Public Health, Boston, MA, USA; 3Tokyo Joto Hospital, Koto-ku, Tokyo, 4Good Medicine Japan, Miyagi, 5Department of General Internal Medicine, Kanto Rosai Hospital, Kawasaki, Kanagawa, 6Department of Cardiology, Juntendo University Graduate School of Medicine, Tokyo, 7Department of General Internal Medicine, Asahi General Hospital, Asahi, Chiba, 8Japan Community Healthcare Organization, Minato-ku, Tokyo, Japan
Background: Both clinical workload and access to learning resource are important components of educational environment and may have effects on clinical knowledge of residents.
Methods: We conducted a survey with a clinical knowledge evaluation involving postgraduate year (PGY)-1 and -2 resident physicians at teaching hospitals offering 2-year postgraduate training programs required for residents in Japan, using the General Medicine In-Training Examination (GM-ITE). An individual-level analysis was conducted to examine the impact of the number of assigned patients and emergency department (ED) duty on the residents' GM-ITE scores by fitting a multivariable generalized estimating equations. In hospital-level analysis, we evaluated the relationship between for the number of UpToDate reviews for each hospital and for the hospitals' mean GM-ITE score.
Results: A total of 431 PGY-1 and 618 PGY-2 residents participated. Residents with four or five times per month of the ED duties exhibited the highest mean scores compared to those with greater or fewer ED duties. Those with largest number of inpatients in charge exhibited the highest mean scores compared to the residents with fewer inpatients in charge. Hospitals with the greater UpToDate topic viewing showed significantly greater mean score.
Conclusion: Appropriate ED workload and inpatient caseload, as well as use of evidence-based electronic resources, were associated with greater clinical knowledge of residents.
Keywords: workload, online resource, knowledge, resident, postgraduate, Japan
Teaching hospitals are expected to provide high-quality educational environments to residents in order to enable them to attain a greater clinical knowledge and experiences. However, educational environments comprise many factors, including workloads, learning resources, clinical teachers, and hospital educational cultures. Among these factors, the programs of teaching hospitals can modify workloads and learning resources, whereas the behaviors of attending physicians and hospital educational cultures can be difficult to modify. Therefore, it is in our interest to evaluate how workloads and learning resources would affect the learning experiences of residents.
Our previous study showed that better, overall educational environment at teaching hospitals was associated with greater medical knowledge, but the study did not provide effects of each component of the educational environments on clinical performance of residents.1 Among components of educational environments, inpatient workload may be a factor for better clinical performance, since clinical experience is considered to enhance learning in clinical medicine. Previous studies suggested that the cumulative number of inpatients in charge was associated with greater score of In-Training Examination (ITE) in internal medicine residents,2 and the cumulative number of general pediatric admissions was correlated with an increase in ITE score in pediatric residents.3 However, these studies evaluated the cumulative number of inpatients as a workload for residents. A few studies have investigated the association between the average number of inpatients in charge at a time as a real-time caseload for residents and clinical knowledge in residents. This number may better capture the actual workload on daily basis. In addition, a few studies have investigated the association between the frequency of emergency department (ED) duties and clinical knowledge in residents. This number could reflect the actual exposure to emergency patients and could also be used as daily workload.
In this context, we aimed to investigate the association of clinical performance and objective workload using the average number of inpatients caseload in charge at a time and ED duties frequency.
Regarding learning resources, previous studies indicated higher scores of residents’ clinical performance for the amount of self-learning time in anesthesiologist residents ITE4 and in internal medicine residents5,6 and self-learning by online resources7 in the Internal Medicine ITE (IM-ITE) and the amount of self-learning time. We also evaluated the online learning resources as a factor for greater clinical performance in residents since these resources are popular in modern clinical environments, in conjunction with the workloads.
Participants and data collection
We conducted a multicenter study involving postgraduate year (PGY)-1 and -2 resident physicians at 114 teaching hospitals in Japan. The participants were trainees of 2-year postgraduate rotation training programs, including internal medicine, emergency medicine, community medicine, and others (PGY-1 and -2), required for all residents regardless of their specialty, before getting into the specialty training programs (PGY-3 or later). The invitation to participate in the study was announced on the Japan Organization of Advancing Medical Education Program (JAMEP: Nonprofit Organization, Tokyo, Japan) Web site, monthly medical journal advertisement (Resident Note; Yodosha Co, Tokyo, Japan), and was sent to program directors of all 1,015 teaching hospitals in Japan certified by Ministry of Health, Labor, and Welfare, including 114 university hospitals and 901 community teaching hospitals.
The residency program directors of each hospital were required to assemble residents in a room at each hospital at a scheduled time and provide the General Medicine ITE (GM-ITE) for their residents. Each program director then collected the completed examination answer sheets and sent these back to us in an envelope we had provided. As the academic calendar in Japan starts on April 1 and ends on March 31 of the following year, the GM-ITE was conducted in March of the 2012 academic year. Immediately after the time of the GM-ITE, we provided a self-reporting questionnaire sheet about residents training loads (ED duties per month and average number of inpatients in charge at a time) and use of evidence-based electronic resources (DynaMed; EBSCO, MA, USA, and UpToDate; UpToDate Inc., MA, USA), obtaining informed consents from each participant. We received data about UpToDate usage from physicians in the hospitals who participated in our study. The UpToDate usage data were directly provided by UpToDate Inc and include the number of topic viewing. This study was approved by the institutional review board of the Mito Kyodo General Hospital, Mito City, Ibaraki, Japan.
The primary exposure variables of our study were ED duties per month, average number of inpatients in charge at a time, and use of evidence-based electronic resources. Japanese PGY-1 and -2 residents are usually required to work in inpatient ward and in ED. Thus, we evaluated the association between the test score and workload in these two settings. The average count of ED duties per month was deemed as overall monthly ED workload. The average number of inpatients in charge at a time was used as overall daily inpatient caseload. The use of evidence-based electronic resources was indicated by regular online use of one of the following two most popular resources (that indeed account for vast majority of the electronic resources the participants used in their hospitals): DynaMed (EBSCO) or UpToDate (UpToDate Inc.) since these two resources were considered as the most popular online systems available in Japan. Other variables we collected and adjusted in the regression models include PGY.
We developed and employed the GM-ITE using the similar methodology of developing the IM-ITE in the USA.8 The purpose of the IM-ITE is to provide residents and program directors with an objective, reliable, and valid assessment of each resident’s clinical knowledge in multiple-choice examination, and the mean scores of each program compared with that of its peers.9,10 The aim of the GM-ITE is to assess clinical knowledge of PGY-1 and -2 residents. The test scores and the grade relative to the overall distributions of the test scores were provided for program directors of participating teaching hospitals and for each resident.
The GM-ITE examination included 100 questions testing a wide range of clinical knowledge, from clinical skills and practical medical knowledge to psychosocial care of patients, and was designed and written by a committee of experienced attending physicians organized by JAMEP. The examination gave a maximum score of 100 and a minimum score of 0; higher scores indicate a better performance of knowledge base of general medicine. Before conducting it, the examination question review was organized, and the content validity was confirmed by peer review of each member of the committee. A 3-hour fixed time limit was set for taking the examination. The GM-ITE was first implemented in 2011. Our study used data of the test that was performed in 2012, and an additional questionnaire was collected for the purpose of our study.
We investigated the association between the residents training loads, personal use of evidence-based electronic resources, and GM-ITE score by constructing a multivariable generalized estimating equations in order to account for the clustering of residents within each hospital. An exchangeable working correlation with identity link function and normal distribution was used for the analyses. Residents training loads in our study included ED duties per month and average number of inpatients in charge at a time. Evidence resource use was defined as personal use of online evidence summary (DynaMed and UpToDate). Continuous variables, such as the number of ED duties per month and average number of inpatients in charge at a time, were categorized into quintiles in order to allow potential nonlinear relationship with the outcome variable. The number of ED duties and inpatients in charge at a time were compared between residents who used evidence-based electronic resources and those who did not.
We also conducted a hospital-level analysis examining the association between the uses of evidence-based electronic resources and GM-ITE score aggregated at hospital level. We analyzed the association between mean hospital score of the GM-ITE and mean number of topic viewing per physician for UpToDate in each hospital for the preceding academic year with multivariable linear regression model. All statistical analysis was performed using IBM SPSS Statistics 21.0 (IBM Corporation, Armonk, NY, USA), and two-tailed P-value of <0.05 was considered as statistically significant.
A total of 431 PGY-1 and 618 PGY-2 resident physicians in-training (290 women, 28%) underwent the GM-ITE and completed the survey (Table 1). As for the PGY-2 physicians, 159 (26%) were women and 459 (74%) were men. As for PGY-1 physicians, 131 (30%) were women and 300 (70%) were men. Among all participated residents, the mean score of the examination was 58.1 (standard deviation, 7.5). There were 114 teaching hospitals. Nine hospitals (7.9%) were university hospitals, while 105 hospitals (95.6%) were community hospitals. Evidence-based electronic resources were used by 790 residents (75.3%). There were 581 residents (55.4%) who had four to five times or more ED duties in an average month. There were 140 residents (13.3%) who cared 12 or more inpatients at any time in average ward rotation.
Table 1 Characteristics of residents
After adjusted for PGY, multivariable generalized estimating equation analysis of total scores for all residents is shown in Table 2. Significant variables associated with the greater score included ED duties four to five times per month with mean score difference of 4.5 (95% CI, 2.7–6.3), number of inpatients in charge at 12 or greater times with mean score difference of 2.5 (95% confidence interval [CI], 0.96–4.0). Thus, residents with ED duties four to five times per month showed the greatest mean scores better than those with 0–3 or those with 6 or greater times of ED duties. Residents who routinely used evidence-based electronic resources showed 2 higher score (95% CI, 0.9–3.0) compared to those who did not use.
The number of ED duties per month was not different between residents who used evidence-based electronic resources and those who did not use (4.5 vs 4.3, P=0.3). The mean number of inpatients in charge at a time was significantly greater among residents who used evidence-based electronic resources compared to those who did not use (8.8 vs 8.1, P=0.04). A multivariable linear regression model evaluating the association between mean scores and UpToDate review counts analyzed at hospital level revealed that UpToDate greater usage per physician was significantly associated with the greater score with mean score difference of 0.061 (95% CI, 0.007–0.114).
We found that residents with ED duties four or five times per month showed significantly greater scores for the GM-ITE compared to those with 0 or once per month of the ED duties. Our results also showed that residents in charge of 12 or greater number (the group with the greatest number of caseloads) of inpatients at a time showed significantly greater GM-ITE scores compared to those with 0–5 inpatients in charge at a time.
A greater usage of evidence-based electronic resources at hospital level was significantly associated with the mean higher scores at a given hospital. Our results imply that greater workloads are needed for better performance of residents and that workload balance should be adjusted for achieving better performance as well as preventing residents’ burnout and adverse effects to patient safety.
Recently there has been a trend for limiting workloads for residents since there might have been association between greater workload and higher risk at burnout of residents. Previous studies on residents’ workloads showed that workload was related to burnout and medical error.11,12 A study indicated that work hour limitation improved outcomes of patients in the teaching service.13 However, some studies, especially on surgical residents, suggested the opposite consequence, indicating that work hour limitation might lead to negative effects,14 null effects15 to patients’ outcome, and negative effects to the achievement of educational outcomes.16 Furthermore, programs with work hour limitation might not be attractive for residents with internal medicine.17
Many graduating students in Japan choose busy residency programs since they consider these are better educational environments for them. Our results also showed that a reversed U-shaped relationship between ED duties and performance of the GM-ITE scores. Thus, excessive workload in ED duties may have negative effects to residents’ performance. The appropriate workload balance should be aimed regarding designs for residency program for ED duties.
Residents who used evidence-based electronic resources exhibited greater scores of the GM-ITE in our study. Previous studies indicated higher scores of residents’ clinical performance for the amount of self-learning time.4–7 Ecological studies on teaching hospitals also suggested that hospitals with evidence-based electronic resources had shorter mean days of their inpatients’ stay, lower mortality, and better quality of care.18 By the use of UpToDate immediately after inpatient rounds, 75% of clinical questions elicited by the rounds received satisfactory solutions and 37% of online reviewing led to change in clinical management.19
Although little time would be considered to be available for self-studying during busy residency programs, our results indicated that residents who used evidence-based electronic resources had improved clinical knowledge and were responsible for the significantly greater number of inpatients. Availability of evidence-based electronic resources at the point of care may be useful for supporting busy residents as a modern version of medical textbook and it could have better implication for the quality of patient care. Since our study provided association between the greater test score and appropriate workloads and use of evidence-based electronic resources, program directors and residents may be advised to consider setting appropriate workloads and using these resources.
Our study has several limitations. First, given that, only 1,049 of the 15,800 residents participated throughout Japan in this study, and they were not a random sample of the underlying population of all residents in Japan, our findings may not be generalizable to all residents in Japan. Program directors were responsible for the decision to participate in this examination and thus there might have been a bias for higher proportions for the participation among teaching hospitals with program directors with higher motivation and subsequently better programs. However, no residents who received the GM-ITE denied the participation for this survey and there was little space for bias toward the participation of better-performing residents. Second, with regard to the use of evidence-based electronic resources, better performers who used them efficiently might have had greater chance to provide care to the higher number of inpatients. Third, the emergency room duty per month and the number of inpatients in charge could be influenced by recall bias or cognitive bias since it is a questionnaire study. We use the term “on average” in the questionnaire about these items over the 1-year period before the test date. Fourth, we did not adjust potential confounders such as age and sex of individual residents in our model. However, these variables are usually not considered as significant variables associated with educational environments of teaching hospitals.
We found that appropriate level of workload, including ED duties and the inpatient number in charge, might have beneficial impact on the achievement of better performance of residents. In addition, the evidence-based electronic resources may be useful supporting tools for busy residents to get solutions for clinical questions raised by rounds. Our study may provide a benchmark for setting well-balanced workload for residents. Further study is needed for the interrelationships between workload, quality of life, and quality of patient care among residents.
The authors thank all executive people for their excellent assistance. The JAMEP funded the study grant and conducted the examination for resident physicians as explained in the text and data collection but had no role in the study design, data interpretation, or preparation of the article. Dr Tsugawa was supported by Honjo International Scholarship Foundation.
K Kinoshita is the guarantor for the integrity of the data and the accuracy of the data analysis. K Kinoshita and Y Tokuda conceived the research idea. K Kinoshita and Y Tsugawa designed the study, conducted statistical analysis, and drafted the manuscript. All authors contributed to the interpretation of data, critically revised the paper for important intellectual content, and agreed to be accountable for all aspects of the work.
Y Tsugawa and Y Tokuda received honorarium from the JAMEP for preparing study questions. The other authors report no other conflicts of interest in this work.
Shimizu T, Tsugawa Y, Tanoue Y, et al. The hospital educational environment and performance of residents in the general medicine in-training examination: a multicenter study in Japan. Int J Gen Med. 2013;6:637–640.
McCoy CP, Stenerson MB, Halvorsen AJ, Homme JH, McDonald FS. Association of volume of patient encounters with residents’ in-training examination performance. J Gen Intern Med. 2013;28(8):1035–1041.
Chase LH, Highbaugh-Battle AP, Buchter S. Residency factors that influence pediatric in-training examination score improvement. Hosp Pediatr. 2012;2(4):210–214.
Philip J, Whitten CW, Johnston WE. Independent study and performance on the anesthesiology in-training examination. J Clin Anesth. 2006;18(6):471–473.
Godellas CV, Hauge LS, Huang R. Factors affecting improvement on the American Board of Surgery In-Training Exam (ABSITE). J Surg Res. 2000;91(1):1–4.
Godellas CV, Huang R. Factors affecting performance on the American Board of Surgery in-training examination. Am J Surg. 2001;181(4):294–296.
McDonald FS, Zeger SL, Kolars JC. Factors associated with medical knowledge acquisition during internal medicine residency. J Gen Intern Med. 2007;22(7):962–968.
Garibaldi RA, Subhiyah R, Moore ME, Waxman H. The in-training examination in internal medicine: an analysis of resident performance over time. Ann Intern Med. 2002;137(6):505–510.
Kanna B, Gu Y, Akhuetie J, Dimitrov V. Predicting performance using background characteristics of international medical graduates in an inner-city university-affiliated internal medicine residency training program. BMC Med Educ. 2009;9:42.
Perez JA, Greer S. Correlation of United States medical licensing examination and internal medicine in-training examination performance. Adv Health Sci Educ Theory Pract. 2009;14(5):753–758.
Block L, Wu AW, Feldman L, Yeh H-C, Desai SV. Residency schedule, burnout and patient care among first-year residents. Postgrad Med J. 2013;89(1055):495–500.
Blanchard P, Truchot D, Albiges-Sauvin L, et al. Prevalence and causes of burnout amongst oncology residents: a comprehensive nationwide cross-sectional study. Eur J Cancer. 2010;46(15):2708–2715.
Horwitz LI, Kosiborod M, Lin Z, Krumholz HM. Changes in outcomes for internal medicine inpatients after work-hour regulations. Ann Intern Med. 2007;147(2):97–103.
Lee DY, Myers EA, Rehmani SS, et al. Surgical residents’ perception of the 16-hour work day restriction: concern for negative impact on resident education and patient care. J Am Coll Surg. 2012;215(6):868–877.
Curet MJ. Resident work hour restrictions: where are we now? J Am Coll Surg. 2008;207(5):767–776.
Businger A, Guller U, Oertli D. Effect of the 50-hour workweek limitation on training of surgical residents in Switzerland. Arch Surg. 2010;145(6):558–563.
Gopal RK, Carreira F, Baker WA, et al. Internal medicine residents reject “longer and gentler” training. J Gen Intern Med. 2007;22(1):102–106.
Isaac T, Zheng J, Jha A. Use of UpToDate and outcomes in US hospitals. J Hosp Med. 2012;7(2):85–90.
Phua J, See KC, Khalizah HJ, Low SP, Lim TK. Utility of the electronic information resource UpToDate for clinical decision-making at bedside rounds. Singapore Med J. 2012;53(2):116–120.
This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.Download Article [PDF]