The Effect of Simulation Wars on EM Residents’ Global Rating Scales and In-Training Examination Scores

Introduction

The Accreditation Council for Graduate Medical Education (ACGME) restricted the duty hours for residents training in 2004. With less time to learn medicine, residents may not develop the clinical reasoning skills needed. Simulation can be a practical solution with intensified hands-on practice outside the hospital. Simulation in medical training has shown to improve clinical skills of residents’ especially clinical reasoning, leading to improved patient care practices and better performance.^1–5 Clinical reasoning, a critical skill for Emergency Medicine residents, includes analytical reasoning using deductive cognitive strategies where information is gathered and applied.^3,4 Clinical reasoning simulation can be a significant educational method for training Emergency Medicine (EM) residents since this is a field where clinical reasoning is most practiced.^6,7 EM physicians use clinical reasoning to narrow their differential diagnosis reinforced by laboratory and radiological tests in order to make a correct diagnosis. There already exists a clinical reasoning exercise, Simulation Wars (SimWars), which is a simulation-based competition between teams for treating critically ill patients. These exercises are prevalent at Emergency Medicine national conferences where emergency medicine residents compete in teams.⁸ By adding a competitive game like segment, SimWars augments learning by promoting interaction between residents as a team and stimulates problem-solving.^9,10 SimWars exercises for residents and medical students have been shown to improve clinical reasoning skills.^5,11,12

There are no studies evaluating the effects of SimWars on the traditional ways residents are evaluated, with Global Rating Scale (GRS) scores and In-Training Examination (ITE) scores. The GRS scores are mostly used in Graduate Medical Education (GME) by the faculty to gauge the residents on variables including medical knowledge and clinical reasoning while working in the ED. The six ACGME competencies for which residents are graded per the GRS scores include; patient care, medical knowledge, practice-based performance, system-based practices, communication and interpersonal skills and professionalism.^12,13 GRS scores by faculty have been observed as a better metric to ascertain residents’ ability to apply knowledge than other methods. However, literature exists showing GRS scores have limited reliability and rater bias.¹⁴ Cognitive knowledge is measured mainly by test scores such as ITE scores; however, ITE scores also measure clinical reasoning skills. The ITE for EM residents explicitly includes a clinical reasoning component which is one-third of the entire test. This component evaluates the residents on tasks that must be performed to provide appropriate emergency medical care.¹⁵ Our study included both the GRS scores and ITE scores since these are the major ways residents are evaluated during training. Our study was focused on four major objectives. The first objective of our study was to evaluate the effect of SimWars on EM residents’ ITE and GRS scores. The second objective was to compare ITE scores over the 3 years of residency of the group that participated in SimWars and those that did not participate. The third objective was to observe if there was a difference in the intervention groups’ ITE subcategory topic score depending on their participation in a SimWars of a matched subcategory topic. Our fourth objective was to evaluate whether participation in a SimWars session, twice or more in a certain subcategory, would affect ITE score in that same subcategory.

Materials and Methods

Study Design

This was a retrospective quasi-experimental study. The main comparison was between historical controls, which included residents who did not participate in the SimWars, with the intervention group, which included residents who participated in the SimWars. The intervention participants were explicitly assigned to a team and were not just in the audience observing the competition. SimWars occurred every 2 months during the study period. The effect of participants’ involvement in SimWars and their ITE and GRS scores were observed during the course of study.

Study Setting and Population

The subjects of this study were comprised of Emergency Medicine (EM) residents at a metropolitan location. The EM residency was a 3-year residency with 15–19 residents per year. The intervention group was composed of EM residents involved in SimWars (cases/intervention) during the years 2012–2015. They were required to attend SimWars unless they were on an outside rotation or on vacation. After 2015, the SimWars was not included in the EM resident curriculum, as decided by the residency education committee, not based on resident evaluation of Sim Wars or GRS or ITE scores. The historical control group was comprised of EM residents from years 2009–2011, prior to the initiation of SimWars. The control group attended only one simulation session at the beginning of their first year and none in subsequent years. This control group session was not in the format of SimWars. The EM residency director made the ITE and GRS scores available to the primary investigator of this study.

SimWars Protocol

The Institutional Review Board (IRB) at The University of Texas Health Science Center at Houston/McGovern Medical School approved this study. Medical School. Demographic, ITE and GRS scores were obtained from the EM Residency Director. Residents’ scores and demographic information was not linked to their names.

SimWars was organized as a competition between two resident teams. The residents were divided into groups of 4, each with an assigned group leader. It was ensured that the residents in a particular group belonged to different levels of training. Different group leaders were assigned for different scenarios, but always stayed in the same group. Two groups competed at a time, while the remaining residents observed the session in the audience. There were at least 20 residents in the audience at each session. Each session lasted for approximately 2 hrs and included three to four cases. An EM attending would first present the patient’s vital signs and physical exam to only one team at a time while the other team was required to sit outside the room. Both teams did not have any prior details about the cases. The attending would then inquire, “What would you do next for this patient?” The residents’ role played through the case and announced critical actions they would perform. Simulator mannequins were not used for these sessions, and the scenarios were based on relevant ED patient presentations, such as an acute myocardial infarction (MI), stroke, or aortic dissection. Scenarios related to Signs and Symptoms (SS), Abdominal/Gastrointestinal (Abgas), Cardiology (Cardio), Environmental (Enviro), Head Eyes Ears Nose and Throat (HEENT), Systemic Infections (SI), Musculoskeletal (Musc), Obstetrics/Gynecology (ObGyn), Psychology (Psycho), Thoracic, Toxicology (Toxic), Trauma and Nervous System (Nerv) were also presented during the study period. The scenarios were reviewed by several EM attendings for their validity while some scenarios were selected from the Council of Emergency Medical Residency Programs (CORD) board review questions. After the session, all critical actions that should have been performed by a resident for each scenario were identified and reviewed. For example, critical actions for a patient presenting with chest pain typical of an acute MI would be to place the patient on oxygen, administer nitroglycerine, morphine, and perform an electrocardiogram (ECG). The attending then scored critical actions performed by each participating group, with the team which scored higher declared the winner.

Data Analysis

For comparison between the historical control group and the intervention group (cases), the continuous variables, ITE and GRS scores, were evaluated using the Independent t-tests. Continuous variables were analyzed with the t-test if scores were normally distributed. Parametric tests such as the Mann–Whitney U-test were conducted if scores were not distributed normally. Nominal variables were analyzed by using the Chi2 test or Fisher’s exact test. Correlation coefficients were produced to analyze if attending two or more sessions of a certain subcategory had any effect on that matched ITE subcategory score.

Results

The control and intervention group (cases) was comparable in terms of gender, race, degree, previous employment in a health field, and type of medical school United States (US) versus (non-US) attended. The first objective was to evaluate the effect of SimWars on EM residents’ ITE and GRS scores (Table 1). There was a significant difference in GRS scores except for ACGME competencies of communication (p<0.001) and professionalism (p<0.001) which were significantly better for the control group. The control group had a higher number of above expectations (AE) scores than the intervention group. For all other competencies, there was no significant difference in GRS scores. This was also true for GRS scores at each level of training. Overall, no significant variation was observed in ITE scores of both groups.

Table 1 ITE and GRS Scores, Between Historical Control Group and Simulation Games Intervention Group (Cases)

The second objective was to evaluate if the ITE scores changed over the 3 years of residency in the group that participated and in the group that did not participate in SimWars. There were no significant differences in ITE scores between the control group and intervention group. The groups, when reviewed by the years in residency showed a significant improvement in ITE scores for those in the control and intervention group from year 1 to year 2 (p≤0.001 for both). For the control group, ITE scores did not have a significant improvement (p=0.61) from year 2 to year 3. There was a trend towards improvement for the intervention group from year 2 to year 3 (p=0.31) (Figure 1).

Figure 1 Comparison of ITE Scores from Year 1, Year 2, Year 3 in Residency.

The third objective was to see if there was a difference in the intervention groups’ ITE subcategory score according to whether they participated in one or more Simulation Wars that matched that subcategory. This analysis was relevant only to those in the intervention group which participated in the SimWars. There was a significant improvement (p≤0.02) in the ITE score in the subcategory of Thoracic Disorders in first-year residents (n=34) who participated in the Thoracic Simulation Wars. There was a significant improvement (p=0.01) in the Abdominal/Gastrointestinal ITE scores for second-year residents (n=59) who attended the Abdominal/Gastrointestinal ITE Simulation Wars. Similarly, there was a significant improvement (p<0.01) in the ITE score in the subcategory of Trauma Disorders for third-year residents (n=14) who participated in the Trauma Simulation Wars. Next, paired comparisons of subcategory score assessments at baseline (Year 1) and at Year 2 and 3 of the intervention in participants who attended one or more simulation wars on the same topic were done. There was a significance (p<0.05) difference in the ITE scores in the subcategory of Trauma Disorders (p=0.003) and ObGyn (p=0.01) in those who participated in a matched topic of Simulation Wars for year 1 to year 2. For years 2 to 3, Thoracic (p=0.010) and Trauma (p=0.008) were the only subcategories of ITE scores where significant improvement was observed if residents attended a SimWars on Thoracic or Trauma, respectively. For subcategory score assessments at baseline (Year 1) and at Year 3 in the intervention (Cases) group who attended sessions in a matched topic, Trauma (p=0.006) was the only subcategory of ITE scores where there was a significant improvement. All other types of Simulation Wars did not show a significant difference in subcategory ITE scores.

Our fourth objective was to evaluate whether there was an effect on subcategory ITE score if a resident attended two or more sessions on that same subcategory. There were only two subcategories where a significant increase in ITE scores in that subcategory was observed which included Abdominal/Gastrointestinal (correlation significance =0.03) and Trauma (correlation significance =0.03).

Discussion

This is the first study that has examined SimWars in relation to traditional evaluation metrics for residents such as ITE scores and GRS scores and included a control group to examine an educational clinical reasoning skills exercise and its true effect. This was also one of two studies that examined a clinical reasoning exercise longitudinally over time in the same set of EM residents. All other studies examining a clinical reasoning skills exercise did not include a control; therefore, our study is unique and adds to the literature, looking for a true difference when this type of exercise is included. As far as our first objective is concerned, there was no difference between intervention and control group for overall ITE scores. Some studies in the literature have shown improvement in ITE scores with clinical reasoning exercises such as the surgical pattern recognition examination (PAT) administered to surgical and medical students at one point in time during one academic year.¹⁶ In this study, researchers found that scores on the PAT were significantly correlated to the Surgery ITE, r=0.67.¹⁶ It can be argued that our results were different because unlike our study, which evaluated analytic reasoning skills, the PAT evaluated clinical reasoning’s nonanalytic skills. Our study focused on analytic clinical reasoning skills which develop over time with clinical experience and include higher intelligence.¹⁷ Another reason for differing results could be because our residents’ SimWar exercises occurred every 2 months as compared to a single exercise in PAT study.

In a study by Humbert et al, the script concordance test (SCT) was used to assess clinical reasoning in an emergency medicine clerkship and residency.¹⁸ They also observed a significant correlation between SCT scores and ITE scores (r=0.69, p<0.001). However, Humbert’s study was conducted over one academic year and the SCT was conducted once during that year. Although our SimWars format included a lot of the components of the validated and reliable Script Concordance Test, it was conducted over a longer duration and more frequent than Humbert’s study, which can be a reason for differing results. It is known that a single exposure to educational material is usually inadequate for good long-term retention. Hundreds of studies in cognitive and educational psychology have demonstrated that spacing out repeated encounters with the material over time produces superior long-term learning.^19,20 Spaced learning enhances diverse forms of learning, including memory, problem-solving, and generalization to new situations.²¹ SimWars longitudinal administration applied the concept of spaced learning by conducting sessions every 2 months over the span of 3 years of residency. Consequently, our results are expected to present a better reflection of a better administered clinical reasoning exercise affecting ITE scores.

A study by Fernandez et al, which examined surgical residents undergoing simulation as part of a boot camp at the beginning of their first year of residency, showed a significant correlation with ITE scores.²² Although they concentrated on analytic processes, their clinical reasoning exercise session was conducted only once over an academic year in contrast to our study. Additionally, our study included a larger sample size, longer period of study, and a larger selection of scenarios specific to EM.

In our study, it was observed that there was no significant statistical difference in the majority of GRS scores for intervention and control group. Surprisingly, the control group had higher scores than the intervention group in Communication and Professionalism. This could have been due to inherent qualities of the residents in this group such as professional or personal maturity. Another possible explanation could be the “halo effect.” The halo effect is a cognitive bias in which an evaluator’s overall impression of a person is influenced by the evaluator’s feelings and thoughts about the person’s character.²³ One negative experience of the faculty member with a resident may bias the opinion to evaluate the resident negatively on all other GRS scores in the future. Therefore, it can be reasoned that SimWars did not contribute to the improvement in GRS scores and lower GRS scores observed in the intervention group may have been due to a halo effect. Unfortunately, there was no significant difference observed in the other competencies, such as Medical Knowledge or Patient Care. Patient Care and Medical Knowledge are the competencies which would be considered to improve after the intervention of SimWars since they include clinical reasoning. Medical Knowledge evaluates the resident on how he/she demonstrates appropriate medical knowledge in care of the ED patients. Patient Care is evaluated on how the resident manages the patient with all available data and resultantly narrows and prioritizes the list of differential diagnoses to determine appropriate management. It was surprising to observe no significant improvement in the competencies of Medical Knowledge and Patient Care with SimWars.

For our second objective, there was a statistically noteworthy improvement in overall ITE scores from the first year to the second year. In the United States (US), the ITE scores for EM residents normally progress from the first year to the second year and from the second to the third year.²⁴ However, no substantial improvement was observed from the second year to the third year for both controls and intervention groups, yet there was a trend for the intervention group. Similar results were observed in a study by Clarke et al on EM residents. They evaluated the crisis resource management ability of the residents with a simulation-based longitudinal study.²² Clarke et al attributed this lack of improvement from the second year to the third year of residency due to a ceiling effect that obscures true differences in ability in the advanced stages of residency.²⁵ It can be argued that our results also were due to the ceiling effect, meaning the optimal potential effect of a SimWars was achieved at the second year of residency thereby resulting in no observable improvement in the third year of residency. However, if the ceiling effect inhibits further improvement in scores beyond the second year, then perhaps concentrating on SimWars should occur only during the first 2 years of residency.

For our third objective, comparing the subcategory ITE scores of the control group to intervention group members who participated in a SimWar session that matched the subcategory topic did not show a significant improvement in the majority of subcategories. There was an improvement in ITE scores for Thoracic Disorders for first-year residents who participated in the Thoracic SimWars. Improvement was also observed in Abdominal/Gastrointestinal ITE scores for second-year residents who participated in the SimWars on Abdominal/Gastrointestinal. Similarly, third-year residents who participated in the SimWars on Trauma showed improvement in the ITE score in the subcategory of Trauma disorders. The reason for improvement in different topics in ITE scores according to year in residency could not be explained. Significant improvement was also observed from first to second, second to third year and first year to third year for the subcategory ITE scores for OBGyn, Trauma, and Thoracic by year and attendance to these specific SimWars. In summary, the ITE scores of the subcategories of Abdominal/Gastrointestinal, Trauma, Thoracic, and ObGyn were positively affected if a resident attended a SimWars session in the relevant subcategory. We initially hypothesized that certain subcategories where SimWars participation positively affected ITE scores, were also due to a larger number of residents attending those particular sessions. However, review of the attendance in Thoracic, Abdominal and GI, Trauma, and ObGyn SimWars where ITE scores improved showed that there was no consistent increase in attendance for these sessions. There were more residents attending the Abdominal and GI (147) and ObGyn (111) SimWars but less in subcategories of Thoracic (82) and Trauma (98). For some reason, the Thoracic and Trauma SimWars, even though less frequently attended, may have made a more memorable impact on the resident than the other subcategory SimWar sessions.

Our fourth objective was to see if there was a change in the ITE scores in a subcategory if a resident attended two or more SimWar sessions in that matched subcategory. It was observed that there were only two categories of SimWars, ie, Abdominal/Gastrointestinal and Trauma, where a resident attended two or more sessions and it resulted in significantly increased their ITE score in that subcategory. Our results are consistent with other studies and show that repetition of simulation improves performance by enhancing memory and problem-solving skills.^11,20–22 Hendricks showed that after 3 or more SimWars sessions, learners’ scores were similar to experts who took the same simulation once.¹²

The clinical implications of improving clinical reasoning skills in a practicing physician have a profound impact on patient outcomes. Medical errors are felt to be the third leading cause of death in the US.²⁶ Since diagnostic error can be caused by faulty clinical reasoning, SimWars could help save lives by allowing physicians to adopt the right path for patient evaluation and diagnosis. This is particularly important for training residents in the Emergency Department where most of the diagnostic evaluation occurs. Even though our study did not show statistical improvement in all subcategories of ITE scores or GRS scores with SimWars, even a slight improvement in clinical reasoning can have a large impact on patient’s lives and outcomes.

The major limitation of this study was its retrospective design. We were not able to randomly assign students to an intervention and control group due to its retrospective nature so we used a historical control. There are limitations with a historical control, because there may have been other educational teaching and clinical experiences during the time of the controls which did not make the ITE scores significantly different. Another limitation was that the Simulation War scores were not consistently recorded to evaluate how well the EM resident performed. Therefore, instead of comparing Simulation War scores with ITE scores and GRS scores, we evaluated whether EM residents’ participation in Simulation Wars affected ITE scores and GRS scores. Residents may not have tried as hard or retained as much because they did not feel pressured as their performance was not consistently recorded. This may have actually made the residents feel less intimidated though, allowing them to make mistakes in a less threatening environment. The extent to which residents felt the competition was important was not evaluated since a postsurvey on Sim Wars was not performed. Another limitation was the fact that this study occurred at a single residency program at a single site which may have affected the generalizability of the results.

Conclusion

SimWars in subcategories such as Abdominal/GI, Thoracic, OBGyn and Trauma improved ITE scores in those subcategories. It is known that Emergency Medicine physicians utilize clinical reasoning skills the most; therefore, any exercise that improves clinical reasoning should be considered vital for EM residents. This will positively affect patient outcomes since faulty clinical reasoning can lead to medical errors.