Shadowing emergency medicine residents by medical education specialists to provide feedback on non-medical knowledge-based ACGME sub-competencies
Received 8 September 2017
Accepted for publication 2 February 2018
Published 4 May 2018 Volume 2018:9 Pages 307—315
Checked for plagiarism Yes
Review by Single anonymous peer review
Peer reviewer comments 2
Editor who approved publication: Dr Md Anwarul Azim Majumder
Anna L Waterbrook,1 Karen C Spear Ellinwood,2 T Gail Pritchard,3 Karen Bertels,1 Ariel C Johnson,4 Alice Min,1 Lisa R Stoneking1
1Department of Emergency Medicine, The University of Arizona College of Medicine, Tucson, AZ, USA; 2Department of Obstetrics and Gynecology, The University of Arizona College of Medicine, Tucson, AZ, USA; 3Department of Pediatrics, The University of Arizona College of Medicine, Tucson, AZ, USA; 4College of Medicine, The University of Arizona College of Medicine, Tucson, AZ, USA
Objective: Non-medical knowledge-based sub-competencies (multitasking, professionalism, accountability, patient-centered communication, and team management) are challenging for a supervising emergency medicine (EM) physician to evaluate in real-time on shift while also managing a busy emergency department (ED). This study examines residents’ perceptions of having a medical education specialist shadow and evaluate their nonmedical knowledge skills.
Methods: Medical education specialists shadowed postgraduate year 1 and postgraduate year 2 EM residents during an ED shift once per academic year. In an attempt to increase meaningful feedback to the residents, these specialists evaluated resident performance in selected non-medical knowledge-based Accreditation Council of Graduate Medical Education (ACGME) sub-competencies and provided residents with direct, real-time feedback, followed by a written evaluation sent via email. Evaluations provided specific references to examples of behaviors observed during the shift and connected these back to ACGME competencies and milestones.
Results: Twelve residents participated in this shadow experience (six post graduate year 1 and six postgraduate year 2). Two residents emailed the medical education specialists ahead of the scheduled shadow shift requesting specific feedback. When queried, five residents voluntarily requested their feedback to be included in their formal biannual review. Residents received milestone scores and narrative feedback on the non-medical knowledge-based ACGME sub-competencies and indicated the shadow experience and subsequent feedback were valuable.
Conclusion: Medical education specialists who observe residents over the course of an entire shift and evaluate non-medical knowledge-based skills are perceived by EM residents to provide meaningful feedback and add valuable information for the biannual review process.
Keywords: evaluation, observation, milestones, postgraduate, residency, behaviors, performance
The performance of emergency medicine (EM) residents is currently reviewed and evaluated semiannually using milestones designed to assess resident progression through their EM training. The Accreditation Council of Graduate Medical Education (ACGME) has implemented a developmental assessment system where residents’ progress is measured according to milestones, a matrix of the knowledge, skills, abilities, and attitudes, aligned with the ACGME competencies in the context of EM experiences.1 All specialties, including EM,2 have a set of sub-competencies that should be demonstrated during specialty training outlining resident progress from novice to more advanced within a developmental framework. Each milestone describes specific behaviors expected and targets for resident performance from entry into residency through graduation.
Generally, most milestone evaluations are completed by EM attendings, who are supervising multiple residents on shift. A smaller subset of evaluations come from ancillary staff, off-service rotations, patients, simulation medicine, and oral board review cases. These evaluations often focus on medical knowledge, patient care, treatment/management, procedures, and disposition-based sub-competencies. It has been empirically found that non-medical knowledge-based sub-competencies, including multitasking, professionalism, accountability, patient-centered communication, and team management (Figure S1) are more challenging for a supervising physician to evaluate while managing a busy emergency department (ED). Non-medical knowledge-based sub-competencies tend to be nuanced, process-oriented behaviors that may vary depending upon interpersonal dynamics, types of procedures being performed, outcomes, or time of shift (early, late in shift) and present greater challenges for evaluation. Attending physicians often have limited time at the bedside with residents leading to many interactions not directly observed by an attending physician. The authors believe having a medical education specialist shadowing a resident for an entire shift allows him/her the opportunity to provide a deeper level of feedback on the nonmedical aspects of performance. However, there is no published evidence on physicians’ ability to evaluate residents’ non-medical knowledge-based sub-competencies.
In an attempt to provide our residents with additional, more meaningful and unbiased feedback on these non-medical knowledge-based ACGME sub-competencies, a shadow shift with a nonphysician medical education specialist with expertise in these areas was instituted. During the shadow shift, a medical education specialist spends an entire shift with the resident, observing during patient interactions, team interactions, consultations with other providers, completion of shift responsibilities, and all other work-related tasks. This assessment from the medical education specialist is in addition to other modalities of resident evaluations. This study sought to understand the residents’ perception of observation and feedback provided by nonmedical personnel, specifically a medical education specialist.
This project was reviewed and approved by an institutional review board (IRB) chair or designee at the University of Arizona and was determined to meet the criteria for exemption. The shadow experience took place at an urban, academic hospital system with an approximate annual ED volume of 54,000 patients within a 3-year categorical EM residency program. The shadow shifts were instituted as part of a quality improvement initiative for our curriculum in 2015 aiming to increase meaningful feedback for residents. Two medical education specialists affiliated with the University of Arizona College of Medicine shadowed each of our postgraduate year 1 (PGY1) and postgraduate year 2 (PGY2) EM residents for a full 9-hour ED shift. Medical education specialists have expertise in medical education and ethnographic research as well as experience working with residents and faculty on communication and presentation skills in various educational contexts. These specialists evaluated resident performance using an observation protocol based upon selected non-medical knowledge-based ACGME sub-competencies and then provided residents with direct verbal and written feedback.
The pilot observation protocol was inspired by observation protocols for clinical teaching created by the University of California, San Francisco, initially adapted by the Academy of Medical Education Scholars for use at the authors’ affiliated university due to their emphasis on communication and interpersonal skills, and the close relationship of the skills and behaviors required for effective patient education, peer and supervisor interaction, and consultation with other physicians (Figure S2). The medical education specialist added descriptions of behaviors to align the protocol with current ACGME expectations for resident performance with respect to patient, peer, and attending communication. These items were checked against behaviors that the medical education specialists expected to see when observing for resident development, as well as a validated 25-item evaluation of teaching,3 to ensure the behaviors and manner of assessing these were grounded in the literature and practice. The medical education specialists used the protocol as a guide, rather than as a strict instrument, for the types of behaviors to look for in residents’ interactions with patients, consulting colleagues, peer residents, and supervising attendings. As a result of the pilot, the protocol was revised to include an evaluation of behavior included in selected nonmedical knowledge milestones (Figure S1) that aligned more closely with observations and reflected a deeper understanding of the nature of the activities and interactions in clinical settings.
During and following the shadow experience, the medical education specialist offered verbal, real-time feedback to the residents directly concerning their performance of behaviors, demonstrating ACGME sub-competencies and EM milestones. In addition, the medical education specialist provided an extensive written evaluation offering constructive feedback with descriptions of specific, relevant observed behaviors. The feedback followed standard guidelines for giving constructive feedback promoting reflection on practice and improvement of performance.4,5 Following the receipt of the written feedback, residents were invited to respond to and/or meet with the medical education specialist to discuss the feedback and/or recommendations for or guidance on how to improve (Figure S2). Residents also were given the option to include their formal written evaluations from the medical education specialist in their formal biannual review with their program director.
The residents were asked to complete a survey prior to the shadow shift, and after receiving written evaluations, documented their perceptions of the value this process would have and/or has for their development. Informed consent for this study was obtained by the residents agreeing to complete the survey. Nine residents completed the surveys, which contained scaled agreement items concerning anticipated and actual value of the shadow shift and written evaluation process for overall development and improving performance of EM milestones. The items included in the survey were as follows: accurately reflected residency performance, identified specific behaviors, skills, or knowledge that I do well, identified specific behaviors, skills, or knowledge that I need to improve, offered me guidance to improve, residency milestones, overall efficiency, communication skills, ability to establish and maintain effective patient rapport, ability to engage in effective teamwork, time management, and organizational skills. Survey items were included and rated for verbal and written feedback separately.
All six PGY1 and six PGY2 residents participated in shadow shift evaluations. Two residents emailed the medical education specialist ahead of the scheduled shadow shifts requesting feedback in specific areas of practice. Five residents “opted in” and requested their feedback to be included in their formal biannual review with the program director. All 12 residents were scored along the milestones and given narrative feedback on the non-medical knowledge-based sub-competencies.
The surveys showed most participants agreed the shadow shift evaluation was valuable (Figure 1). Only one resident strongly disagreed. Anecdotally, residents who were shadowed (with or without completing the pre- and post-surveys) indicated to both program directors and medical education specialists the shadowing experience and subsequent feedback were valuable. One resident contacted their medical education specialist to discuss how to expand the pilot with a focus group to enable residents to provide constructive feedback on the shadow shift and evaluation process.
Figure 1 Residents’ perceived value of the shadow shift evaluations.
Abbreviation: EM, emergency medicine; VF, verbal feedback; WF, written feedback.
This study examines residents’ perceptions of having a medical education specialist shadow and evaluates nonmedical knowledge skills. Our residents reported the shadowing experience and subsequent feedback were valuable, indicating their receptivity to the shadowing and evaluation by nonphysician medical education specialists. This inspired the residency program directors to implement the shadow/evaluation shift as a regular part of the standard resident evaluation process. These written evaluations provided important additional data on resident performance in the ED that is now sent to the Clinical Competency Committee for review prior to the biannual review with the program director. Shadow shift evaluations also resulted in referring two residents to the medical education specialist to receive additional support to further develop skills, highlighting the developmental, rather than punitive, purpose of the process.
Shadow shift evaluations aim to expand how program directors can conduct the 360° evaluation process by providing narrative, formative feedback within a competency-based evaluation. The purpose of a 360° evaluation process is to provide assessments of the resident performance from multiple perspectives of various members of the health-care team as well as the patient. These assessments are being used more frequently in postgraduate medical training. One previous study in our ED revealed positive resident perception of nonclinician evaluation and feedback following social workers’ observation of residents’ delivery of “bad news” to patients and their families.6 However, we have not found in the literature the regular incorporation of an evaluation by a nonclinician, nonpatient evaluation, nor one based on an entire shift or focused on the non-medical knowledge-based sub-competencies. The aim of a 360° evaluation process is to provide a “more comprehensive framework for the assessment of physician performance.”7 The value of any evaluation for the learner lies in the formative feedback with guidance on how to improve performance.4,5 For the program directors, the value of a 360° evaluation is the ability to identify the problems in performance that might not be captured by the attendings’ observations but rather noticed by nurses, case managers, consulting physicians, or patients and their families.
Shadow shift evaluations aim to expand how directors can conduct the 360° process by providing narrative, formative feedback within a competency-based evaluation. Concentrating on clinical skills such as communication and professionalism and considering multiple patient, peer, staff, and attending interactions over 9 hours rather than a single interaction provides a more accurate representation of the residents’ day-to-day performance. The shadow shift has the added advantage of removing the potential for workplace relationship bias since the medical education specialists are not part of the health-care delivery team, though they have been immersed in the process as observers.
The literature regarding the evaluation of trainees strongly suggests the need for using multiple assessment tools for formative and summative evaluation. An overview of effective assessment methodologies by Sherbino et al includes both observed clinical behavior and multisource feedback to be effective tools.8 In the study the authors note, “…comprehensive assessment of competence requires the use of multiple tools in a longitudinal fashion.”8
One model utilized across undergraduate and graduate medical education is workplace-based assessment (WPBA). “Workplace based assessment (WPBA) refers to a group of assessment modalities which evaluates trainees’ performance during the clinical settings.”9 There are at least three key components in using a WPBA model: observation in the actual workplace, providing timely feedback, and the use of multiple sources in making judgments about the learner’s knowledge and abilities. Guraya9 and Norcini and Burch10 explained that, depending on the assessment tools incorporated in WPBA, trainees and evaluators gain insight through observation of clinical performance, direct observation of procedural skills, discussion of clinical cases, multisource feedback, and team assessment of behaviors. The shadow shift evaluation tool is similar to the observation of clinical experience with at least two exceptions. Rather than a single encounter between the trainee and a patient over the course of 15 minutes (eg, mini clinical exam skills and direct observation of procedural skills), the shadow shift occurs over a 9-hour period and across many trainee–patient encounters. Relatedly, the shadow shift adds an additional voice to multisource feedback – the medical education specialist. One criticism of WPBA is the time it takes to collect multiple assessments and then to provide timely feedback. This is a key attribute in utilizing medical education specialists in a shadow shift. They are uniquely positioned to observe and provide real-time feedback regarding the non-medical knowledge-based sub-competencies (multitasking, professionalism, accountability, patient-centered communication, and team management). The ultimate goal of the shadow shift assessment tool is to support the development of a trainee’s comprehensive evaluation.
We found no studies directly addressing resident clinical evaluations on non-medical knowledge-based ACGME sub-competencies nor providing direct feedback by non-physician medical education specialists. One study at George Washington University School of Medicine and Health Sciences discussed shadowing PGY-1 Obstetrics and Gynecology residents in their second month of training.11 This study investigated the activities residents reported performing, how often they were observed, who observed them, how often they received feedback from those who observed them, and their perception of its usefulness.11 These Obstetrics and Gynecology residents reported performing various procedures in their second month of training, with some observations from faculty, senior residents, and nursing staff. Feedback was an important aspect of their early training, although not reported with each patient encounter. This study prompted many questions as follows: Who should provide feedback to residents? Should senior residents and nursing staff be trained to give feedback to residents? Additionally, a study at the University of Mississippi sought to reveal the utility of end-of-shift evaluations of EM residents by clinical faculty.12 They concluded end-of-shift evaluations “rarely (8%, 372 of 4633) rated a resident as not achieving milestones.”12 Other studies within anesthesiology,13 general surgery,14 and internal medicine15 residency programs also highlighted the difficulties with evaluating residents and assigning them scores according to the ACGME milestones. Another study by Angus et al16 surveyed Internal Medicine residents to assess resident perceptions of receiving feedback in the milestone framework. This study revealed that just less than half of the residents found milestone-based feedback “extremely useful” or “very useful” in identifying strengths (44%), weaknesses (43%), specific areas for improvement (45%), and appropriate education progress (48%).16 This study assessed perception of the utility of milestone-based feedback compared to previous forms of non-milestone-based feedback. Our study only assessed non-medical knowledge-based ACGME sub-competencies and did not formally compare to prior forms of evaluation.
Due to the feedback from this program’s experience, we have initiated a similar shadow shift at our sister EM residency program. We are now involved in an IRB-approved study regarding the implementation of these evaluations at both residency programs. This is an ongoing study formally addressing resident’s perceptions of the process and the value of shadowing by medical education specialists as an additional way to evaluate non-medical knowledge-based ACGME sub-competencies. The study will also explore the attendings’, program directors’, and medical education specialists’ perceptions of the value of including the written evaluations in the biannual review of resident performance. We have a threefold goal: 1) enable program directors to continually improve the quality of this assessment process and resident feedback with the input of multiple stakeholders; 2) improve the observation guide to more accurately capture the kinds of interactions and skills observed during the shift; and 3) potentially develop a staff development program where additional nonphysician staff will also be able to shadow and evaluate non-medical knowledge-based sub-competencies as part of the 360° evaluation process. Resident perceptions can be used to adjust and improve the shadow shift evaluation technique for future use, including length of observation time, which milestones to include, usefulness of the feedback, how to use the evaluations, and how to sustain such a program.
This study was limited by the small number of residents who participated in the shadow shift evaluation and the small number of residents who completed pre- and post-surveys. We will continue the study seeking a larger population of residents participating in the study and giving feedback. Specifically, during the shadow shift evaluation process, one medical education specialist requested a second observation of two residents. These two residents were some of the first to be observed and based on what she had learned about the observation/feedback process, she felt she would be able to offer more actionable feedback with a second observation. One of the two residents was amenable and the second observation took place. Continued tailoring of the shadow shift evaluation process and protocol will improve all evaluations to prevent reevaluation.
Another limitation to consider is the possibility residents perform differently while being shadowed by the medical education specialist. We found the length of time spent together did not allow for residents to forge a false presentation of themselves. In addition, a full shift with the resident also allowed the resident to excel and not be judged by a possible poorly managed single incident or encounter. Interactions with an attending physician and resident are generally much shorter and allow for misconceptions of a resident, whereas 9 hours of direct observation can avoid misimpressions. The extended time with the resident can reveal ordinary behavior in a clinical setting, rather than focus on what might be atypical behavior in a given encounter. We recognize having medical education specialist follow a resident during a busy emergency shift may seem cumbersome and inhibitory. However, our experience has taught us the feedback and knowledge gained from this evaluation technique outweighs the possible inconvenience. This novel additional evaluation process for residents is not yet validated. In future research, we hope to achieve validation so this technique can possibly be incorporated into other residency programs.
Medical education specialists who observe EM residents over the course of an entire shift for demonstration of skills unrelated to medical knowledge, such as interpersonal and communication skills and professionalism, are perceived by EM residents as providing meaningful feedback and adding valuable information to the biannual review process.
The authors report no conflicts of interest in this work.
Holmboe ES, Edgar L, Hamstra S. The Milestone Guidebook. 2017. Available from: http://www.acgme.org/Portals/0/MilestonesGuidebook.pdf. Accessed July 12, 2017.
Beeson M, Christopher T, Heidt J, Jones J, Promes S, Meyer L. The Emergency Medicine Milestone Project. 2015. Available from: http://www.acgme.org/Portals/0/PDFs/Milestones/EmergencyMedicineMilestones.pdf. Accessed on July 12, 2017.
Mintz M, Southern DA, Ghali WA, Ma IW. Validation of the 25-item stanford faculty development program tool on clinical teaching effectiveness. Teach Learn Med. 2015;27(2):174–181.
Cantillon P, Sargeant J. Giving feedback in clinical settings. BMJ. 2008;337:a1961.
Hewson MG, Little ML. Giving feedback in medical education: verification of recommended techniques. J Gen Intern Med. 1998;13(2):111–116.
Min AA, Spear-Ellinwood KC, Berman M, et al. Social worker assessment of emergency medicine residents’ skill at bad news delivery: a novel direct-observation milestone assessment. Intern Emerg Med. 2016;11(6):843–852.
Donnon T, Al Ansari A, Al Alawi S, Violato C. The reliability, validity, and feasibility of multisource feedback physician assessment: a systemic review. Acad Med. 2014;89(3):511–516.
Sherino J, Bandiera G, Frank JR. Assessing competence in emergency medicine trainees: an overview of effective methodologies. CJEM. 2008;10(4):365–371.
Guraya YS. Workplace-based assessment; applications and educational impact. Malays J Med Sci. 2015;22(6):5–10.
Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007;29(9):855–871.
Tully K, Keller J, Blatt B, Greenberg L. Observing and giving feedback to novice PGY-1s. South Med J. 2016;109(5):320–325.
Dehon E, Jones J, Puskarich M, Sandifer JP, Sikes K. Use of emergency medicine milestones as items on end-of-shift evaluations results in overestimates of residents’ proficiency level. J Grad Med Educ. 2015;7(2):192–196.
Ross FJ, Metro DG, Beaman ST, et al. A first look at the Accreditation Council for Graduate Medical Education anesthesiology milestones: implementation of self-evaluation in a large residency program. J Clin Anesth. 2016;32:17–24.
Rialon KL, Barfield ME, Elfenbein DM, Lunsford KE, Tracy ET, Migaly J. Resident designed intern orientation to address the new ACGME Common Program Requirements for resident supervision. J Surg Educ. 2013;70(3):350–356.
Choe JH, Knight CL, Stiling R, Corning K, Lock K, Steinberg KP. Shortening the miles to the milestones: connecting EPA-based evaluations to ACGME milestone reports for internal medicine residency programs. Acad Med. 2016;91(7):943–950.
Angus S, Moriarty J, Nardino RJ, Chmielewski A, Rosenblum MJ. Internal medicine residents’ perspectives on receiving feedback in milestone format. J Grad Med Educ. 2015;7(2):220–224.
Figure S1 Milestone assessment
Notes: Copyright (c) 2015 The Accreditation Council for Graduate Medical Education and The American Board of Emergency Medicine. All rights reserved. The copyright owners grant third parties the right to use the Emergency Medicine Milestones on a non-exclusive basis for educational purposes.1
Abbreviation: ACGME, Accreditation Council of Graduate Medical Education; SDOT, Standardized Direct Observation Tool.
Figure S2 Observation protocol for resident shadow/evaluation process and sample evaluation content.
Beeson M, Christopher T, Heidt J, Jones J, Promes S, Meyer L. The Emergency Medicine Milestone Project. 2015. Available from: http://www.acgme.org/Portals/0/PDFs/Milestones/EmergencyMedicineMilestones.pdf. Accessed on July 12, 2017.
© 2018 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.