Identification of gender differences in ultrasound milestone assessments during emergency medicine residency training: a pilot study

Introduction

Although significant advancements have been achieved over the past few decades, gender bias remains pervasive throughout medicine. Emergency Medicine (EM) is still a male-dominated field, with women making up only 38% of physicians.¹ There are fewer women than men in EM residencies, emergency departments (EDs), and in positions of leadership.^2,3 Several studies have explored gender differences in resident training evaluations among various specialties, including EM, with results suggesting an implicit bias towards female residents.

Recent literature has demonstrated that male residents attained higher level milestones, most evident by graduation time. The concept of milestone development for residency training was introduced in 2008. The educational milestones represent specialty-specific accomplishments and behaviors that occur during the process of becoming a competent physician. They are useful in identifying and tracking resident attainment of core knowledge and skills throughout their training.⁴ Bedside ultrasound is now recognized as one of the essential procedural skills for an emergency physician. The Accreditation Council for Graduate Medical Education (ACGME) has designated 23 required milestone competencies for EM residents, one being ultrasound. The Patient Care 12 (PC12) ultrasound sub-competency outlines five advancing skill levels ranging from level 1 through 5. The acquisition of ultrasound sub-competency skills is assessed for every resident every 6 months. EM residents are expected to meet level 4 PC12 sub-competency requirements prior to graduation.⁵ Prior literature among other procedure-heavy specialties have demonstrated significant incongruities among males and females in regards to the assessment of attainment of procedural skills during residency. Studies have found disparities not only in the level of autonomy female residents are given while performing procedures when compared to their male colleagues, but also in the type and number of procedures performed overall.^6–8 To our knowledge there were no previous studies specifically investigating gender differences in the assessment of psychomotor or procedural skills among EM residents, such as those required by ultrasound. The purpose of this study was to evaluate for gender differences in ultrasound milestone assessments among EM residents.

Methods

Study design/study setting

This is a retrospective study at an academic institution that includes residents from three residency programs: two categorical 3-year EM residency programs and one combined 5-year pediatric-EM residency program. All programs are ACGME-accredited. Residents train in two urban academic EDs totaling approximately 110,000 visits per year. Institutional board review approval was obtained for this study. Hospital-based credentialing in point-of-care ultrasound (POCUS) is available for attending physicians at both EDs and was developed from American College of Emergency Physicians ultrasound guidelines.⁹ POCUS examinations were performed at bedside by both EM residents and attending physicians. All POCUS studies performed at both EDs were stored in the web-based workflow solutions database, Qpath (Q-path, Telexy Healthcare, BC, Canada). All images and interpretation reports were reviewed for quality assurance by emergency ultrasound fellows or ultrasound fellowship-trained faculty.

Study population/inclusion criteria

The study population includes residents trained at one of three residency programs over a three-year period between June 2014 to June 2017. The milestones evaluation data on these residents were collected as a part of their residency training and competency assessment. All former or current residents who had milestone evaluations recorded over a three-year period were included in the study.

Study protocol

The PC12 sub-competency outline recommended by ACGME was used to determine the ultrasound competency for both female and male residents. The ultrasound milestone levels were assessed biannually for both male and female residents. There were a total of six milestone assessments taken during the study period. Several components were considered when determining which level milestone each resident achieved. Residents were evaluated based on their performance during a dedicated ultrasound rotation by both male and female emergency ultrasound faculty. During this rotation, emergency ultrasound faculty members directly observed residents’ ultrasound skills and the integration into clinical care. Residents were also evaluated based on the number and quality of POCUS studies completed during their regular shifts in the ED. The evaluation scores at each assessment were between 1 and 5: Level 1: “describes the indications for emergency ultrasound;” Level 2: “Explains how to optimize ultrasound images and identified the proper probe for each of the focused ultrasound applications, performs an eFAST;” Level 3: “Performs goal-directed focused ultrasound exams, correctly interprets acquired images;” Level 4: “Performs a minimum of the 150 focused ultrasound examinations;” Level 5: “Expands ultrasonography skills to include: advanced echo, TEE, bowel, adnexal and testicular pathology, and transcranial Doppler”.

Outcome measures

The outcome measures included overall mean milestone levels, milestone levels at baseline and graduation, and differences in milestone achievement between baseline and graduation.

Statistical analysis

Descriptive statistics were used to summarize the data. An unpaired Student’s t-test was used to compare milestone scores between female and male residents at each assessment.

Results

A total of 456 ultrasound milestone evaluations were collected from 91 EM residents (34 females [37%] and 57 males [63%]) during the study period. No significant differences were noted in the overall mean milestone level between females (2.3±0.6) and males (2.2±0.6) (P=0.387). Also, there were no significant differences noted in the ultrasound milestone level between females (0.8±0.6) and males (0.7±0.7) at baseline (P=0.754). Although it did not reach statistical significance (P=0.197), the increase in the mean ultrasound milestone level from baseline to graduation was greater in males (3.4±0.7) compared to females (3.1±0.7) (Figure 1).

Figure 1 Mean milestone evaluation scores at 6-month intervals over a 36-month period.

Discussion

In this pilot study, we found no statistically significant differences in the overall mean ultrasound milestone levels between females and males. However, it was noted that the increase in the mean ultrasound milestone level from baseline to graduation was greater in males compared to females, although it did not reach statistical significance. It is possible that an element of implicit gender bias is still present. Implicit gender bias is an unconscious prejudice for one gender over the other, for which the evaluator may not be aware. Many studies in a variety of fields have demonstrated a gender bias. A recent study by Dayal et al found that despite having similar skills and fund of knowledge at the beginning of residency, female EM residents were evaluated lower on EM sub-competencies by the time of graduation than their male counterparts.¹⁰ Axelson et al showed that men were given more credit for being a “quick learner” than females, which led to males receiving more positive evaluations.¹¹

As many of the procedures performed in the ED require ultrasound guidance, the ultrasound milestone evaluations are closely tied to procedural skills assessments. In creating this study, we considered that perhaps the outcomes from prior literature would also translate into similar findings in ultrasound training among EM residents. Evaluations of EM sub-competencies showed greater disparity between female and male residents for procedural sub-competencies.¹⁰ When male residents are learning a skill or procedure, they are more likely to receive congruent feedback from faculty members, as opposed to female residents, who receive discordant, conflicting feedback especially regarding autonomy and assertiveness.¹² Differences in procedural evaluations are also seen in surgical specialties. Meyerson et al found that female cardiothoracic surgical residents had less autonomy in the operating room when compared to their male counterparts. When females asked for more autonomy, they were viewed negatively and described as arrogant, self-promoting, or conceited.⁶

Interestingly, despite what was uncovered in prior related prior literature, our study found no significant differences demonstrating a gender bias. It is unclear that while other areas of medicine may have gender bias, ultrasound assessment does not. One suggestion is that EM is a relatively newer specialty, with women entering into this field early on in its development and evolution. This is in contrast to older specialties such as surgery. Another possibly is that there is in fact a gender difference among ultrasound milestone assessments among EM residents, but our single center study was not able to detect it.

Gender bias is deeply rooted in the socialization of females and males, and it remains pervasive in medicine. The gender disparity does not stop with medical school or residency training. It continues and is exacerbated as women advance further in their career. Although both men and women harbor biases, we must continue to work towards gender equality for the future of our specialty. Publications such as this one that shed light on the fact that there perhaps there is progress within EM. Albeit a limited study, it appears that in some fields of medicine we slowly closing the gap to attaining gender equality.

Limitations

There are several limitations to this pilot study. This study was retrospective and therefore comes with some inherent limitations in how the milestones were assessed prior to data collection by the authors. However, while a retrospective study design weakens its validity, a prospective study on gender bias could also have limitations due to a Hawthorne effect. If an evaluator were aware that they were being studied, they may be inclined to change their behavior when completing the milestone assessments.

Another limitation is that this study was conducted at a single site, restricting our sample size to only the residents of the three local residencies. This also led to only a small number of evaluators who completed the milestone assessments. This institutional bias can be overcome in future studies by conducting a larger, multicenter study. As this was a small, pilot study, no apriori power analysis was performed. By sampling a larger number of residencies throughout the country we would be able to increase our sample size and perform a power analysis to better contextualize our results. A multicenter study would also increase the diversity of evaluators which may assist in decreasing the effect of subjective bias with regards to the assessments by the evaluators.

In this study, residents were evaluated by both male and female emergency ultrasound faculty members. Unfortunately, the analysis did not include whether each resident was evaluated by the male or female evaluator. Incorporating the effect the faculty member’s gender had on resident evaluations would have led to a more robust study. This would be a valuable topic to address in any future research.

There is also concern in regards to the reliance on milestone assessments as a valid way to gauge a resident’s progression in ultrasound. At the institution in this study, the resident milestone assessments were completed by two evaluators, with a process that remained consistent throughout the study period. As it stands, the process of assessing ultrasound milestones leaves room for inconsistences among different evaluators. The PC12 was largely criticized by the ultrasound community for the lack of congruence with resident progression of learning.¹³ If the process of utilizing milestones as a means of assessing residents’ ultrasound competency is indeed flawed, then the concern is that this study has a type II error. For example, it may be true that a gender bias exists but the current methodology may be incapable of proving that due to limitations of the current framework of milestone assessments. Additionally, at our institution the evaluators didn’t pair the timing of the resident ultrasound rotation with milestone assessments which could have contributed to differences in the assessments.

Conclusion

Overall, there were no statistically significant differences in the mean ultrasound milestone levels between females and males. The rate of ultrasound milestone level achievement during EM residency training at our institution had a slight tendency to be higher for males than females in the observed residency programs; however, this also did not reach statistical significance. Possible gender bias while evaluating ultrasound milestone levels needs to be further studied on a larger scale.

Acknowledgments

The abstract of this paper was presented at Society for Academic Emergency Medicine, Research Forum, IN, May 2018 as an oral presentation. The abstract of this paper was presented at Society for Academic Emergency Medicine, Western Regional Meeting, NM, February 2018 as an oral presentation.

Disclosure

The authors report no conflicts of interest in this work.

References

1. The state of women in academic medicine: the pipeline and pathways to leadership, 2013–2014 [Internet]. Washington, DC: Association of American Medical Colleges; 2014. Available from: https://members.aamc.org/eweb/upload/The%20State%20of%20Women%20in%20Academic%20Medicine%202013-2014%20FINAL.pdf. Accessed August 3, 2017.

2. Cheng D, Promes S, Clem K, et al. Chairperson and faculty gender in academic emergency medicine departments. Acad Emerg Med. 2006;13(8):904–906. doi:10.1197/j.aem.2006.01.025

3. Wehner MR, Nead KT, Linos K, et al. Plenty of moustaches but not enough women: cross sectional study of medical leaders. BMJ. 2015;351:h6311. doi:10.1136/bmj.h6311

4. Beeson MS, Carter WA, Christopher TA, et al. The development of the emergency medicine milestones. Acad Emerg Med. 2013;20(7):724–729. doi:10.1111/acem.12157

5. The Emergency Medicine Milestone Project: A Joint Initiative of the Accreditation Council for Graduate Medical Education and the American Board of Emergency Medicine [Internet]. Chicago, IL: American Board of Emergency Medicine; 2015. Available from: http://www.acgme.org/acgmeweb/Portals/0/PDFs/Milestones/EmergencyMedicineMilestones.pdf. Accessed June 9, 2018.

6. Meyerson SL, Sternbach JM, Zwischenberger JB, et al. The effect of gender on resident autonomy in the operating room. J Surg Educ. 2017;74(6):e111–e118. doi:10.1016/j.jsurg.2017.06.014

7. Garcia-Rodriguez JA, Dickinson JA, Perez G, et al. Procedural knowledge and skills of residents entering Canadian family medicine programs in Alberta. Fam Med. 2018;50(1):10–21. doi:10.22454/FamMed.2018.968199

8. Chaytors RG, Szafran O, Crutcher RA. Rural-urban and gender differences in procedures performed by family practice residency graduates. Fam Med. 2001;33(10):766–771.

9. Ultrasound guidelines: emergency, point-of-care and clinical ultrasound guidelines in medicine. Ann Emerg Med. 2017;69(5):e27–e54. doi:10.1016/j.annemergmed.2016.08.457

10. Dayal A, O’Connor DM, Qadri U, Arora VM. Comparison of male vs female resident milestone evaluations by faculty during emergency medicine residency training. JAMA Intern Med. 2017;177(5):651–657. doi:10.1001/jamainternmed.2016.9616

11. Axelson RD, Solow CM, Ferguson KJ, Cohen MB. Assessing implicit gender bias in medical student performance evaluations. Eval Health Prof. 2010;33(3):365–385. doi:10.1177/0163278710375097

12. Mueller AS, Jenkins TM, Osborne M, Dayal A, O’Connor DM, Arora VM. Gender differences in attending physicians’ feedback to residents: a qualitative analysis. J Grad Med Educ. 2017;9(5):577–585. doi:10.4300/JGME-D-17-00126.1

13. Nelson M, Abdi A, Adhikari S, et al. Goal-directed focused ultrasound milestones revised: A multi-organizational consensus. Acad Emerg Med. 2016;23(11):1274–1279. doi:10.1111/acem.13069