Leading from Behind: An Educational Intervention to Address Faculty and Learner Preparedness for Competence By Design in Psychiatry

Introduction

Competency-based medical education (CBME) represents a significant paradigmatic shift in how postgraduate medical education is delivered across Canada. CBME is described as an outcomes-based approach to postgraduate medical education with less emphasis on time-based training and more emphasis on learners’ achievements of specific educational objectives, or “competencies”.¹ As many countries are implementing the principles of CBME into their postgraduate medical education curriculum, the Royal College of Physicians and Surgeons of Canada (RCPSC) has started to implement their version of CBME across Canadian specialty training programs, called the Competence By Design (CBD) framework. CBD includes the pre-existing CanMEDS framework with the aim of improving patient care through the enhancement of physician training.² The goal of CBD is to address the gaps and challenges within the current postgraduate educational model during a time of increasing system complexity, limitations in healthcare resources, and changing patient and societal needs.³

The implementation of CBD in Canada is occurring through an iterative process, beginning in 2017 in anesthesiology and otolaryngology – head and neck surgery and continuing through 2021. Within the CBD framework, residents progress through four stages of residency training, including the transition to discipline, foundations of discipline, core of discipline, and transition to practice stages.⁴ Each stage has its own defined CanMEDS-based milestones and entrustable professional activities (EPAs). Milestones describe the expected ability of a healthcare professional at a certain stage of expertise, and EPAs describe a key task of the discipline that an individual should be able to perform independently.⁵ Through frequent workplace-based observations of their performance of EPAs, residents will be provided with timely feedback to further enhance their learning and clinical performance.

While CBD can be viewed as a novel and transformative approach in medical education, several concerns have been raised about its design and implementation. For instance, there is uncertainty regarding the new medical education terminology being implemented that is specific to the CBD framework, theoretical and practical disagreements about how “competence” will be measured, concerns about excessive reductionism when deconstructing skills into component competencies, lack of robust evidence to support sweeping, national changes to medical education, and concerns with the number of resources required to complete this implementation.^1,6–9

As faculty and learners alike are introduced to the CBD framework, implementation of an educational paradigm without first assessing program readiness is a potential impediment to successful implementation.^10,11 In the 2018 Resident Doctors of Canada (RDoC) national survey of residents, 30.2% of respondents indicated that their programs had rolled out CBME and 17.0% of respondents indicated that they were formally participating in this training model.¹² Of those residents in training programs which had rolled out the CBME model, 64.4% of residents reported that they felt adequately informed about CBME and 36.5% of respondents felt that their preceptors were adequately informed about CBME; however, it is noted that this survey did not provide preceptors’ perceptions of their preparedness for CBD. If researchers and educators can prospectively identify/assess faculty and learner preparedness for the implementation of a national programmatic shift (such as CBD), then efforts and resources can be appropriately allocated to address perceived challenges ahead of time in order to promote a successful transition. Given the continuous international policy changes observed in medical education, it is timely and indeed relevant to dedicate research efforts to identify and address anticipated challenges to new policy implementation initiatives.

Hence, the goal of this paper was to assess faculty members’ and learners’ understanding of, and preparedness for, the national shift to CBD in psychiatry before and after an educational intervention.

Materials and Methods

The authors received written notification from the Hamilton Integrated Research Ethics Board indicating that the current research was exempt from oversight given its quality improvement focus.

Study Context

This research was conducted during a special one-hour “Education Day”, an annual education event hosted by the Department of Psychiatry and Behavioural Neurosciences at McMaster University (Hamilton, Ontario, Canada). This annual event occurred during (and replaced) the regularly scheduled weekly grand rounds event. Grand rounds events, which typically involve one-hour educational presentations on a psychiatry-related topic, are accredited by the RCPSC for continuing professional development credit; therefore, these events are highly attended by faculty and learners alike. Furthermore, previous “Education Day” events have also been well attended by both faculty and learners, thus providing a fruitful opportunity for convenience sampling for the current research. “Education Day” events are, by design, one hour in duration to coincide with most faculty members’ and learners’ schedules.

The special “Education Day” event was organized by a departmental committee comprised of senior education leadership, faculty representation, inter-disciplinary representation, and student representation. The committee was responsible for: planning the event, inviting the speaker, selecting the topic, and disseminating the event information to the target audience. This event was specifically advertised to faculty and learners within the Department of Psychiatry and Behavioural Neurosciences. The event included an educational presentation by a CBME expert that aimed to educate faculty and learners about CBME and CBD. More specifically, the presentation included content which sought to address the following learning objectives: (1) to describe the principles of CBME and their application to postgraduate residency training in psychiatry; (2) to present the requirements of faculty and learners within the new CBD framework developed by the RCPSC; and (3) to identify educational and training needs to help facilitate the rollout of the CBD framework in psychiatry (see Supplementary File 1).

Study Design

The current project utilized a pre-test/post-test quantitative study design to evaluate faculty members’ and learners’ understanding of, and preparedness for, CBD. A brief, paper-based pre-survey (Supplementary File 2) and post-survey (Supplementary File 3) were developed specifically for this study. The surveys were administered to all educators, physicians, and learners (n = 104) who attended the special one-hour “Education Day” grand rounds event. Participants completed the pre- and post-surveys immediately before and after the presentation, respectively. Participation in this research was voluntary. Participants consented to participate in this research through their completion and submission of the survey(s). Both surveys were completed anonymously, and all participants were instructed that the data collected would be disseminated in a scholarly way. Completion of both pre- and post-surveys allowed participants to enter a draw to win a $100 gift card as an incentive to complete both surveys. An alphanumeric code was used to link the two surveys.

Materials

The pre- and post-surveys were designed to elicit faculty and learner perceptions of the following concepts: general knowledge of CBME and CBD, resources required for CBD implementation, faculty and learner receptiveness towards CBD, confidence in educational/clinical outcomes secondary to CBD, and perceived training requirements needed to implement CBME/CBD. Two researchers (BB, SP) conducted a comprehensive literature review to guide survey design and question development. All questions were reviewed by members of the Education Scholarship Committee in the Department of Psychiatry and Behavioural Neurosciences for accuracy, relevance, and importance. This committee is comprised of more than twenty educators, researchers, learners, and physicians in psychiatry. The final pre- and post-surveys consisted of 15 and 12 quantitative questions, respectively, with 11 questions repeated in both surveys to capture pre-post perceptions (see Supplementary Files 2 and 3). Answer formats included 7-point Likert scales and multiple-choice (i.e., single response and “check all that apply”) questions. An additional open-ended question was included in the post-survey which allowed participants to provide written responses to the question, “Do you have any suggestions for improvement?”

Data Analysis

Participants were categorized into two groups according to their self-reported professional identity: faculty (psychiatrists, psychologists, medical teachers, clinical supervisors, other [i.e., pharmacist, behavioral therapist, nurse educator, non-physician assistant professor, research assistant]) and learners (resident by training level, medical student by training level, and other [i.e., physician assistant student]). Descriptive analyses were performed on the demographic and multiple-choice questionnaire data for the pre- and post-surveys for both faculty and learners. Differences between the pre- and post-survey Likert scale data for faculty and learners were compared using Wilcoxon signed rank tests with the Benjamini-Hochberg correction for multiple comparisons. Effect sizes were calculated for significant pre-survey and post-survey differences using Vargha and Delaney’s A.¹³ Values of A were interpreted using the following anchors: small effect size (A = 0.56), medium effect size (A = 0.64), and large effect size (A ≥ 0.71).¹³ All statistical tests were performed using R software.

Results

Of the 104 session attendees, 83 (79.8%) completed and submitted the pre-survey (45 faculty and 38 learners). A total of 80 attendees (76.9% of all attendees, 96.4% of attendees who completed the pre-survey) completed and submitted the post-survey (44 faculty and 36 learners). Participants’ professional demographics are presented in Table 1.

Table 1 Participants’ Self-Reported Professional Demographics

Knowledge of CBME and CBD

Faculty and learners both reported a moderate baseline level of knowledge of CBME (i.e., M = 3.8, SD = 1.7, and M = 2.8, SD = 1.4, respectively). The median level of knowledge of CBME was significantly higher following the one-hour grand rounds education session for both faculty (i.e., Md = 4.0 pre-survey vs 5.0 post-survey, W = 683, p = 0.03, A = 0.66, small effect) and learners (i.e., Md = 3.0 pre-survey vs 4.0 post-survey, W = 394, p < 0.01, A = 0.73, medium effect). Similarly, both groups reported a moderate level of knowledge of the CBD framework at baseline (i.e., faculty: M = 3.8, SD = 1.6; learners: M = 2.6, SD = 1.2); however, only the learners demonstrated a significantly higher level of knowledge of the CBD framework following the education session (i.e., faculty: Md = 4.0 pre-survey vs 4.0 post-survey, W = 1172, p = 0.08; learners: Md = 3.0 pre-survey vs 4.0 post-survey, W = 1019, p < 0.01, A = 0.76, large effect). Finally, faculty and learners both reported a weak-to-moderate baseline level of knowledge regarding the concepts of milestones and EPAs (i.e., faculty: M = 3.5, SD = 1.6; learners: M = 2.5, SD = 1.4). Only the learners demonstrated a significant increase in knowledge of these concepts following the education session (i.e., faculty: Md = 4.0 pre-survey vs 4.0 post-survey, W = 1203, p = 0.08; learners: Md = 2.0 pre-survey vs 3.0 post-survey, W = 880, p = 0.04, A = 0.69, medium effect).

Preparedness for CBD and Perceived Needs

While faculty reported that they were somewhat prepared for psychiatry’s transition to the CBD framework (i.e., M = 3.5, SD = 1.4), learners reported that they were largely unprepared for this transition (i.e., M = 1.9, SD = 1.2). A total of 17 (38.6%) faculty and 7 (19.4%) learners reported that they had accessed at least one resource provided by the RCPSC (i.e., documents and/or presentations on CBD rationale, implementation, and/or accreditation). Both faculty (Md = 4.0, SD = 0.7) and learners (Md = 4.0, SD = 0.6) reported that these resources were only somewhat helpful. Additionally, following the education session, only learners’ self-reported preparedness for the transition to CBD significantly improved (i.e., faculty: Md = 4.0 pre-survey vs 4.0 post-survey, W = 1124.5, p = 0.13; learners: Md = 1.5 pre-survey vs 2.0 post-survey, W = 819, p = 0.02, A = 0.62, small effect). The majority of faculty and learners reported the need for at least a moderate amount of training and/or education to help develop their skills within the CBD framework (see Table 2).

Table 2 Summary of Survey Results for the Multiple-Choice Questions

Expectations of Outcomes

The majority of faculty and learners anticipated that the greatest benefit associated with psychiatry’s transition to CBD will be the ability to provide meaningful feedback to learners (see Table 2). On the other hand, both faculty and learners anticipated that the greatest barrier or challenge associated with psychiatry’s transition to CBD will be the greater demands on time and the increased frequency of evaluation (see Table 2). Nevertheless, both faculty and learners reported that they were somewhat confident that the transition to CBD would improve the educational and/or training experiences of the psychiatry residents (i.e., faculty: M = 5.0, SD = 2.0; learners: M = 4.3, SD = 1.8); however, confidence did not significantly improve following the education session for faculty (i.e., Md = 4.0 pre-survey vs 4.0 post-survey, W = 954, p = 0.12) or learners (i.e., Md = 4.0 pre-survey vs 5.0 post-survey, W = 666, p = 0.12). Furthermore, both faculty and learners reported that they were somewhat confident that the implementation of the CBD framework would improve faculty development among departmental clinicians and educators (i.e., faculty: M = 5.2, SD = 2.2; learners: M = 4.4, SD = 1.7). Analyses revealed that the education session had no significant influence on faculty members’ (i.e., Md = 4.0 pre-survey vs 4.0 post-survey, W = 935, p = 0.13) or learners’ (i.e., Md = 4.5 pre-survey vs 5.0 post-survey, W = 546.5, p = 0.39) confidence in improving faculty development.

Evaluation of the Intervention and Future Learning Needs

Overall, both faculty (i.e., M = 4.8, Md = 5.0, SD = 1.1) and learners (i.e., M = 4.5, Md = 5.0, SD = 1.4) reported that the single education session was at least somewhat effective at improving their knowledge of the CBD framework. Both faculty and learners endorsed “several educational sessions or workshops” as the best format to help further develop their skills within the CBD framework, with “the distribution of relevant materials” and “online training/educational modules” representing other popular formats (see Table 2). Both faculty and learners agreed that the most useful resource that the department could provide to facilitate the transition towards CBD practices would be faculty training (see Table 2).

A total of 9 faculty (20.6% of all faculty who completed the post-survey) and 7 learners (19.4% of all learners who completed the post-survey) provided written comments to the post-survey question, “Do you have any suggestions for improvement?” (Supplementary File 4). Overall, faculty reported that the inclusion of more concrete examples of how to implement CBD, and what the framework would look like once it is implemented, would have improved the intervention. Furthermore, faculty reported that the distribution of take-home/hand-out materials and/or an executive summary of the presentation would have been beneficial. Learners, on the other hand, reported that more information regarding basic CBME and CBD principles would have improved the intervention.

Discussion

CBME is being implemented across many countries worldwide, yet there continues to be a dearth of information related to faculty members’ and learners’ experiences with this transition.^14,15 The current research aimed to identify faculty and learner preparedness for the transition to CBD in the psychiatry postgraduate training program at McMaster University before and after a brief educational intervention. Results from the pre- and post-surveys demonstrated that faculty and learners reported having a moderate baseline level of knowledge of CBME and that there was a significantly higher level of knowledge of CBME following the one-hour education session. These were surprising findings given that only the minority of attendees identified having previously accessed any of the RCPSC resources on CBD. These findings bring into question the distinction between familiarity of terms and true knowledge, as well as the role of self-assessment bias.

Despite reporting moderate baseline knowledge, faculty reported feeling only somewhat prepared for psychiatry’s transition to CBD, whereas learners reported feeling largely unprepared for this transition. This is in contrast to results from the RDoC National Resident Survey that showed that 64.4% of residents in training programs that had already rolled out CBME reported that they felt adequately informed about CBME, whereas only 36.5% felt that their preceptors were adequately informed about CBME.¹² Perhaps there is a shift that occurs through experiential learning as residents learn by necessity how to operate within the CBD framework once the framework is formally launched. That learners in the current study felt unprepared for the transition to CBD is an important finding given that learners have been found to play a key role in increasing staff awareness and understanding of CBME in programs that have already undergone the launch of this educational framework.¹⁵ This highlights the need to target educational initiatives on CBD to learners.

The majority of faculty and learners predicted that the greatest benefit associated with psychiatry’s transition to CBD will be the ability to provide more meaningful feedback to learners. Simultaneously, both faculty and learners also anticipated that the greatest challenge associated with psychiatry’s transition to CBD will be an increased frequency of evaluation and the associated increased demands on time. This is consistent with other studies from anesthesiology and urology residency training programs in which evaluation fatigue and greater resource intensity have been identified as significant concerns associated with CBME.^10,12,14,15 Our findings highlight an inherent tension between faculty members’ desire to provide meaningful, timely feedback and a perceived lack of time to do so. Indeed, previous evidence suggests that faculty are currently underprepared for their role in workplace-based assessment.¹⁴ Therefore, in order to respond to the anticipated increased demands on time associated with CBD implementation, faculty will need to engage in development opportunities to help them hone the skill of providing effective feedback while simultaneously managing competing demands.¹⁴

Our survey results demonstrate that the majority of faculty and learners reported a need for at least a moderate amount of training and/or education to help develop their skills within the CBD framework. Participants identified “several educational sessions or workshops” as the best format to help further develop these skills. Hanley and colleagues emphasized the importance of a clear communication strategy in transitioning to CBD, including a constellation of timely and recurring approaches to the delivery of information.¹⁵ Such approaches can include grand rounds, news bulletins, faculty workshops on assessment, resident information sessions, and person-to-person discussions. These researchers described communication as being essential to the uptake of CBD, especially amongst late adopters (those that are skeptical of change).^14,15 In the present study, both faculty and learners agreed that the most useful resource that the department could provide to facilitate the transition towards CBD would be faculty training. The need for faculty training around the theory and rationale behind CBD, as well as effective assessment methods, has also been highlighted in other studies.^14,16 Since some programs have found that commitment to faculty development in CBME is low outside the early adopters (those who are open to the adoption of new ideas), it has been postulated that incentivization of these opportunities may be necessary.¹⁴ The presence of CBME champions serving as educational mentors for other faculty members is also noted to be important for the implementation of this new educational paradigm.^15,17 In highlighting the shortcomings of the traditional model of training and promoting the successes to-date associated with CBME, these educational leaders may be able to help facilitate adoption of this model by their colleagues.¹⁵

The brief, one-hour educational intervention employed in the current research was rated favorably by both faculty and learners. The qualitative comments provided by faculty further supported that faculty do not necessarily desire more information on the theory and principles of CBME and CBD, but rather that they require concrete, tangible educational interventions to help facilitate their transition to CBD (i.e., take-home and/or hand-out materials). Learners, on the other hand, reported the opposite; that is, learners’ qualitative comments reflected a need for more information concerning the principles of CBME and CBD. Overall, the qualitative findings converged with the quantitative findings for both faculty and learners, and the results included specific recommendations on how to optimize future educational interventions aimed at promoting knowledge of CBME and CBD.

Strengths and Limitations

A particular strength of the current research was the high response rate (79.8%). Furthermore, our sample comprised a diverse group of faculty and learners in various stages of clinical and educational experience. Finally, this research may inform procedures to facilitate the implementation of future national policy shifts in medical education.¹⁸ There are, however, several limitations to the current research. First, the educational intervention was brief and may not have allowed sufficient time for changes in perceptions and attitudes to be recognized by participants; however, the research design used in the current study did provide a unique opportunity to evaluate the effectiveness of a brief intervention on perceptions and attitudes. Second, the education session aimed to educate faculty and learners about CBME and CBD. As such, the content of this session was not exclusively comprised of CBD-specific material. Third, the sample consisted of participants with prior interest in learning about the principles of CBME and CBD, so self-selection bias was present and may limit the generalizability of the findings. Finally, while our sample comprised a majority of psychiatrists, clinical supervisors, medical teachers, and psychiatry residents (see Table 1), there were several other professionals included in the study that may experience less direct implications of the transition to CBD, which may have influenced their responses.

Conclusion

The majority of faculty and learners reported that the single education session was at least somewhat effective at improving their knowledge of, and preparedness for, psychiatry’s transition to CBD. Both groups identified a need for more educational interventions pertaining to CBD to address shortcomings in their preparedness. Strategies to facilitate a successful transition should include a clear, ongoing communication plan, identifying champions to promote CBME and CBD, and incentivizing faculty development opportunities. Indeed, these approaches, in addition to our brief education intervention, may be used during future policy changes in medical education to promote successful outcomes for all stakeholders.