Exploring Perceptions of Competency-Based Medical Education in Undergraduate Medical Students and Faculty: A Program Evaluation

Introduction

Undergraduate medical programs have begun to adopt a competency framework, with the rationale that this may better prepare students for the competency-based learning and assessment style in clerkship and residency.¹ Postgraduate medical education has traditionally followed a time-based approach, which assumed that learners could acquire the skills necessary to be independent practitioners after participating in a defined number of cases or after completing a defined number of years.² However, a growing body of evidence is unveiling the pitfalls of this traditional time-based model, with limitations including a lack of interpersonal and professional soft skill development and an over-emphasis on knowledge rather than clinical skill advancement.³

Competency-based medical education (CBME) is an outcomes-based approach that emphasizes flexibility and learner-centeredness, where learner progression is dependent on achievement of competencies, rather than the amount of time that has passed.² The Royal College of Physicians and Surgeons of Canada began a nationwide shift to Competency by Design (CBD) in 2014 and competency frameworks have been incorporated into postgraduate medical education (PGME) programs.^4,5 Subsequently, in 2016, the Association of Faculties of Medicine of Canada (AFMC) launched a list of 12 Entrustable Professional Activities (EPAs), outlining the competency expectations to be achieved by all graduating medical students.⁶

Canadian medical schools are in the process of implementing CBME in undergraduate medical education (UME).⁷ UME programs hold the decision on how this transition to CBME would look for their respective programs. At our institution, undergraduate CBME was launched in September 2019, and involved implementation of new methods of education and assessment that aligned with the 12 EPAs established by the AFMC.⁸ The majority of the curricular changes have been implemented for Year 1 and 2 of pre-clerkship curriculum.

The Context, Input, Process, and Product (CIPP) model is commonly used for program evaluation in education.⁹ The Context component explores the conditions that spurred the curricular change. The Input component assesses the content of the curricular changes, as well as personnel involved in the transition. The Process component examines how the implementation process is currently going, and the Product component assesses the lessons learned from the curriculum change.

A 2013 study compared a competency-based curriculum with a former non-CBME curriculum and found no significant difference in clinical performance or perceived preparedness.¹⁰ However, this study was limited by a short study period within a single institution. Other studies since then have described their process of implementing undergraduate CBME, but without examination any prospective outcomes that have resulted from these curricular changes.^11–14 Other researchers have administered a new competency-informed medical course, which showed improved participant knowledge when comparing pre- and post-test results, but without change to the overall medical curriculum to CBME.^15,16 There were studies that interviewed medical students to determine important aspects of CBME implementation.^17,18 However, these studies were conducted with the aim of CBME curriculum development, rather than post-implementation program evaluation.

There is limited evidence exploring both students’ and faculty’s perception of CBME after its implementation in an undergraduate medical program. As such, the perceptions of CBME by undergraduate medical students and faculty requires exploration. The aim of this study is to conduct a program evaluation using the CIPP model, comparing medical students versus faculty on: (1) how they perceive competency-based medical education, (2) how they recognize competency framework implementation at a single institution, (3) and to elucidate perceived benefits and barriers to implementing CBME in undergraduate medicine. This program evaluation provides insight on the context behind implementation of undergraduate CBME, the process of implementation, and the lessons learned.

Methods

Phase 1 and 2: Context and Input

In the Context section, we described the rationale for the transition to a competency-based undergraduate medicine curriculum. Understanding the context for this curriculum change involved review of national and university documents and collaboration with curriculum experts.⁸ The Input section examined the content of the new CBME curriculum, as well as the committees involved in curriculum development.

Phase 3: Process

Survey Design

To gain formative feedback on how medical students and faculty perceive the new undergraduate CBME curriculum, we conducted a cross-sectional survey using Research Electronic Data Capture (REDCap) data collection platform. This study was approved by the Western University Health Sciences Research Ethics Board (IRB#119166). Participants’ informed consent was obtained for the collection and publication of both quantitative and qualitative anonymized survey responses. The survey was developed using a modified Delphi method from April to May 2021.⁷ The initial survey questions were developed by one person and was modeled based on a survey previously administered at our institution, which was developed by three educators with iterative feedback by five faculty experts.¹⁹ Feedback was collected from a panel of four expert faculty members and three medical students. A consensus of the final questionnaire items was established after two rounds of email correspondence, with the first round involving feedback from expert faculty, and the second round involving repeat feedback from the expert faculty after incorporating medical student feedback, as well as previous revisions. A consensus of the final questionnaire items was established after a final virtual video meeting.

The first section assessed perceptions of CBME and encompassed 13 questions comprising a 5-point Likert scale (5=Strongly agree) with the addition of a sixth response (“Not Sure”). Previous studies that evaluated faculty and residents’ perception of CBME in postgraduate programs commonly reported participant uncertainty pertaining to CBME.^19,20 Furthermore, “Don’t Know” responses have been shown to reflect a meaningful lack of knowledge when survey items assessed the perceived incidence of an event.²¹ Thus, we included the “Not Sure” response to capture participant uncertainty on CBME. The second section assessed CBME implementation and included four multiple-selection questions, with 6 to 11 nominal response options.

Phase 4: Product

To obtain the Product—lessons learned from the new curriculum—the last section of the survey assessed the benefits and challenges of CBME and encompassed 14 questions involving a 3-point scale. Respondents rated each prompt as either a “Benefit/Challenge”, “Not a Benefit/Challenge”, or the neutral response of “Not Sure”. Given that the benefits and challenges of CBME can either be a benefit, a challenge, or neither, we used a 3-point scale for this section. Two open-ended questions also elicited written responses on perceived benefits and challenges. We conducted a thematic analysis of medical students and faculty’s written responses.⁸ Two researchers (E.A.L and C.W.) independently reviewed the responses. The first researcher (E.A.L) coded the responses using inductive coding and accuracy was confirmed by the second researcher (C.W.). The codes were categorized into themes and discussion took place where discrepancies were noted until agreement was reached.

Reflexivity Statement

This research was conducted at Western University, Ontario, Canada, with the aim of evaluating medical students and faculty’s perception of the undergraduate CBME curriculum implemented at this institution. Two of the authors are medical students, both of whom are currently experiencing the new CBME curriculum. Two authors are faculty members involved in teaching and curriculum development in undergraduate medicine. As well, two authors are well-versed in both qualitative and quantitative research but are new to the area of CBME.

Study Population and Recruitment

Surveys were administered to medical students with graduating years from 2021–2025. Faculty involved in delivering lecture content, facilitating small group discussions, or teaching clinical skills sessions from the 2020–2021 school year were considered to be involved in undergraduate medical education. Participant recruitment entailed distribution of an anonymous REDCap survey link via email to both medical students and faculty. For medical students, a post to each year’s class Facebook group was made. A reminder email or Facebook post was sent one week and three weeks after the initial recruitment message. Consent was implied upon advancing past the letter of information and submitting the survey. The survey was administered and open for an 8-week period in October 2021.

Quantitative analysis was conducted to examine the differences in responses between students and faculty. Cronbach’s alpha assessed the reliability of the survey items in measuring each construct. An unpaired t-test assessed mean differences in prior familiarity with CBME. Responses were grouped into three categories (Agree/Strongly Agree, Neutral/Disagree/Strongly Disagree, and Not Sure). Fisher’s exact tests compared the incidence of responses between students and faculty. Benjamin-Hochberg Adjustments controlled for false discovery rates and due to the large number of comparisons, only statistically significant adjusted p-values were reported. Statistical analysis was performed using R (Version 4.1.2., Boston, USA.). A p-value of less than 0.05 was considered statistically significant.

Results

Context Evaluation

With the launch of the 12 EPAs for graduating medical students, the transition to an undergraduate CBME curriculum fulfilled evolving educational standards for Canadian medical schools.⁸ This involved a de-emphasis on didactic, time-based learning models, with a shift to self-directed and group-based learning. This change was also spurred by students’ reported need for earlier experiential learning, involving exposure to real-life patient care early in medical school training. This shift allowed for integration of education on body systems, social determinants, and public health. Furthermore, with an increased focus on measurable skills with frequent low-stakes feedback, this fulfilled increasing expectations from public and financial stakeholders to ensure medical school graduates’ competency.⁸ This in turn, aimed to benefit learners in their transition to PGME.

Input Evaluation

The new CBME curriculum introduced six new courses for year 1 and 2 of medical school at our institution, with an emphasis on integration of courses over time and across disciplines.⁸ Interactive large group sessions, independent learning, and case-based small group sessions were implemented. In year 1, a Foundations of Medicine course incorporated the clinical disciplines of hematology and infectious diseases with social determinants of health. In the latter half of year 1, as well as the first half of year 2, Principles of Medicine I and II were two courses which covered the remaining medical disciplines. In the latter half of year 2, a Transition to Clerkship course prepared students for clerkship through covering electronic health record use and advanced clinical decision making. Throughout year 1 and 2, there was also the Patient-Centered Clinical Methods course, as well as the Professionalism, Career, and Wellness course, which honed history taking/physical examination skills and resilience, respectively. During pre-clerkship, students also acquired early clinical exposure through being paired with a practicing family physician. There are leads for each course, each clinical topic, and for each development process. Development of the new CBME curriculum was led by the MD program Curriculum Committee, along with collaboration with the Curriculum Renewal Executive Committee and Quality Committee.⁸

Process Evaluation

Demographic and Survey Characteristics

The survey was distributed to 855 medical students and 435 faculty. There were 51 medical student respondents (40.8% male) and 50 faculty respondents (51% male), corresponding to a 6.0% and 11.5% response rate for students and faculty, respectively. Mean student age was 25.3 ±2.6 years and mean faculty age was 44.8 ±12.0 years. Compared to medical students, faculty had a significantly greater self reported prior familiarity with CBME (p<0.05).

Perception of CBME and Its Current Implementation

Compared to faculty, students were significantly more likely to agree that they value CBME (p<0.05) and that CBME will allow them to become better future physicians (p<0.01) by improving the care they provide (p<0.01) (Table 1). Faculty were unsure of whether CBME involved regular feedback (p<0.01), whether feedback needed to be personalized to students’ strengths and weaknesses (p<0.05), and whether feedback should be structured based on a predetermined framework (p<0.01). Faculty were also unsure as to how CBME was currently implemented in lecture-based learning (p<0.05), small group learning (p<0.01), and clinical skills learning (p<0.05). Non-significant comparisons were reported in Appendix 1.

Table 1 Significant Findings When Comparing Medical Students and Faculty in Their Perception of CBME, Understanding of How CBME is Currently Implemented, and Perception of Benefits and Challenges to Implementing CBME in UME

Product Evaluation

Benefits and Challenges of Implementing CBME

There were no significant quantitative differences between students and faculty on perceived benefits (Table 1). For perceived challenges, significantly more faculty perceived an increased faculty time commitment to teaching (p<0.01) as a challenge.

Qualitative Analysis

For perceived benefits, two themes were expressed: enhanced learning and positive future impact. Both students and faculty reported improved structure and clarity of learning objectives, which enhanced learning by aligning students to competence goals (Table 2). Additionally, CBME had a positive future impact for students by aiding the transition from pre-clerkship to clerkship in medical school. For faculty, CBME was perceived to improve the transition from medical school to residency. These benefits were attributable to earlier clinical exposure to patients, as well as to enriched professionalism development through early rapport-building with healthcare team members.

Table 2 Representative Quotes for Themes Identified for Perceived Benefits and Perceived Challenges of Implementing CBME in UME

Two themes were identified for perceived challenges: an increased time commitment and logistical concerns (Table 2). Medical students and faculty noted an increased time commitment in completing student evaluation paperwork. These forms were perceived as poorly tailored to students’ learning needs and participants reflected on increasing the role of administration to facilitate this evaluation process. Respondents also reported logistical challenges with transitioning from a time-based to a competency-based curricular focus. Limited faculty time may hinder learner-centeredness by limiting the opportunities for faculty to observe students’ performance and assess their competency. Faculty emphasized the need for faculty education on undergraduate CBME initiatives, which may promote buy-in and improve accountability in teaching.

Discussion

This study explored students and faculty’s perception of undergraduate CBME after its implementation at our institution. Faculty were less certain about how CBME was currently implemented in the pre-clerkship curriculum. This suggests that faculty’s reservation towards CBME may be, in part, due to a lack of knowledge on how it is currently implemented. Faculty’s qualitative responses mirror these concerns and highlight the need for more faculty education regarding CBME, with a focus on logistics. This may improve faculty perceptions of CBME, reduce pushback, and aid in development of future CBME initiatives. These findings are unsurprising given that the transition to CBME in PGME faced similar challenges.^22–25

What differentiates undergraduate from postgraduate CBME is the design and implementation of a competency-based pre-clerkship curriculum for year 1 and 2 of medical school. Our work showed that CBME-informed curriculum provided clearer benchmarks for learning, improved identification of strengths and weaknesses relative to these benchmarks, and enhanced perception of preparedness for subsequent clerkship and residency. To increase the frequency and specificity of feedback, our institution implemented more frequent formative assessments, as well as the addition a “Strengths and Opportunities” report, outlining students’ specific areas for improvement. These features mirror the frequency and style of feedback implemented in postgraduate CBME. Despite this, most student respondents did not recognize CBME as involving frequent or personalized feedback. These findings likely stem from inadequate communication to students regarding the rationale for this method of feedback delivery. Nevertheless, both students and faculty reported improved preparedness for postgraduate education, which reflects the underlying benefit of undergraduate CBME that participants may not have been able to directly pinpoint.

Additionally, the implementation of early family medicine exposure at our institution has allowed students to apply their knowledge, gain an appreciation for the social determinants of health, and hone their professionalism with other healthcare professionals, all before entering clerkship. This is an aspect of medical education that is unique to undergraduate medicine and this study highlights the importance of the timing of clinical exposure—specifically early timing—to improve student preparedness for future training.

Faculty were more likely to be unsure when asked whether CBME involved feedback that was “given regularly” or “structured based on a predetermined framework”. A major component in medical education is the type, quantity, and frequency of feedback provided by teachers.²⁶ These results demonstrate the uncertainty that some faculty may have in terms of how best to provide feedback. Faculty at other institutions expressed a collective interest in topics related to improving feedback delivery and enacting evidence-based entrustment decisions.²⁴ Thus, guidance on how to give good feedback may aid the transition in undergraduate CBME.

Faculty recognized an “increased faculty time commitment to teaching” as a challenge for CBME implementation. Qualitative responses from medical students also emphasized the student time commitment required for completion of evaluations. Students favoured verbal feedback over written feedback because written assessments were viewed as generic and untimely. This perception seemed to be shared by PGME students at other institutions.²⁴ Certainly, time constraints have been frequently reported as a barrier to CBME implementation.²⁰

A major limitation in this program evaluation is the low response rate for both the student and faculty populations. Strategies were used to optimize the response rate, such as sending weekly reminders, as well as using both mass email and Facebook recruitment posts. The low response rates may predispose to nonresponse bias, which may limit the conclusions drawn. This is also a single-centre study with site-specific CBME implementation strategies, which may impact perceptions at our site but may have limited generalizability to other institutions. Nevertheless, this program evaluation sheds light on the rationale for the transition to undergraduate CBME, the changes that took place, and the insight gained from implementing CBME in undergraduate medicine. Future research that maximizes the response rate in a smaller randomized sample may improve the representativeness of the involved populations.

Summary of Lessons Learned

Continued Communication to Both Students and Faculty

Allocating time to inform students of curricular changes, how they are informed by CBME, and how they mirror postgraduate education, can enhance student engagement by allowing recognition of how their current learning prepares them for the future. Improving faculty education on how CBME is currently implemented in UME may increase faculty buy-in. Additional information on how to deliver prompt, specific, and actionable feedback to students may enhance faculty accountability in teaching. Incorporating scheduled time for faculty to observe students and complete evaluations may ease time constraints.²⁰ Incentivizing faculty through awards of recognition or through achievement of continuing professional development credits may also be considered.²⁷

Having a Closer Look at Formal Evaluation Forms

Educating medical students on the importance of formal written evaluations for tracking progress and providing evidence of competency may alleviate perception of these assessments as an administrative burden. Providing prompts for open-ended narrative evaluations may encourage faculty to give specific feedback. Previous authors suggest three prompts corresponding to what the student should 1) continue to do, 2) start to do, and 3) stop doing.²⁰ Providing faculty with feedback on their evaluation forms could also promote faculty accountability.

Opportunities for More Collaboration

Collaboration with administrative staff may improve the delivery and tracking of formal assessment forms, as well as enhance communication of new CBME initiatives to faculty and students. Integration of multiple sources of feedback—such as from allied healthcare professionals or patients—provides holistic feedback to students while reducing the administrative burden for staff.¹³

Conclusion

This program evaluation described the rationale for implementing undergraduate CBME, the content of the new curriculum, and the lessons learned from implementing CBME in undergraduate medicine. We identified opportunities for continued faculty development, re-appraisal of formal assessment forms, and enhanced administrative collaboration. Students and faculty agreed on the benefits of CBME in UME and we described actionable next steps to address the reported challenges. Future guidance from the AFMC regarding strategies to address implementation challenges may aid the transition to CBME in UME.