Developing and Mapping Entrustable Professional Activities with Saudi Meds Competency Framework: A Consensus Study

Introduction

Medical students as future healthcare providers hold immense responsibility to provide safe and competent patient care due to increased societal, social and institutional accountabilities. In pursuit of competence, medical education is undergoing a transformative change towards competency-based medical education (CBME), which is becoming a resurgent paradigm of educational theory and practice.¹ CBME is an evidence-based approach to preparing physicians for practice that possess the desired knowledge, skills and attitudes outlined by the careful consideration of societal and patient needs.²

The fundamental ideology of CBME is to develop standards that focus more on the practical aspects of medicine rather than mere gain of knowledge. Furthermore, it urges on systematically testing the competence of medical graduates throughout their training.³ In the past decade, many competency frameworks have been designed to guide the curricular design of undergraduate training programs. Some classical examples include Can-MEDS,⁴ outcome project,⁵ Scottish Doctor,⁶ and the Netherlands national framework.⁷ In line with competency-based training models, a recent development is the Saudi Meds competency framework⁸ designed for Saudi undergraduate medical programs. This framework was designed in response to the national “Saudi Future Doctor” vision⁹ and is serving as a benchmark in the national implementation program “Saudi Medical Education Directives-Ministry of Education, Saudi Arabia.”¹⁰ The resulting framework outlines six overarching competency domains (i.e., patient care, scientific approach, research and scholarship, professionalism, communication and collaboration, and community-oriented practice) and 30 associated competencies that are expected of medical graduates and reflect the principles of professional medical practice in Saudi Arabia.⁸ The essential purpose of designing this framework was not to use it as a unified national curriculum, but as a national framework that can ensure equivalent standards across all Saudi undergraduate training programs.

Although this framework is holistic in design, a major issue with this framework is the suboptimal translation of theoretical competencies into practice. Entrustable professional activities (EPAs) have been proposed to operationalize competencies into practice and eventually filling the longstanding theory to practice gap.^7,11,12 EPAs have been defined as:

Core units of professional practice that can be fully entrusted to a trainee as soon as he or she has demonstrated the necessary competence to execute the activity unsupervised.¹³

EPAs serve as a valuable tool to assess the competence of a trainee through entrustment decisions that define learners’ level of expertise and autonomy. While competencies are descriptors of individuals, EPAs are units of professional activity or work that can be observed and entrusted to a trainee in a clinical workplace.^14,15 An EPA usually requires proficiency in multiple competencies simultaneously and provides a more structured and realistic approach of competence assessment.¹⁶

Many scholars advocate that the key principles underpinning EPAs (i.e., workplace training and entrustment) are generalizable and an EPA-based curriculum can help in creating a framework of continuous and transparent assessment of undergraduate trainees, starting from first year of medical school till graduation.^17–20 With growing acceptance and suitability of EPAs-based undergraduate training worldwide, there has been a call to develop a national EPA framework to streamline Saudi undergraduate medical education training programs.²¹ This study aimed to develop a national consensus on a generic, comprehensive and region-specific set of EPAs and map them with the existing national Saudi Meds competency framework. Locally contextualized EPAs linked with Saudi Meds framework will help the curriculum planners and medical educators in configuring the competency-based undergraduate training programs by increasing the focus of the training on actual professional activities and progress evaluation of our trainees.²¹

Materials and Methods

A three phased approach was used to develop EPAs (Phase 1), build a national consensus on developed EPAs (Phase 2) and map them with Saudi Meds competency framework (Phase 3). Below we provide a description of each phase.

Phase 1: EPA Identification and Development

In the first phase, nominal group technique was used to outline an initial list of EPAs. We recruited expert medical educators with a national profile in undergraduate curriculum development and CBME. Prior to the expert meeting, a preliminary list of EPAs was drafted which were drawn from a robust literature review of current EPA frameworks designed for undergraduate medical education. After devising a potential list of EPAs and recruiting participants, a face-to-face meeting of the working group was convened during Saudi International Medical Education Conference (SIMEC). During the meeting, each EPA was briefly described to the experts in order to provide context and scope. Participants were then asked to brainstorm, critically analyze, and identify the professional activities which are relevant to the local undergraduate medical education. The stages of our nominal group were: silent generation, round robin, clarification and voting (ranking).

Phase 2: EPA Validation Process

The aim of Phase 2 was to validate the initial draft of EPAs (resulted from Phase 1) using modified Delphi technique. In the first Delphi round, Phase 1 participants were approached again via online questionnaire using Question-Pro^® (Survey Analytics LLC, Beaverton, Oregon, USA) survey tool. This time, we asked the experts to rate each EPA for clarity and relevance using a dichotomous (Yes/No) scale. We then calculated the recorded scores for clarity and relevance by adding the number of positive (Yes) responses. We used a cut-off 80% positive responses to keep or discard an EPA. For instance, we retained an EPA that received 80% or more positive responses, whereas an EPA was discarded if the positive response was less than 80%. The results of first Delphi round led to a revised list of EPAs that was used in second Delphi round.

In the second Delphi round, we targeted national academic leaders across the country who were involved in development and implementation of medical curricula in their respective institutions. Participants were purposively recruited using a snowball sampling technique. We then sent an email to the recruited participants in which we provided a brief study description and requested them to complete the electronic survey, which was active for six weeks. Participants were asked to rate each EPA according to its level of importance using a 5-point Likert scale (1 = not important at all; 2 = somewhat important; 3 = important; 4 = very important; 5 = extremely important). We did not use a neutral response (i.e., do not know) to avoid complacency in participant responses. We also asked them to suggest additional EPAs using open ended questions at the end of the online questionnaire. We decided to include any suggested EPA provided that it is an independent, discrete, observable and measurable task with a clear beginning and end.¹³ The questionnaire also included items requesting participants’ sociodemographic and academic information.

Data Analysis

We used a classical test theory-based item analysis to determine corrected item-total correlation and internal consistency (Cronbach’s alpha) for the full list of EPAs. Classical test theory was chosen because it is a reasonable reliability and validity testing approach when the sample size is relatively small.²² We calculated corrected item-total correlation for each EPA and decided to keep the EPAs with correlation values between 0.3 to 0.8. We chose these values because correlation higher than 0.8 suggests redundancy and lower than 0.3 suggests noise (measurement error) in true score estimation. Cronbach’s alpha was calculated to evaluate the internal consistency of each retained EPA with a desired value in the range of 0.7 to 0.9. Descriptive statistics (means and standard deviations) were calculated for each retained EPA. All statistical analyses were performed using the Statistical Package for Social Sciences (SPSS) version 24.0.

Phase 3: EPAs and Competencies Mapping

After developing consensus on the list of EPAs, an online mapping link was generated in which all competency domains of the Saudi Meds framework were listed against each EPA. This link was shared with the same group of national academic leaders who participated in the EPAs validation process. We asked the participants to decide if a certain competency domain is required or not to perform each EPA using a dichotomous (Yes/No) scale. If more than 70% of participants voted “Yes” then that competency domain was considered “Essential” to perform an EPA. If 30–70% voted then it was considered “Important” to perform an EPA, and If <30% voted then it was considered “Relevant” to perform an EPA.

Results

Phase 1: EPA Identification and Development

A group of 15 national medical education experts participated in Phase 1. They were presented with a list of 21 EPAs which was generated through an iterative literature review of existing EPA frameworks. After critical review, brainstorming and group discussions, 14 EPAs were finalized that were used in first Delphi round.

Phase 2: EPA Validation Process

Same working group of 15 national expert medical educationists participated again in the first Delphi round, and a consensus was achieved on 10 out of 14 EPAs. Participants did not suggest any further EPAs at this level. In the second Delphi round, 186 national academic leaders were invited, of which 109 (58.6%) participants from 23 medical schools completed the online survey. Most participants were Saudis (52%), males (61%) and had more than 10-year experience in medical education (46%). Table 1 provides demographic information of the participants.

Table 1 Participant Demographics of Second Delphi Round Survey

The overall reliability (Cronbach’s Alpha) of 10 EPAs was 0.814. The mean values of retained 10 EPAs ranged from 4.06 to 4.83 with standard deviation ranging ±0.42 – ±0.94. The item-total correlation ranged from 0.341 to 0.642. Table 2 provides descriptive and item wise reliability statistics of core EPAs. Participants also suggested some new EPAs but none of them met our inclusion criteria. The suggested EPAs were either micro level learning outcomes, competencies, or a part of an existing EPA.

Table 2 Descriptive and Item Wise Reliability Statistics of Core EPAs

Phase 3: EPAs and Competencies Mapping

Of 109 participants, 94 (retention rate = 86.23%) participated in Phase 3. Two domains, “Patient care” and “Communication and collaboration”, received highest value in majority of the EPAs. “Patient care” was found to be essential for seven EPAs and “Communication and collaboration” was found to be essential for five EPAs. The domain “Research and Scholarship” was found to be least relevant and was not considered important or essential for any of the EPAs (< 30% votes). Table 3 provides a complete mapping of validated EPAs with Saudi Meds competency domains.

Table 3 Mapping of EPAs with Saudi Meds Competency Domains

Discussion

This study aimed at developing a national consensus on generic, comprehensive and region-specific EPAs and mapping them with Saudi Meds competency framework in order to facilitate competency-based curricular reforms in local medical schools. We achieved a national consensus on the enlisted 10 EPAs and their mapping with competency domains through expert educationists and academic leaders from almost all medical schools within the kingdom. Majority of respondents 95 (87%) advocated the potential value of developing national EPAs for undergraduate MBBS program. This advocacy is in line with many international initiatives that have incorporated EPAs in undergraduate clinical curriculum.^20,23,24 We used multiple methodological and analytical approaches including nominal group and modified Delphi techniques, and classical test theory based item analysis in order to make the validation process more robust. The final list of EPAs achieved desired psychometric properties that provide high reliability and validity evidence.

Expert medical educationists did not highlight any further activities in first Delphi round because they were involved in generating the initial list of EPAs in Phase 1 of the study. Some EPAs were suggested by academic leaders in second Delphi round but were not included as these suggestions did not meet our inclusion criteria. This is common finding in other EPA validation studies.²⁵ For any activity to qualify as an EPA, it has be an independent, discrete, observable and measurable task with a clear beginning and end.^13–16

EPA7 “Participate in health quality improvement initiatives” received variable importance with minimum mean score. This finding could be seen as a reflection of the current practices in Saudi healthcare system where health quality improvement is executed by either national regulatory bodies or specific departments with designated roles. The culture of health quality assurance and development as being the responsibility of undergraduate students is yet to be recognized by the Saudi medical education and practice systems. Second lowest mean was observed for EPA 8 “Perform general procedures of a physician”, which is probably because undergraduate trainees are mostly not allowed to perform procedures as part of patient safety protocols. Although prioritizing patient safety is commendable, restricting students from performing procedures might hinder in their skills development. Alternative solutions such as simulation-based training could serve as a viable approach to overcome this challenge.²⁶ Two EPAs (“participating in health quality improvement initiatives” and “Educating patients on disease management, health promotion and preventive medicine”) have a strong representation of the “Research and scholarship” domain as most medical colleges in the kingdom acknowledge research as a main goal in their vision and mission statements. Another driver for this recognition is that the regulatory body (Saudi Commission for Health Specialty) urges each graduate to have at least one publication for enrollment in any postgraduate training program.

The EPAs and their mapping with Saudi Meds framework hold multiple potential implications to further CBME in Saudi Arabia. The Saudi Meds competency framework is already serving as a benchmark in undergraduate medical education programs. Introducing EPAs will provide means to better operationalize Saudi Meds competency framework by defining essential activities and their required competencies that students could be entrusted with. When systematically operationalized, the entrustment decisions are known to provide structured feedback opportunities to the trainees by pinpointing exactly what is needed for them to achieve entrustment.^27–29 Additionally, designing medical curricula around EPAs will help in trimming the unnecessary science jargon, bridging the gap between theory and practice, and directing the assessment to the most relevant competencies. Incorporating EPAs in a curriculum will essentially direct teaching and training activities towards real-life professional activities and will assist in bridging the gap between the planned and actual curriculum. Finally, as the Saudi clinical postgraduate programs led by the Saudi Commission for Health Specialties (SCFHS) are already in the process of incorporating EPAs in the postgraduate curricula, EPA-based undergraduate training might help in creating a true continuum of medical education.³⁰

This study is not without limitations despite that we used robust methods to achieve our objectives. We recruited expert medical educationists using purposive sampling technique that might have led to an unconscious bias. Although we targeted highly expert group of academic leaders in the second Delphi round, it is quite possible that some of them might not be familiar with the concept of EPAs. An orientation workshop on EPAs, their definition and purpose could have helped us in overcoming this limitation. However, this study was conducted in the early days of COVID-19 pandemic when online workshops were not a norm and it was difficult for us to organize such event in an uncertain situation. Finally, this consensus study does not provide specifications (i.e., context, expected work description, required resources for entrustment decisions) for the enlisted EPAs. We call academic leaders and researchers to address this crucial gap by designing contextually relevant specifications for each EPA that can be used to facilitate EPAs-based undergraduate training programs.

Conclusion

This study results in a national consensus on generic, comprehensive and region-specific EPAs that have been mapped with Saudi Meds competency framework. We achieved a national consensus through expert educationists and academic leaders from almost all medical schools within the kingdom. Our study is the first step in the direction of facilitating EPA-based curricular reforms in local medical schools. We have a long road ahead of us in order to successfully implement these EPAs in Saudi undergraduate medical education.