The Ethiopian Anesthetist Licensing Examination and Associated Improvement in in-School Student Performance: A Retrospective Study

Introduction

In 2015, the Lancet Commission reported that an estimated five billion people worldwide desperately need essential surgical care. Access is worst in low and middle-income countries (LMICs), where nine out of ten people cannot access basic surgical care.¹ Major physician specialist surgical workforce shortages, compounded by uneven distribution of the existing workforce, disproportionately affect LMICs.^2–4 This necessitates the production of a non-physician workforce through task-sharing in order to have cadres with shorter professional training provide services with or without supervision.⁵

In Africa, 38 out of 51 surveyed countries were previously found to implement anesthesia-related task-sharing using widely varying training standards, qualifications and scopes of practice.⁶ Ethiopia too prioritized task-sharing and increased the anesthesia workforce through the training of associate clinician anesthetists, simply referred to as ‘anesthetists’ in the country (therefore, this term will be used in the remainder of this article).^7–9 However, simply training more health professionals may not be the answer to the vexing issue of improving access to safe essential surgical care.¹⁰ Studies conducted in Ethiopia have revealed that some graduated anesthetists may fall short of meeting competency requirements despite completing their education programs.^11,12 In addition, an increase in perioperative medical errors attributed to anesthesia practice has been reported in recent years.¹³

Nearly a decade ago, the World Health Organization issued guidelines to transform the health workforce and improve health outcomes. One of the five key recommendations was the need to establish sustainable workforce regulation mechanisms through program accreditation and workforce licensing.¹⁰ Many countries have developed a licensing examination system to assess and monitor the inflow of newly licensed practitioners. These exams are designed and administered in different formats, assessing cognitive and psychomotor skills in single or multiple steps. The cognitive assessment formats mostly emphasize the integration of information across basic and clinical science disciplines and their application in clinical scenarios.^14,15

Exam validity is considered the most essential and critical criterion of the national licensing examination, without which the interpretation of results will be less meaningful and useful.^16,17 The validation process requires the systematic synthesis of relevant evidence to support the exam’s use.¹⁸ Given that the benefits of exams should outweigh their costs and downsides, data on exam consequences, though notoriously difficult to collect, are the most fundamental source of validity evidence.^18–20 In this regard, assessing improvements in in-school student performance (eg, academic performance differences between cohorts that took the exam and those that did not) may provide validity evidence for NLEs, if these were to demonstrate a positive impact of the NLE on the quality of pre-service education.¹⁹

Some evidence from high-income countries shows that students with better in-school academic performances perform better in NLEs, but not that NLE implementation improves students’ in-school performance.^15,21 Published evidence assessing the associated changes in in-school student performance in LMICs following NLEs with non-physician groups is nonexistent. Previously, we qualitatively studied changes to anesthesia education due to the NLE. We found that student motivation, faculty performance, and learning and assessment practices have improved with clinically relevant written exams practiced regularly to assess higher-order cognitive thinking.^22,23 Identifying quantitative associations may strengthen the validity evidence base and help teaching programs identify performance measures that result in good and poor changes and make necessary curricular modifications.

Ethiopia piloted a national licensing examination (NLE) from 2015–2018 using a single-step multiple-choice question (MCQ)-based format as a means to ensure that only competent graduates join the practicing workforce. Since 2019, health workforce graduates from 13 undergraduate training programs, including anesthesia, are required to pass the NLE to become licensed practitioners.²⁴ The Ethiopian NLE is currently limited to MCQs for financial, logistical, and technical challenges; psychomotor skill assessments heavily relied on assessments conducted during the university training, which employed contemporary methods such as objective structured clinical examination (OSCE) and workplace-based assessments. The aim of this study was to compare students’ in-school academic performance between those who completed their training before and after NLE implementation at multiple teaching institutions.

Materials and Methods

Study Design

We employed a cross-sectional research design by retrospectively collecting the academic records of graduate anesthetists.

Study Population and Setting

This study used data from eight of the ten public universities that have had graduate anesthetists since the start of the anesthetist NLE. Four teaching institutions were first-generation (oldest), three were second-generation, and one was third-generation (newest). Data could not be collected from the two remaining institutions due to conflict in the northern part of the country. Our study population was anesthetists who had completed their baccalaureate anesthesia training at the eight teaching institutions between 2011 and 2021. Accordingly, academic records of 1493 eligible anesthetists were gathered. Students who did not complete the entire pre-service training or had incomplete academic records were excluded.

Training Program Description

The baccalaureate anesthesia training program accepts trainees from high school and nursing backgrounds and trains them for four and three years, respectively. A typical program for high school graduates is composed of one-year biomedical science studies, a one-year core clerkship (surgery, gynecology and obstetrics, internal medicine, and pediatrics), and two years of anesthesia specialty training (junior and senior year studies composed of theory and four practical attachments). Because of their prior qualification and experience, clinical nurses are exempted from the second-year core clerkship modules, reducing their total training duration by one year. Both groups of trainees in our study followed the same nationally standardized curriculum.

Data Source and Collection Procedure

Eight senior anesthesia instructors at the respective universities collected academic data digitally between September 20, 2021, and January 30, 2022, from university registrar offices. In-school academic performance was measured using yearly GPAs and the cumulative GPA (CGPA). GPAs are yearly unweighted composite scores of all module grades assigned for the academic year. Individual module grades are computed out of 100%, and this total score is converted to a four-point scale and recorded on student transcripts. Multiple assessment methods are employed in each module to assess all three learning domains (knowledge, skill, and attitude). Written exams in the form of context-based MCQs and structured essays, structured oral exams, viva, projects, and assignments are commonly employed knowledge assessment methods. Aside from traditional long and short exams, skill (and attitude) assessment methods for clinical-oriented courses and modules include OSCE and a variety of workplace-based assessments such as DOPS, Mini-CEX, and CBDs. The CGPA is a composite score of the yearly GPAs.

We collected the yearly GPAs (Y1 GPA, Y2 GPA, Y3 GPA, and Y4 GPA) and the CGPA with a 4-point scoring system for our study. To ensure data quality, academic records were collected by trained data collectors and cross-checked with data from respective programs, and data quality was regularly checked after submission.

Data Analysis

Academic data were cleaned and separated into two sets: before NLE (2011–2018) and after NLE implementation (2019–2021). Years 1 to 4 GPAs and CGPA were used to compare in-school student performance differences between the groups. Due to the non-normal data distribution, the Kruskal–Wallis and Mann–Whitney U-tests were used to compare score differences, while the Benjamini-Hochberg method²⁵ was used to adjust for multiple comparisons. The results are presented in tables as median (Md), interquartile range (IQR), and 95% confidence interval (CI) for median differences. Also, Cohen’s r values were calculated to compare the effect size for the comparison between two medians and interpreted using Cohen (1988) criteria.²⁶ All analyses were performed using SPSS ver27 (IBM Corp.).

Results

Characteristics of Study Participants

Academic records of 1493 anesthetists, including 932 who graduated before and 561 after the NLE implementation, were analyzed, accounting for 96.8% of all enumerated anesthetists. Study participants were predominantly male (67.2%), of relatively young age (80.8% aged 17–22 years at enrollment in to their studies), had entered directly from high school (83.4%), and were enrolled in a first-generation university (67.6%).

Differences in Student Academic Performances

A Mann–Whitney U-test revealed a statistically significant difference in first-generation university graduates’ median GPAs and CGPAs, with Y1GPA, Y2GPA, and CGPA exhibiting medium to large effect sizes (r≥0.5, p<0.005). However, after adjusting for multiple comparisons, there were no statistically significant differences in the scores of second and third-generation university graduates (Table 1).

Table 1 Differences in Anesthesia Students GPAs Between the Before and After NLE Groups by University Generation, Ethiopia, 2011–2021, (n=1493)

On the other hand, the significant performance gaps in Y1GPA before the NLE in favor of third-generation students (Md 3rd generation = 3.48; Md 1st generation = 3.09; performance gap = 0.41; 95% CI = 0.22, 0.60; r=0.42; p <0.005) disappeared after NLE implementation, and even first-generation students significantly outperformed the second generation students in similar performance measure (Md 2nd generation = 3.10; Md 1st generation = 3.36; gap = −0.25; 95% CI = −0.35, −0.15; r=0.75; p <0.005). Besides, after NLE implementation, first-generation students persistently outperformed the younger-generation students on the rest academic measures (Table 2).

Table 2 Differences in Anesthesia Students GPAs Among the Three Generation Students by the Before and After-NLE Status, Ethiopia, 2011–2021, (n=1493)

A sub-group analysis by entry behavior showed that high school graduates in the NLE group showed considerable positive differences in their performance scores with small to moderate effect sizes (r =0.27–0.46), except for Y4GPA. Nurse entrants, on the contrary, had a statistically significant difference in their Y2GPA scores only, but with an actual drop in performance following NLE implementation (Md before = 3.43, n=167; Md after = 3.14, n=81; median difference = −0.31; 95% CI = (−0.43, −0.19); r = 0.51; p<0.005) (Table 3).

Table 3 Differences in Anesthesia Students GPAs Between the Before and After NLE Groups by Entry Behavior, Ethiopia, 2011–2021, (n=1493)

Comparing high school and nurse entrants also revealed that all performance differences observed before NLE in favor of nurses had vanished, and even high school entrants performed better in Y1GPA after NLE (Md high school entrant = 3.30; Md nurses = 3.02; median difference = −0.20; 95% CI, −0.31, −0.07; r = 0.24; p<0.005 (Table 4).

Table 4 Differences in Anesthesia Students’ GPAs Between High School and Nurse Entrants Among the Before and After-NLE Groups, Ethiopia, 2011–2021, (n=1493)

Although positive gender-based performance differences were found in all performance measures between the before and after NLE groups, none were statistically significant (table not shown). On the other hand, the significant gender gaps observed on Y4GPA before the NLE in favor of males (Md males = 3.47; Md females = 3.27; gender gap = 0.19; 95% CI = 0.12–0.26; r=0.66; p <0.005) disappeared after NLE implementation (Md males = 3.51; Md females = 3.39; gap = 0.10; 95% CI = 0.01–0.18; r=0.17; p >0.05). Although the significant gender gaps in Y1GPA, Y3GPA, and CGPA observed before NLE favoring males also narrowed, all these gaps persisted with strong large effect sizes after NLE implementation (Table 5).

Table 5 Differences in Anesthesia Students GPAs Between Male and Females Among the Before and After-NLE Groups, Ethiopia, 2011–2021

Discussion

This study provides new consequence validity evidence on the associated improvement in in-school student performance following the implementation of NLE. Overall, modest in-school student performance changes were observed after NLE, with high school entrants and those attending their study in first-generation universities showing relevant improvements.

In this study, we found a positive shift in most student performance measures among first-generation (oldest) universities following NLE implementation. However, there was no change in student performance after NLE among the newest (second and third) generation universities, which produce more than three-quarters of the country’s yearly graduate anesthetists. This could be attributed to better-quality students enrolling in first-generation universities, as well as challenges in improving educational quality in newer generation universities, such as critical shortages of educational resources, shortage of qualified faculty and lower staff profiles (oldest universities require a second degree as a mandatory requirement, while first-degree holders are still involved in teaching in newest generations), and limited student clinical exposure due to fewer clinical practice sites and preceptors.^9,12,27

The overall positive student performance changes following the NLE were also confined to high school entrants. In contrast, nurse students experienced a statistically significant decrease in Y2GPA, while the rest of the measurements also decreased but not significant. Besides, all performance differences observed before NLE in favor of nurses disappeared. Even high school entrants performed better in one performance parameter (Y1GPA) after the NLE. A pre-post competency assessment study in Ethiopia found similar results, with anesthetist competence scores significantly different based on entry behavior.^9,12 Studies in other countries found no difference in the competence of graduates who followed different paths to obtain the same qualifications.²⁸ The persistent group differences reported in this study could be attributed to smaller sizes of nurse entrants in our study (17.9% of the before and 14.4% of the after) or differences in quality of education.

We found no significant gender-based differences in student performance between the before and after NLE groups. Besides, gender gaps observed before NLE in favor of males narrowed after NLE implementation. Our study findings were consistent with a government report²⁹ and the national education development roadmap,³⁰ which stated that gender gaps in the performance of higher education students had recently improved. The improvements observed in our study could be attributed to the strengthened institutional gender-responsive interventions.⁹ However, the sustained gender disparity in four of the five performance measures favoring males is alarming. A program evaluation study in Ethiopia found contradicting results, with anesthetist competence scores differences based on gender bridged after program interventions.^9,11 The persistent group differences reported in this study could be attributed to differences in study methodology, as the current study used GPAs, whereas, the previous studies employed a one-off objective structured clinical examination (OSCE).

Another intriguing finding from this study was the decrease in in-school performance differences (between the before and after NLE groups) as the training years progressed. Given that the NLE focuses primarily on the final two years of anesthesia specialty content, it’s inevitable for anesthesia schools to pay due attention to these training years to prepare their graduates for the exam.^22,23 These senior-year-centered educational reforms, characterized primarily by repeated assessments using different methods, may have made in-school assessments more stringent, lowering senior students’ performance after NLE. The rising stress and workload on senior-year students due to heightened expectations from programs to increase institutional NLE pass rates could also contribute to the decreasing performance differences. Furthermore, increased efforts by senior students to prepare for the comprehensive in-school exit exam and the subsequent NLE may contribute to lower scores by consuming more of their time and distracting their focus.

Conclusion

In conclusion, the current study will contribute to the body of validity evidence for the Ethiopian anesthetist licensing examination. The results showed that the Ethiopian anesthetist NLE is associated with modest improvements in in-school student performance. However, performance improvements were mostly positive and statistically significant only for those entering directly from high school and those attending first-generation university program. Performance stagnation and drops among nurses and newest generation university students warrant examination of education quality and systems to align institutional assessment with NLE. Furthermore, the small effect sizes and decreasing performance differences (as the training year progressed) reinforced the need to strengthen junior-centered educational interventions to improve graduates’ competence and licensing exam pass rates through the proper use of NLE data.

Recommendations

Given the rapidly increased number of anesthesia teaching institutions and newly trained anesthetists, a more stringent quality assurance system, including program accreditation that takes NLE pass rates into account, is needed. Younger generation universities can benefit from increased engagement in and orientation into NLE design, content (eg by sharing blueprints and sample items) and administration processes. Besides, nurse entrant training programs can benefit from proactive sharing of experiences with high school entrant programs, effective use of NLE data to inform their training, and strengthening overall systems for educational quality improvement and assurance. Collectively, we strongly believe the time has come to scale up the infantile voluntary accreditation efforts and standardize education quality through mandatory accreditation that takes NLE pass rates into account. In this regard, establishing a fully autonomous regulatory authority is a fundamental milestone.

The sustained gender disparity favoring males calls for strengthening gender-responsive institutional interventions primarily targeting junior-year students, thereby contributing to the 2030 sustainable development goals 4 and 5 on inclusive, equitable and gender-sensitive quality lifelong learning.³¹ Standardizing institutional exit exams, introducing small-scale clinically relevant summative assessments, such as year-based qualification exams, and incorporating some form of skill assessments (eg OSCE) in the NLE may help distribute the burden that caused academic performance to decline as the training year progressed and promote comprehensive learning.

Finally, given the growing concerns about the quality of patient care and safety¹³ that question the competency of graduates and the appropriateness of the NLE’s decision to filter less competent graduates, further research into the relationship between licensing examination scores and in-school academic performance and patient outcomes is recommended.

Strengths and Limitations

Strengths of the Study

This study has a number of strengths. It involved all eligible anesthetist teaching institutions obtaining nationally representative sample. Secondly, a relatively large-scale data on student performance before and after NLE implementation was collected, allowing comparisons of changes in student performance between groups over a decade. Finally, multiple in-school performance measures were evaluated, allowing for triangulation.

Limitations of the Study

The lack of a control group was the most significant limitation of this study, limiting establishment of casual relationships. As a result, our study design may fail to distinguish between the NLE’s direct effects and any other effects caused by external factors.³² However, we conducted analyses in strata of potential confounders (gender, entry behavior and university generation), which by and large did not alter our outcomes. Another important limitation is that we only looked at anesthesia programs, so our findings may not be generalizable to other programs as learning and assessment contexts may differ. Finally, data were not collected from two institutions due to security concerns, but the authors believe that their context does not differ significantly from that of the institutions involved.