Effectiveness and Feasibility of Using Local Medical Universities for Capacity Building to Improve the Immunization Program in Ethiopia: Quasi-Experimental Study

Introduction

Ethiopia has a decentralized three-tier health system comprising primary, secondary and tertiary levels of care which have hierarchical supervisory mechanisms at each level. Appropriate and timely supportive supervision conducted on a regular basis is an essential element in the process of transforming the health sector. Hence, Ethiopia has invested heavily in health system strengthening which helped the country to remarkably achieve most of the MDG targets.^1,2

Despite major improvements in reducing under-five mortality in Ethiopia, vaccine preventable diseases are still an important cause of deaths. Currently, under the Sustainable Development Goals (SDGs) Ethiopia plans to end preventable deaths of newborns and children by implementing different strategies.³ Strengthening routine immunization is one of the child survival strategies in Ethiopia. Currently, 11 antigens are provided for child vaccination in Ethiopia at birth, 6 weeks, 10 weeks, 14 weeks, 9 months and 15 months.¹

Although the different immunization strategies adopted in Ethiopia have brought some improvement, the coverage level is still below the recommended 90% level for the sustained control of vaccine preventable deaths (VPDs).^1,2,4 According to an Ethiopian DHS 2016 report, the overall immunization coverage was 38.5% at national level and 45.8% in Amhara region, and there are substantial geographic and social inequities, high dropout rates and declining trends in some poor performing regions.⁵

A committed, confident and competent vaccination workforce is integral to ensuring a quality immunization service with high vaccination coverage.^6,7 A lack of experienced and high quality human resources can easily jeopardize the success of any health program including an immunization program.⁸ Reports from developing countries indicate that health workers seem to be overwhelmingly concerned with only raising immunization coverage.^8,9

The quality of immunization services is equally important for the achievement of the ultimate goal of the Extended Program of Immunization (EPI) and has been neglected.^8,9 To counter this, the Reach Every District (RED) approach was launched in 2003 that aims to build capacity of the district health facilities to address common obstacles to improving immunization services. This district planning and monitoring approach promoted by the World Health Organization is based on five operational approaches or components that all specifically aim at improving coverage in every district; planning and management of resources, reaching target populations, linking services with communities, supportive supervision and monitoring for action.^10–12

Supportive supervision is one of the five essential components of the WHO’s RED strategy, an operational approach to reach 90% coverage.¹² Globally evidence shows that supportive supervision provides critical support for the delivery of health services and has a positive impact on improving service quality and coverage.^8,12–14 National experience also indicates that onsite training has the most lasting effect on the quality of health services including immunization programs.¹⁵ However, one of the major lacunae observed in the implementation of the immunization program leading to low coverage and compromised quality is lack of supportive supervision by the supervisors in the health system.¹⁶

Currently, supervision is mostly crisis-oriented and supervisory visits are generally carried out in case a need arises and the feedback given is not problem-solving.¹⁴ Common barriers to effective supervision in developing countries include lack of supervisors’ skill and knowledge, lack of supervisors’ authority to reward or sanction performances and lack of resources and accountability in the health system.²

Ethiopia currently has 54 universities distributed across the country which have medical colleges with high levels of expertise. In this research, there was no published literature on direct engagement of local medical and health science colleges in building the capacity of health system staff through continuous supportive supervision practice to improve immunization programs in Ethiopia.

Hence, this study aimed to assess the effectiveness of capacity building and mentorship intervention by medical and health science colleges through the development of direct and indirect capacity building, mentorship, and community involvement to reach every child in each household in selected districts of northwest Ethiopia.

Objectives

The goal of this study was to assess the effectiveness and feasibility of medical and health science colleges’ mentorship and supervision (CBMP) interventions on improving the coverage and quality of immunization services, RED strategy implementation and improvements in capacity of health workers in selected districts of northwest, Ethiopia.

Materials and Methods

Study Setting

The study was conducted in Wogera and Dabat districts of Amhara region, northwest Ethiopia. Wogera district has an estimated total population of 278,942 of whom 37,769 are children under five and 8675 are infants. In terms of health system delivery Wogera district has 44 health posts, 10 health centers and one primary hospital. Dabat district has a total population of 145,509 of whom 24,680 are children under- five and 5676 are infants. In terms of health service delivery, Dabat district has 39 health posts and 6 health centers. The health facilities deliver an immunization service with static and outreach service delivery approaches.

Study Period

The study was conducted from October 1, 2018 up to May 30, 2019.

Study Design and Methods

The study employed a pre-post quasi experimental design. In addition, a qualitative study design was used to evaluate the implementation process of the CBMPI program.

Study Population

A cohort of healthcare providers responsible for the supervision and provision of immunization services was included from the selected 15 health facilities in both groups.

Intervention Package

The intervention package was developed mainly on the basis of using local medical university staff for capacity building. The intervention was employed by university research teams who have ample experience with the immunization program supervisory and mentorship activities. The supervisors and mentors were recruited from the medical and public health experts at the University of Gondar. The supervisors and mentors provided capacity building for the health workers at immunization service delivery points regularly every month for six consecutive months. The package included the following interventions.

Development of supportive supervision and mentorship guidelines and tools based on WHO endorsed checklists and guidelines.

Training supervisors on supportive supervision and mentoring based on updating supervisors on current policies, new immunization practices, techniques, and management skills. Training on how to coach, mentor, effectively communicate, and conduct performance planning.

Preparation of tools for supervision which include detailed instructions for conducting supervision and mentoring, namely, sequence for conducting supervision meetings, checklist for supervisory visits, self-assessment for providers, work planning action sheets, supervisor competencies, giving feedback, resolving conflict, and conducting difficult conversations.

Sample Size and Study Participants

Health workers working in the 30 health facilities (15 health facilities from intervention and 15 health facilities from control groups) were considered for this study. In addition, district health office heads, health center heads, health workers, health extension workers, supervisors from the selected district health offices and medical college staff who participated in the program were approached for the key informant interview.

Study Procedures and Instruments

In this quasi experimental study, we assessed the intervention effectiveness and implementation process of the CBMPI program. For effectiveness, we used RED programmatic outcome and immunization service delivery outcomes which were assessed at baseline and at the end of the intervention implementation. For the qualitative part, RE-AIM framework has been applied to evaluate the implementation process over time.

Data Collection

Primary data sources for the evaluation of the interventions included: (1) Interviews with immunization providers and supervisors; (2) Document review (checklists, filed notes, project documents) and (3) District administrative healthcare databases. Measurements were done at baseline and end line in both the intervention and control areas at individual and organizational levels.

Quantitative Data Collection

All the quantitative data were collected using a structured questionnaire. The data were collected by six independent data collectors in the pre-implementation and post-implementation phase of the study.

Qualitative

Semi-structured questionnaire was used for the key informant interviews which took place during the data collection period. Interviews were facilitated through the use of interview guides which were prepared in English and then translated into the local language and field tested away from the study sites. Interviews were digitally recorded for subsequent analysis and transcribed verbatim by an independent transcriptionist and verified for accuracy.

Document Review

A standardized checklist was developed and validated to evaluate the immunization programs.

Data Analysis

To assess effectiveness of immunization coverage and a change in RED strategy implementation, a difference in difference analysis (DID) was used to estimate the effect of the intervention implementation over time. For assessing effectiveness in improving capacity of health workers, chi square test was used. Stata 14 software was used for analysis and a significance level of 0.05 was considered statistically significant. All qualitative data sources, including interview transcripts and documents, were compiled and analyzed thematically.

Data Quality Management

The quality of the study output was ensured through employing piloted and validated data collection tools. A pilot test was done before the actual data collection period. We provided rigorous training for all study staff and data collectors. A standard operating procedure was developed for the data collection tool, interview protocols, and document review protocols. Continuous supervision was carried out during data collection to maintain the integrity of the collected data. We also established appropriate record keeping, documentation systems, and reporting mechanisms.

Operational Definitions

• Complete vaccination coverage: proportion of under 1-year-old infants vaccinated with all 11 vaccines as recommended by the national immunization schedule in Ethiopia.¹
• Quality of immunization services: Immunization services that consistently meet appropriate levels of potency, safety and efficacy of vaccines with standard cold chain management delivered by trained and technically competent health workers.^17,18
• Supportive supervision: A process that promotes quality at all levels of the health system by strengthening relationships within the system, focusing on the identification and resolution of problems, and helping to optimize the allocation of resources towards promoting high standards, teamwork, and better two-way communication.^16,19
• Capacity building: A process that improves the ability of a person, group, organization, or system to meet objectives or to perform better.²⁰

Ethical Considerations

This study obtained ethical approval from University of Gondar Institutional Ethical Review Board Ref. No: O/V/P/RCS/05/781/2018. Interventional program approval was also gained from the regional health bureau, the district health office, health centers and community health workers. Written informed consent was sought from study participants. The data collection procedure was anonymous and their privacy was also kept.

Results

Profile of Health Facilities

In this study a total of 30 health facilities were included from the 100 health facilities in the two districts. Among those health facilities included for this study, 15 (50%) were from Wogera district receiving the intervention and 15 (50%) were from Dabat district as the control group. Finally a total of 28 facilities (13 from the intervention group and 15 from the control group) were included for analysis. Two health facilities (one health center and one health post) were excluded from the intervention group due to security reasons. During the intervention period we were able to train 25 supervisors from the medical college and the district health office who conducted 65 mentorship activities (86.7% completion) in a six-month period, reaching 40 health workers in both health centers and health posts with a high rate of content fidelity measured monthly.

Profile of Key Informants

Key informant interviews were done with district health office head (1), CBMPI project supervisors (2), health center head (1), EPI focal individuals in health centers (2) and health extension workers from health posts (2). Among the key informants 5 were males and 3 were females. The average age of the respondents was 32 years ranging from minimum of 28 years to maximum of 47 years. The respondents had on average 6 years’ experience in EPI related programs which ranged from 1 year to 14 years.

RED Strategy Implementation Score for Component Indicators

Table 1 shows the mean score of each component indicator of the RED strategy in the study area before and after the implementation of the intervention package. The findings showed that the mean score for all five component indicators has increased after the implementation of the intervention package in the study area. The overall RED strategy implementation score was 2.71 (range: 0.86–4.1) during pre-intervention while the score increased to 4.36 (range: 3.39–4.99) post-intervention. The use of data for action with a mean score of 0.95 (range: 0.8–1) and community engagement with a mean score of 0.94 (range: 0.66–1) component indicators had the highest score with better implementation after the introduction of the CBMPI project (Table 1).

Table 1 RED Implementation Score for Component Indicators for RED Strategy During Pre and Post Intervention Implementation

Effect of the CBMPI Intervention on RED Implementation Strategy Components

The findings of the study revealed that the CBMPI intervention package has a statistically significant effect on implementation of planning management, data use for action and outreach components of the RED strategy. The difference in differences estimator is positive for planning and data use components which showed that health facilities in Wogera district receiving the intervention package have a statistically significant higher (0.3) implementation score than health facilities of Dabat district in the control group over time. The outreach component of the RED implementation strategy was also better implemented in the intervention group with a statistically significant higher (0.5) implementation score (P = 0.001). In general, the intervention package has a positive statistically significant effect on overall implementation of the RED strategy (P = 0.0001) (Table 2).

Table 2 The Effect of the CBMPI Intervention Package on RED Implementation Strategy Components

Effect of the CBMPI Intervention Package on Vaccination Service Delivery Outcomes

The difference in differences estimator is positive for Penta III coverage and shows that health facilities in Wogera district receiving the intervention package have a statistically significant higher (17.4%) Penta III coverage during 2018 to 2019 than health facilities of Dabat district in the control group over time. Similarly, the difference in differences estimator is positive for full vaccination coverage and shows that health facilities in Wogera district receiving the intervention package have a statistically significant higher (16.6%) full vaccination coverage during 2018 to 2019 than health facilities of Dabat district in the control group over time. On the other hand, the Penta drop-out rate has reduced on average by 2.6% in the intervention group compared with the control group during 2018 to 2019 over time (Table 3).

Table 3 The Effect of the CBMPI Intervention Package on Vaccination Service Delivery Outcomes

Knowledge of Health Workers on Cold Chain Management

In this study knowledge of health workers working in the immunization program of each health facility were asked about the correct methods of maintaining the cold chain system. Accordingly, after the implementation of the intervention package 46% of the respondents correctly explained the correct distance of an ice lined refrigerator (ILR) and deep freezer (DF) from the wall (P = 0.011) which showed a statistically significant difference as compared with pre--implementation (Table 4).

Table 4 Knowledge of Health Workers Pre- and Post-Intervention Implementation

Similarly, the knowledge of health workers on correct temperature log maintenance (P = 0.39), appropriate cabinet temperature for ILRs (P = 0.016) and the correct placement of diluents in an ILR (P = 0.011) were significantly improved after implementation of the CBMPI intervention package (Table 4).

Assessment of Skills and Practice of Health Workers During Child Immunization

Assessment of the skills and practice of health workers were assessed for five children at each health facility. Accordingly, provision of appropriate vaccines (P = 0.027), communication on key vaccination messages (P = 0.005) and safe disposal of used syringes/needles (P = 0.004) were performed with correct procedures after the implementation of the CBMPI intervention package. Meanwhile, administration of vaccines with the correct route (P = 0.141) and touching/recapping of needles by vaccinators did not show a statistically significant difference after the implementation of the intervention package (Table 5).

Table 5 Skills and Practice of Health Workers During Child Immunization Measured Pre- and Post-Intervention Implementation

Qualitative results Using RE-AIM Implementation Science Framework

The findings from the implementation science qualitative synthesis were presented using the RE-AIM framework as follows.

Reach

All the eight key informants were willing to participate in the CBMPI program and explained that the CBMPI program is reachable by all health facilities. One of the health center heads reported that

I believed that the CBMPI project will give us better support and improvement in our health facility’s immunization program which is actually reachable to our health posts and other health facilities.

Effectiveness

In terms of effectiveness, all the key informants have reported that the CBMPI project has brought overall change in improving the immunization program of the supervised health facilities. All the respondents mentioned that the CBMPI program had better support to solve previously existing basic problems in the EPI program of supervised health facilities. One of the health center EPI focal individuals reported that; “The CBMPI program brought significant change. There were changes in improving knowledge of health workers on cold chain system management especially in vaccine labelling management and ice pack use for vaccine carrier.” One health extension worker also mentioned that

With the help of CBMPI program supervisors, we were strictly educating mothers on anticipated vaccination side effects so that they will not drop from the vaccination schedules.

From the qualitative findings, data for action, presence of EPI monitoring chart and presence of defaulter tracing system were reported as having better achievement after the introduction of the CBMPI program. In addition, health workers from District and Health facilities identified outreach sessions as a positive contributor to immunization service delivery and the number and quality of outreach sessions has been improved with the CBMPI program.

Adoption

Regarding program acceptance, the majority of the key informants mentioned that adopting the new intervention package in the routine immunization program was acceptable and the commitment of health facility staff to accept the new program was encouraging. One of the supervisors mentioned that “the health workers were committed and supportive in implementing the CBMPI project.”

Implementation

All key informants explained that the CBMPI program had been implemented very well in a way that could bring change. Regarding the implementation of the intervention package as intended, one of the supervisors reported that

The mentorship was conducted very well. Mentorship brings change immediately. We gave the health workers activities to be done in one round that were then improved for the next round.

The District health office head also mentioned that;

Action points were prepared in each visit and we tried our level best to manage the problems encountered especially in fulfilling logistics. We also make follow-ups in the implementation of the action points by responsible bodies.

Maintenance

All the respondents emphasized that involvement of different stakeholders and government ownership are critical for the integration and sustainability of the program’s effectiveness. For the program maintenance one of the supervisors recommends that

for academics it is usually theoretical and they need to work in the community more in practical support that will bring change in their local communities. They should work in collaboration with program people and the community.

The district health office head also explained that

We, as the district health office are committed and collaborative to work in integration with all other stakeholders in the future. The university should continue its support by increasing the number of health facilities to be included in the project.

Discussion

National health programs often have a weak supervision system with traditional approaches that are more attuned towards fault-finding and mere inspection.^{12,19,21–23} While there is plenty of evidence on supervision, there is a lack of sufficient evidence to inform the role of medical science colleagues in improving the quality and coverage of immunization programs. The overarching goal of our proposed project is, therefore, to close the gap in this evidence required for enhanced immunization program-related supportive supervision.

The findings of this study revealed that the overall RED implementation status has increased after the introduction of the capacity building and mentorship program. This finding is consistent with other evaluations which showed that there are a number of elements of the RED strategy which are well implemented.¹¹

In African regions, planning and management of resources have lacked details, and this hampers the effective implementation of RED strategies.²⁴ The purpose of the RED strategy is to support countries so as to strengthen immunization systems to sustainably and equitably increase access to and use of vaccination services.¹¹ This study found that among the individual indicators under the planning management component of the RED strategy, the presence of micro plans, identification of high risk areas and having plans for outreach sessions have shown a statistically significant improvement in implementation after the introduction of the CBMPI intervention package. Studies have also indicated that immunization service delivery works best when health facilities make their own micro plans and used their local data for decision making.^9,12,20

Similarly, conducting supportive supervision and the availability of checklists during supervision have been improved significantly by the CBMPI program. In other studies, the supportive supervision component of the RED strategy was the best implemented component.^9,17 Other studies indicated that supportive supervision may not have independent effects on improving the quality of immunization services.¹³ These could be attributed to the availability of essential logistics, supply chain management, human resource and financial resources that may complement the supportive supervision strategy in improving immunization service delivery.

Regarding the outreach related component of the RED strategy, there was a significant improvement in the number of outreach sessions conducted after using a strategy to address hard to reach areas after introduction of the CBMPI project. Evidence also reported an increase in the number of outreach sessions planned and held since the introduction of the RED strategy.²⁴ In Ethiopia, outreach was often provided door to door to increase vaccination coverage.²⁷

Community leaders can be tremendously useful to healthcare providers, especially when it comes to improving vaccination coverage.²¹ Pertaining to the community engagement component of the RED strategies, there was a statistically significant improvement in the intervention group in having planned meetings with the local leaders to discuss routine immunization activities after the introduction of the CBMPI program. Other studies also showed that community volunteers typically assist with defaulter tracking, outreach and community education.^19,22 The qualitative findings revealed that the community engagement after the introduction of the intervention package was not as strong. This is mainly because the women’s development army is in place and there has been a lot of investment and empowerment in rural Ethiopia.¹⁶

Evidence also showed that local health managers in Africa must work strategically to improve health service capacity and effectiveness and increase the use of health services.^22–24 This study also has demonstrated that supportive supervision has a significant role for overall immunization program strengthening. The results of this study showed that Penta III and full immunization has shown a statically significant higher coverage that is attributable to the CBMPI intervention. A study from India indicated that capacity building through supportive supervision improves immunization coverage and also serves as an efficient tool to strengthen the local health system.¹⁶ On the other hand, the intervention package reduced the Penta III dropout rate in the intervention group significantly. Other evidence also indicated that without an effective system to deliver routine immunization, coverage rates cannot be increased.²² Studies and program evaluations also suggested facilitating on-the-job mentoring, quality improvement, and problem-solving to make supervision more conducive to improving health workers’ performance.¹⁹ The qualitative findings also showed that the program was very effective in increasing immunization service quality and coverage.

Inadequate health worker performance is a very widespread problem in developing countries. If correctly done, supervision could be a mechanism for providing professional development, improving health workers’ job satisfaction, and increasing motivation.¹⁶ Interventions such as training can increase the knowledge and self-reported practice of immunization providers.²⁹ In this study knowledge of health workers was improved through the CBMPI program in terms of correct temperature log maintenance, appropriate cabinet temperature for ILRs and the correct placement of diluents in ILRs. Studies also showed that capacity building can play a role in improving knowledge of health workers working in remote sites.²³

Assessment of the skills and practice of health workers was done for five children at each health facility. Accordingly, provision of appropriate vaccines, communication on key vaccination messages and safe disposal of used syringes/needles were performed with correct procedures after the implementation of the CBMPI intervention package. From the qualitative findings, key informants reported that the CBMPI program had a major role in building the skills of health workers working under the EPI program. Supervision of community health workers could be strengthened by streamlining supervision protocols to focus less on report checking and more on problem solving and skills development. In a study in Odisha, those supervisors who had undergone a rigorously planned capacity building and demonstration exercise in the field were better able to transfer skills to the supervisees.¹³

Limitations

Since the study used a quasi-experimental study design, it might be difficult to show a causal association between an intervention and an outcome. In addition, there might be recall and observer bias during data collection. The fact that the study was conducted in two districts may limit the generalizability of the finding across the country.

Conclusions

The newly introduced Capacity Building and Mentorship Program by well-trained personnel from medical colleges of universities had significant positive effects on strengthening the immunization program in rural settings of northwest Ethiopia. Moreover, the study showed that most of the RED strategy components were well implemented as a result of the implemented intervention package. The study indicated that the newly introduced intervention package has improved the knowledge and skills of health workers working in the health system. Thus, it is recommended to do an implementation study of the program within the framework of the national immunization program setting to facilitate policy adoption and readiness for scale up for routine use on a large scale. In addition, further rigorous studies are recommended at large scale to test the effectiveness of the intervention package across all settings in the country.

What is Already Known on This Topic

• The immunization program in Ethiopia is challenged with low coverage.
• The immunization service delivery was not as per the standard.
• Reaching Every District strategy implementation was poor in Ethiopia.

What This Study Adds

• The study indicated that the newly introduced capacity building and mentorship program was effective to strengthen the immunization program.
• The study indicated that the newly introduced capacity building and mentorship program improved the knowledge and skills of the health workers.
• The study indicated that the newly introduced capacity building and mentorship program is feasible and acceptable by health workers.