Development of Monitoring and Evaluation Systems in Four National Programs Addressing Mother and Child Health in Cote d’Ivoire: Qualitative Analysis of the Emergence and Formulation Process

Introduction

Monitoring and evaluation (M&E) is a set of processes of planning, collecting and synthesizing information. Then this information is report with the necessary means and skills so that the results contribute to decision-making and capitalization in the context of a project.¹

An M&E system is an integrated information system dedicated to the selection, collection, analysis, and use of information about development programs and projects for measured risk-taking and improved decision-making.² According to the WHO, an effective M&E system includes an M&E unit staffed by qualified personnel who oversee the M&E of national health services and develop and maintain human resources for M&E. This unit involves all partners in the M&E process. The system defines objectives and interventions for specific targets with a plan for the regular evaluation of progress. It provides clear M&E guidance to the decentralized levels and includes guidelines for involving other sectors, especially the private sector. Furthermore, it coordinates the M&E needs of technical and financial partners (TFPs) and national partners; it develops priority key indicators, a data management and dissemination plan; and it has a centralized database.³ An M&E system is robust if it provides relevant information about the entire process of using inputs to conduct activities and achieve results to achieve an impact in the target population or group.⁴ M&E emerged after the First World War in the North, especially in the USA, to control public resources when setting up development programs and policies.⁵ It quickly became a priority and even a requirement for many international institutions as well as development and humanitarian organizations.^6,7

M&E was introduced in African countries in the 1980s and 1990s and has made its way through several stages.^8,9 From an initial lack of awareness and rejection,^10,11 we see a growing demand for M&E in state structures, nongovernmental organizations, community-based organizations, associations, and especially the establishment of well-structured national M&E systems supported by political commitment.^5,12,13 In the mid-2000s, the national M&E system emerged (Ghana, Kenya, Rwanda, and Zambia), as did national evaluation systems (Benin, Uganda, Nigeria, Kenya, and South Africa), through the initiative of governments in some African countries.^13,14 National evaluation systems are government-led institutional arrangements that guide the selection, implementation, and use of evaluation results to improve governance and accountability to citizens.¹⁵ They are formalized through a national evaluation policy and supported by a government-wide M&E system that provides an enabling environment for their optimal functioning.¹³

The establishment of a national evaluation system where a national M&E system already exists meets the need to develop a culture of evaluation that has long been neglected in African countries due to limited resources and expertise.^16,17 Senegal has developed a more context-specific M&E practice with the support of the Senegalese evaluation network (SenEval) and the African Center for Advanced Studies in Management in Dakar, which have fostered a culture and training of a critical mass of M&E professionals. This has strongly influenced the creation of a Commission for the Evaluation and Monitoring of Public Policies and Programs by the government.⁵

Several studies have examined the gaps and challenges faced by the M&E system in African countries,^4,8,18,19 the evolution of M&E, its adoption as a governance and development tool,^{8,14,16,20–24} and the effectiveness of the M&E system in the success of programs and projects.^4,15–17 The contribution of M&E to this effectiveness was approximately 70%.^6,25 Studies have also examined the effect of capacity building and routine health information systems on the performance of M&E systems in Africa.^26–30 However, these studies and many others^2,15,31,32 have focused on M&E systems implemented in nonhealth sectors or developed and piloted by technical and financial partners with substantial external funding. These programs, projects and policies are often of limited duration and are dynamic. In the health sector, to our knowledge, no study has taken an interest in the M&E system developed in national health programs established by the Ministry of Health, which are characterized by a long life span in Africa.³³

In fact, despite the operation of these programs, certain indicators, especially those for maternal and child health, are hardly improving. Maternal and infant morbidity and mortality in Cote d’Ivoire remain the highest in the West African subregion. The maternal mortality rate is estimated at 645 deaths per 100,000 live births (LBW), compared to 546 deaths per 100,000 LBW in Sub-Saharan Africa. Neonatal mortality (38 deaths per 1000 LBW) and morbidity in children under 5 years of age (108 per 1000 LBW) are very high. Mother-child health remains a public health issue and is included in the policy agenda of the government and technical and financial partners.

For this reason, the objective of this study is to analyze the emergence and formulation of the M&E system developed in national health programs with a particular focus on maternal and child health in Cote d’Ivoire.

Materials and Methods

Conceptual Framework

This study was inspired by Kingdon’s conceptual framework modified by Lemieux and adapted by Ridde.³⁴ According to this framework, the public policies (PP) comprises activities oriented toward the resolution of public problems in an environment in the presence of actors whose relationships are structured and evolve over time. It is composed of three subprocesses, namely, emergence, formulation, and implementation.³⁵ According to Kingdon’s theory, in case of political crises or changes, an opportunity (the political window) arises that combines three trends: the problem trend (situations that could lead to an intervention by the public authorities), the solution trend (different solution options related to the identified problem) and the political trend (political climate, changes in public opinion, social movements). These trends are driven primarily by entrepreneurs (actors who use their knowledge of the process to advance their own policy objectives) who lie in wait for favorable opportunities to place an issue of particular interest on a policy agenda. Kingdon’s theory focused primarily on the emergence process.

Following Kingdon, Lemieux and Ride^34,35 extended their reflections on the processes of formulation and implementation. It is through these processes (emergence, formulation, and implementation) that actors seek to control, to their advantage, PP regulations. For Lemieux, the three trends (problems, policies, and politics) described by Kingdon meet and converge two by two to define the three processes (emergence, formulation, and implementation). The coupling of the political current and the problem current produces emergence. The coupling of the political trend and the solution trend leads to formulation, while the coupling of the problem trend and the solution trend is responsible for implementation. Lemieux, still drawing from Kingdon, described four major groups of actors: (1) officials (elected officials and their associates); (2) officers, who include other people associated with the government machinery; (3) partners, who include business and professional groups, unions, the media, and specialists; and (4) the group of individuals responsible for public opinion and those who have an interest in decisions without being formally organized. All of these actors evolve in an environment and interact in the different processes, often making PPs “swirling”.³⁵

In this study, which analyzes the processes of emergence (Q1) and formulation (Q2) of the M&E system in national health programs, we used the analytical framework adapted from Ridde,³⁴ which considers the context, the actors and the entire process from emergence to the effects of the implemented policy.

The political and health context of Cote d’Ivoire in the 1980s and 1990s could have an influence on decision-making and strategic orientations in the health field. The types of actors and decision-makers at the time and the technical skills available are all factors that can influence these decisions. These different factors will be the subject of our analysis to appreciate the processes of emergence and formulation of the M&E system implemented in the Ivorian national health programs at the time of their creation and to understand the problems related to their current functioning.

Study Setting

Cote d’Ivoire is bordered in the south by the Gulf of Guinea and the Atlantic Ocean. It is limited in the east by Ghana, in the west by Guinea and Liberia, and in the north by Burkina Faso and Mali. The diversity of the Ivorian climate is favorable to the presence of many endemic diseases.

Study Design

We conducted a multiple-case study at a level of analysis that combines a qualitative investigation and a documentary review. The cases were the M&E systems of national health programs analyzed at the central level of the Ivorian health system.

Sampling

The Ivorian health system has 23 national health programs set up to carry out concrete national policy actions on specific targets. Among these national health programs, 2 are exclusively dedicated to maternal and child health (PEV, PNSME), 14 deal with diseases that also affect the mother-child couple. However, we have made the reasoned choice to conduct the present study in the first two and then two others (PNN, PNLS) among the 14 in order to make comparisons. Because the priority of these health programs is maternal and child health: (i) the National Mother and Child Health Program (PNSME), (ii) the National Nutrition Program (PNN), (iii) the National AIDS Control Program (PNLS), and (iv) the Expanded Program on Immunization (PEV), all located in the city of Abidjan. We have decided to focus on maternal health programs because the health indicators of mother and child are high and it’s also our domain of interest.

Participants

The study population was composed of two main categories of participants: (i) former actors of the health system and (ii) technical and financial partners who support the health system.

We listed the former actors from the 1980–1990 period by document review and identified them using the “close to home” technique from a key informant in our circle. Each former actor we met put us in touch with peers and former collaborators at the central level by providing us with their updated telephone and e-mail contacts and sometimes by calling the people concerned to introduce us. We also did this to enlist the technical and financial partners whose agencies had supported the health system during the period covered by the study. We enrolled 31 actors, including the following:

• Twenty-five former actors who were directly involved in the design, planning, and implementation of health programs. They had served as Minister of Health (02), Director of Cabinet of the Ministry of Health (02), Director General of Health (03), Central Director (03), Coordinating Director (CD) of health programs (10), Research Officers in the Cabinet of the Ministry of Health (02), M&E Officers and Data Managers (02), and Head of Rural Health Base (01). Some of these former actors were still active at the national (08) and international (04) levels.

Four key actors, including a former Minister of Health, declined to participate due to unavailability and lack of interest.

• Six employees in 4 TFP agencies: the World Health Organization (WHO), the Centers for Disease Control and Prevention (CDC), the United States Agency for International Development (USAID) and the United Nations Children’s Fund (UNICEF). Unfortunately, we were unable to locate the former actors from the 1980s and 1990s, who were mostly European and Asian. The employees of the technical and financial partners agencies that we enlisted were Ivorians who had served in the national health system before being recruited by the partners.

Overall, thirty-one (31) actors were interviewed.

Data Collection

Two interview guides, one for each target population, were designed around the two main terms. Data were collected from January 10 to April 20, 2020. Individual interviews were conducted at the workplace for key informants who were still working and at their home, in a public place or in the office of the first author (EEML) for retired actors. The interviews were recorded as much as possible with the consent of the interviewees. Detailed notes were taken on the spot if respondents were reluctant to be recorded.

Data were collected by individual face-to-face interviews or by Skype. Skype interviews were conducted during the COVID-19 restriction period and with two key informants living outside the country. The interviews were conducted using interview guides covering two central themes: emergence and formulation. Questions related to the emergence process focused on the reasons for introducing M&E into health programs and the roles, interests, and responsibilities of key stakeholders. The formulation questions focused on the regulatory texts, the process of drafting them, and the key actors in the process.

In addition, document analysis using a reading grid made it possible to exploit the documents collected from the Ministry of Health and TFP agencies. These were the 1995 and 2010 national health development plans, the decrees creating and organizing the health programs, the health system evaluation reports, the communiqués of the Council of Ministers relating to M&E and other documents of interest found in the mail service of the Ministry of Health.

Data Analysis

All the interviews were transcribed verbatim in Microsoft Word from audio recordings by sociologists with a PhD in sociology and several years of experience in the field. The first author (EEML) listened to the entirety of the recorded interviews from these transcripts to verify the accuracy of the technical terms and the quality of the documents. The resulting transcripts were deidentified and stored securely in an online storage service.

We referred to the code book previously elaborated by two encoding agents. This codebook allowed us to identify the central themes and secondary subthemes that were used to encode the data in NVIVO 12 software. To ensure the consensual quality of the nodes, a Kappa test was used for comparison, with a score greater than or equal to 90% considered the acceptable level of agreement. The codebook was used to classify the data collected during the interview. Each interview was subjected to a content analysis by performing in-depth verbatim readings. Certain verbatim excerpts from the key messages were retained to support certain ideas by way of illustration. The corpus thus obtained was processed based on thematic content analysis. The triangulation of the interview analysis with the document analysis made it possible to identify the windows of opportunity, the entrepreneurs, the actors’ perceptions of the problem to be solved, and the role and interests of the actors in the choice of solutions and to describe the emergence and formulation of the M&E system in the priority health programs for mother-child pairs.

Ethical Considerations

Ethical approval was granted by the National Ethics Committee for Life Sciences and Health of Cote d’Ivoire (N/Ref:108–19/MSHP/CNESVS-kp of September 05, 2019). Informed consent was required before the participation of any individual selected for the investigation. The confidentiality of the information collected has been mentioned in the informed consent form as well as benefit and risk. We applied confidentiality of information and anonymity of responses throughout the process, from data collection to writing the article. The participants informed consent also included publication of anonymized responses.

Results

The 31 people in our sample who were interviewed included 25 people (4 women and 21 men) from the Ministry of Health and 6 (including 1 woman) from technical and financial partner agencies. In total, 5 women (16%) and 26 men (84%) were interviewed.

Our results are presented according to two parts: (i) the emergence of M&E systems in national programs addressing maternal and child health, (ii) the processes of formulating M&E systems in these health programs.

Emergence of Monitoring and Evaluation Systems in Health Programs

According to some key informants, the new Minister of Health, appointed in 1989, discovered the concept of community health during a visit to Latin American countries and Canada. His ambition was to make primary health care operational in Cote d’Ivoire according to the recommendations of the Bamako Initiative (1987). The Community Health Directorate was created in 1991 to implement primary health care and ensure epidemiological surveillance, hygiene promotion and disease prevention. A dozen national health programs were set up in the city of Abidjan starting in 1992. An Information and Management System Department was set up in 1995, and the first national health development plan was developed Achieving results was a major concern of the new Minister of Health when the national health programs were established. However, since Ivorian public administrations had long operated with the resources available without any obligation to achieve results or accountability, the health authorities had great difficulty in monitoring the progress of emerging national programs and capturing the results of their interventions, according to the explanations of these former actors:

We had a lot of programs that were emerging, and we didn’t even know how to monitor them… they needed to have plans… the idea was to monitor and evaluate them, to know where they were in terms of program progress and the progress that was being made (Community Health Directorate’s Former Manager n°1, male gender, 30 years seniority in the system)

Several other respondents agreed, including this former Program Director Coordinator, male gender, 27 years seniority in the system:

We found that after a few years of struggle, we didn’t have a clear picture of where we were. We didn’t have a clear idea of the activities that had been undertaken since the 1960s. It was difficult to mark where we really started and where we arrived, which objectives were achieved, and which were not.

The health authorities had difficulties reorganizing health programs, especially getting them to capitalize on what they had learned and to rely on evidence to make advocacy. These authorities were aware that the entire health system needed mechanisms to monitor activities and capture results to inform decision-making, according to this former Ministry of Health official, male gender, 30 years seniority in the system:

… We needed to set up a mechanism that would allow us to monitor the implementation of interventions on a day-to-day basis… detect deviations in order to catch them in time…

The main window of opportunity that fostered the emergence of the M&E system was the accountability requirements and conditionalities of major international donors, who were concerned about development effectiveness:

…So inside the USAID projects, there was a lot of M&E. Because these donors also have constraints, ie, they have to report to the US Congress that gave them the money. They have to present them with results every year. For example, I served such and such a number of people, of such and such an age group, in such and such a geographical area with such and such a sum of money (USAID employee, male gender, 25 years seniority in the system).

Similarly, one MEASURE-Evaluation respondent, female gender, 25 years seniority in the system said:

Donors got together and realized that there were no tangible results after several years of funding health programs. So, they thought they would put in place a system to monitor and show that the funding they allocate to programs or countries is leading to a noticeable change in the health status of populations.

Indeed, the notion of accountability became paramount when development partners’ funding became important, and they demanded accountability for its use:

The fundamental reason for doing M&E for donors is simply that they put money into something, and they want to know what that money was used for and how people used it. This all goes back to the 80s and 90s… with the early projects’ partners put a lot of money in and which often covered most of West Africa… (USAID employee, male gender, 25 years seniority in the system)

Thus, donors decided to implement M&E systems in health programs, according to this former Director General of Health, male gender, 26 years seniority in the system:

M&E kind of came from pressure from external partners who wanted health programs to achieve the results they set out to achieve and that got the partners to put their money in.

Another former Director Program Coordinator, male gender agreed, 20 years seniority in the system:

I think it’s not just a national reflection; it all started with the WHO recommendations that all health projects and programs must necessarily have an M&E component.

The local health authorities agreed with this recommendation, according to this respondent, male gender, 38 years seniority in the system:

If there is no M&E chapter in the program, how are you going to have an idea about the effectiveness, efficiency, performance of all this? It is through these M&E approaches. So, there’s no program that works without an M&E chapter.

Therefore, the Community Directorate’s managers pooled their skills and expertise to organize and coach the first action plan development seminar with all health program managers in 1995 in Aboisso. The plans that emerged from this exercise had an intervention component with operational activities and an M&E component to better capture program effectiveness, efficiency, and performance.

“At the Aboisso workshop, the teams had a model plan of what a health program is and I based on WHO from the rationale to the evaluation” (Community Health Directorate’s Former Manager n°1, male gender, 38 years seniority in the system).

Following this workshop, a global decree for the creation of health programs was issued, including the introduction of the M&E function in the organizational chart of national health programs, according to this second former manager of Community Health Directorate, male gender, 30 years seniority in the system:

The idea of M&E in health programs started from the Community Health Directorate. It was not the technical partners who initiated it, and the Ministry of Health created the texts… with the decree of the Minister of Health at the time, which instituted the executive directorates of the program with a structuring of the services, including that of M&E.

Process for Formulating M&E Systems in Health Programs

The formulation of the M&E system in health programs followed a two-deed process, the first of which was the Aboisso planning workshop held in 1995. It was during this workshop, which brought together all health program managers, that M&E was first added as a component of project planning. It was then formalized through the 1995 health program creation decree (Decree No. 174 MSPAS/CAB of March 27, 1995), which explicitly mentioned M&E in the program activities. This decree served as the basis for drafting specific decrees for all national health programs with an M&E component.

There was a global order in 1994-95, which explained how to write a decree for the creation of a health program with, namely, a scientific committee and the M&E department. So, everyone used this as a basis for writing the preliminary drafts of the order to be submitted to the General Manager of Health and then to the Minister of Health. (Former Director General of Health n°2, male gender, 40 years seniority in the system)

There was a diversity of responses and contradictions concerning the actors and the process of drafting decrees for the creation of health programs, which reflected the interviewees’ real lack of knowledge of the drafting approach. In reality, the draft texts of the creation decrees and their approval were made by the Cabinet of the Ministry of Health according to a top-down approach that excluded any search for consensus among the stakeholders, according to the words of this former Cabinet research officer or former Cabinet program coordinator, male gender, 26 years seniority in the system:

Consensus-building through group work was not really making things progress, so I wrote the regulatory texts of the health programs based on my knowledge and vision of the evolution of the health system. Then I would use the legal department of the ministry and the general secretariat of the government for compliance and their agreement.

In so doing, the original vision of the initiators of the first workshop in Aboisso was obscured during the process of drafting the creation decrees, according to former Community Health Directorate officials, due to a lack of consultation. The basic creation decree did not mention any standards or guidelines that could be used as criteria for establishing a robust M&E department in health programs. It merely listed the various entities responsible for administering, facilitating, and monitoring health programs with a brief description of their respective activities, some of which are supportive of M&E.

…It was often a hasty decree that was issued to say that a program was created. There was no particular organization until a certain year. I think it was when they started to create the executive directions that we really started to structure the organization of national health programs. But I don’t know if an M&E department was actually created within these executive directions. (Community Health Directorate’s Former Manager n°2, male gender, 30 years seniority in the system).

Unfortunately, by relying on this first decree to draft all the other decrees, as explained by most of the key informants, the actors perpetuated the shortcomings of a time when expertise in M&E and the drafting process was limited.

Discussion

This study, which analyzed the processes of the emergence and formulation of the M&E system for national programs addressing maternal and child health in Cote d’Ivoire, highlighted the context and the role of key actors and raised two major issues that deserve to be discussed. The emergence of an M&E system supported by development partners and the approximate formulation of an M&E system in health programs through a nonparticipatory and exclusive process.

Emergence Supported by Development Partners

M&E brought a new paradigm that consisted of going beyond administrative management or simple management by objectives to identify the effectiveness or performance of an intervention.

According to our results, M&E has indeed emerged in health programs in Cote d’Ivoire thanks to the meeting of the will of the actors in the Ministry of Health and a window of opportunity constituted by the concern for the effectiveness of external aid as in the study by Coulibaly and allies.³²

The involvement of external actors in the emergence of policies in Africa was demonstrated in previous studies^32,33 as well as the critical role played by development partners in the introduction and implementation of M&E in African countries.^3,11–13,18 External funding favored the introduction of M&E in Africa when it became significant and required a transparent and accountable environment to track the achievement of expected results.^13,18

According to Lemieux, elected officials and their partial associates were the most active partners in health policies. The advent of a new liberal government facilitated the emergence of policies for integrated health care organizations in Canada.³⁰ The appointment of a new Minister of Health favorable to the performance-based funding program and a national political will to fight corruption favored the emergence of performance-based funding in Cameroon.³⁴

The decision to create a result-based financing unit within the Ministry of Health in Burkina Faso was made in response to the appointment of a new minister and administrative changes in budget allocation.³⁵ In general, crisis situations and political or governmental changes create the conditions for policy emergence and formulation.^30,36–38 However, the Ministry of Health and Population in Nepal was not able to act as a policy entrepreneur, and the first windows of opportunity to address gender-based violence as a health issue were missed.³⁹ According to our results, M&E emerged in national health programs in Cote d’Ivoire as a result of the coupling of the problem trend (structuring health programs to achieve their objectives in a corrupt social environment without control or rigor) and the political trend (the vision of the new Minister of Health) by local actors (acting as entrepreneurs) at the opening of a window of opportunity (accountability due to donor funding).^30,31,40,41

Approximate Formulation of M&E Systems in Health Programs Through a Nonparticipatory and Exclusive Process

The formulation of M&E systems in health programs was accomplished through a top-down approach during the editing of the texts of preliminary drafts of the creation decrees and their approval. This approach, like the hierarchy, is based on the principle of constraint or restriction (Lemieux, 1994). Thus, the formulation occurred through a nonparticipatory process that excluded any contribution from stakeholders and experts in the field, ignoring even the original vision of M&E of the Community Health Directorate initiators. As a result, the basic creation decree that served as a model for the drafting of all the other health program decrees was not detailed enough and did not mention the content of the M&E system because of the context in which the practice of M&E was not known. The characteristics of a strong M&E system should be defined at the time of formulation to facilitate its implementation and serve as criteria for its future evaluation. Similarly, standards and policy frameworks should be specified to codify the running of robust M&E systems.¹⁸ M&E systems need a conducive environment to function effectively and inform decision-making.^6,12,24,42

These shortcomings in formulation, linked to the lack of M&E expertise at the time, suggested a “whirlwind” implementation with numerous inconsistencies, failures and wasted resources.^43,44 Above all, they made the coordinating directors the “decision-makers” for the implementation of M&E systems in national health programs. The “decision-makers” could not effectively decide alone without a proposal from the “designers”.⁴⁵

For example, the inclusion of public policy evaluation in the French constitution is framed by regulations to ensure that M&E systems are rigorous, transparent, adaptive, and respectful of multiple viewpoints.⁴⁶

An M&E agency policy has been constitutionally supported in Uganda and Ghana. In South Africa, an executive prerogative was made for the operation of key elements of the M&E systems.^11,18 This government-wide policy framework, supported by national funding for regular evaluations, created a true M&E culture in these countries. In so doing, these countries have become true pioneers of M&E in the African context, where shortcomings in health policy formulation are legion.^18,41,43 In Cote d’Ivoire, the inadequate formulation of M&E in national health programs has been a weakness in the achievement of certain convincing results. In addition, this situation has been detrimental to the emergence of a true M&E culture at the national level and of national champions in this area. This has contributed to the relegation of M&E to the background and the absence of a coherent national M&E system.

Study Limitations

The main limitations of this study were the absence of documents, a lack of access to the first key actors (deceased or not found), memory bias, and fear of criticizing the ruling system, especially for those actors who are still in office.

Conclusion

This study shows that the emergence of M&E systems in national health programs has been the work of both the national party and the donors in the interest of performance, accountability and informed decision making. External funding provided a window of opportunity for national actors to introduce M&E into the management of national health programs with the new appointment of a Minister of Health who had the vision to make community health functional.

However, its formulation through a top-down process was sketchy and standard in all health programs. It would be appropriate for Cote d’Ivoire to fill this gap by reformulating it objectively, considering the specificity of each national health program and involving all stakeholders to reach a consensus that will facilitate implementation and the achievement of results.