Development and Validation of a Clinical Prognostic Risk Score to Predict Early Neonatal Mortality, Ethiopia: A Receiver Operating Characteristic Curve Analysis

Introduction

Early neonatal mortality is the death of neonates within the first seven days of life.¹ Despite a 50% reduction in childhood mortality around the globe, the reduction of early neonatal death has significantly lagged behind other Millennium Developmental Goal achievements.²

Globally, there is a 52% reduction in neonatal mortality rate from 37 deaths per 1000 live births in 1990 to 17 deaths per live births in 2019. Neonatal mortality rate declined from 5 million neonatal deaths in 1990 to 2.4 million in 2019. In 2019, the neonatal mortality rate was three deaths per 1000 live births in Europe and Northern America to 27 deaths per 1000 live births in sub-Saharan African countries. Sub-Saharan Africa countries contribute has a great share of neonatal mortality in 2019 (27 deaths per 1000 live births) followed by Central and Southern Asia (24 deaths per 1000 live births).³ In the meantime, much of neonatal death is in the early neonatal period.^4,5 In Ethiopia, the neonatal mortality rate declined from 39 deaths per 1000 live births in 2005 to 30 deaths per 1000 live births in 2019.⁶ The neonatal mortality rate in Ethiopia is higher than in the other sub-Saharan African countries.

Reducing neonatal mortality up to 12 deaths per 1000 live births in all countries across the globe is part of the sustainable development goal of ending child mortality by 2030.^7,8 Even though a significant number of child mortality decline around the globe, neonatal mortality is not decreasing in the recommended standard.⁹

The causes of neonatal mortality are multiple and complex. Prematurity, birth asphyxia, birth weight, tetanus vaccination, infection, ANC follow-up, birth order, multiple birth, and obstetric complications during labour are the predictor variables of early neonatal mortality.^5,10,11 Most of the causes of early neonatal death are preventable by implementing high quality maternal and newborn health care.⁷ For the appropriate medical care, knowing the risk and probability of early neonatal death is very critical.

In clinical settings, patients with their care providers are challenging to make decisions on an estimated risk or probability that a specific event will occur in the future (prognosis) or a specific disease or condition is present (diagnosis).¹² In the prognostic setting, predictions can be used for planning lifestyle or therapeutic decisions based on the risk of developing a particular outcome or condition of health.¹³ In the prognostic setting, estimates of probabilities or risks are rarely based on a single predictor.¹⁴ Health care professionals naturally integrate several patient characteristics and symptoms (determinants, test results, etc.) to make a future prediction. Therefore, the prediction model is inherently multivariable in nature. Prediction models (also commonly called “prognostic models,” “risk scores,” or “prediction rules”) are tools that combine multiple predictors by assigning relative weights to each predictor to obtain a risk or probability condition to happen.^13,15

Therefore, developing a prediction model to predict early neonatal mortality in the neonatal intensive care unit could be useful especially in low-resource settings. Multivariable predictive models for early neonatal mortality developed in developed countries might differ with the inclusion of laboratory markers that are not easily available due to resource or practical constraints in low-resource settings. So, this study aimed to develop and validate the clinical and prognostic risk score to predict early neonatal mortality at Debre Tabor General Hospital neonatal intensive care unit, Northwest, Ethiopia.

Methods and Materials

Study Design, Settings and Period

An institutional-based prospective follow-up study was conducted among the neonates admitted to the neonatal intensive care unit at Debre Tabor General Hospital from December 2018 to January 2020. Debre Tabor General Hospital is found in Debre Tabor Town away from 666 km Addis Ababa the capital city of Ethiopia and 103 Km Bahir Dar Town, the capital city of Amhara Regional State.

Study Participants and Data Collection Procedures

Since there are no prior prediction studies to calculate the sample size, as the rule of thumb 10 events per prognostic determinant was considered for prediction studies.¹⁶ A total of twelve (sex of the neonate, exclusive breastfeeding status, birth weight, birth asphyxia at five minutes, hypothermia, maternal nutrition status, gestational age, mode of delivery, tetanus vaccination status, residence, antenatal follow-up, and Rh status factor) prognostic maternal and neonatal determinants were considered for this study. Even though a small sample is needed, a total of 393 neonates prospectively followed for prediction accuracy at the neonatal intensive care unit at Debre Tabor General Hospital.

Data were collected prospectively using pre-tested structured questionnaires. The questionnaire was initially developed in English by the authors after reviewing different literature and then translated into the local language Amharic and back to translated in English to check its consistency. The clinical profile of the neonates was collected from the neonate chart. Two nurses who work in the intensive care unit at Debre Tabor General Hospital collected the data.

Operational Definition

Maternal under nutrition: Is Mid Upper Arm Circumference (MUAC) less than 21 cm.¹⁷

Low birth weight: When the newborn weight becomes less than 2500 grams at birth.¹⁸

Asphyxia: When the APGAR score of the newborn becomes less than 7 at five minutes after birth.^19,20

Variables of the Study

The outcome of interest is neonatal death (Yes/No) within seven days of the postpartum period. Prognostic determinants were sex of the neonate, exclusive breastfeeding status, birth weight, birth asphyxia at five minutes, hypothermia, maternal nutrition status, gestational age, mode of delivery, tetanus vaccination status, residence, antenatal follow-up, and Rh status factor.

Data Management and Analysis

The data were entered into EpiData (v4.6.0.0) software and imported into R software for analysis. Descriptive statistics including mean, standard deviations (SD), percentages, and rates were ‎employed. Binary and multivariable logistic regression was employed to identify potential prognostic determinants for early neonatal mortality. Afterwards, we used a ‎stepwise backward elimination technique with a p-value <0.10 for the likelihood ratio test to fit ‎the reduced model of easily obtainable determinants. The results were reported as an adjusted odds ratio with 95% CI at a two-sided P-value of less than 0.05. ‎

Assessment of the Model Performance and Validation

Model calibration was assessed by plotting the predicted probability of early neonatal death against the observed neonatal death. Thus, ‎the model calibration was checked using ‘givitiR’ R-package. To check the model discrimination probability, the area under the receiver operating ‎characteristics curve (AUROC) was conducted using ‘pROC’ R-package. The AUROC value of 0.5 indicates no discriminative probability, 0.7 is considered as good, and 1 is a perfect discrimination probability. The internal validation was checked using bootstrapping technique with 2000 ‎iterations of re-samplings. The model’s predictive performance after bootstrapping is considered as the ‎performance that can be expected when the model is applied to future similar populations. Decision curve analysis was checked to identify the cost-benefit analysis of the prediction tool.

Risk Score Development

To construct an easily applicable and interpretable algorithm tool for early neonatal mortality, we ‎transformed each odds ratio from the model equation by dividing the lowest odds ratio to the highest odds ratio and round to the nearest integer. We determine the total score for an individual by assigning the points for ‎each variable and adding them up. ‎The predicted probability of early neonatal mortality was presented ‎according to two categories of the risk score for reasons of statistical stability and practical ‎applicability. The risk score cut point was chosen using the Youden index value and reasonable size of each ‎category on the risk. Later, the score was transformed into a dichotomous ‎‎“prediction test,” allowing each neonate to be classified as a low or high risk of death. The risk score tool ‎sensitivity, specificity, the positive and negative predictive value was determined at different values of the risk score cut-points.‎ Finally, the probability of early neonatal death was predicted using each prognostic determinant risk score value.

‎Results

Maternal and Newborn Characteristics

Overall, a total of 393 neonates were followed prospectively for this study. Among the total of neonates, 56.30% were females. Among the total of the neonate, 24% of them were low birth weight and approximately 16% of them were not exclusively breastfed. Eighty-two per cent of the neonates have adequate antenatal follow-up (more than four antenatal follow-up) (Table 1).

Table 1 Characteristics of the Study Participants

A Predictive Model for Early Neonatal Mortality

For the prediction of early neonatal mortality, maternal and neonatal clinical characteristics are considered. In the bivariable model, Rhesus (Rh) factor, nutritional status of the mother, antenatal care visit (ANC), gestational age, birth asphyxia at five minutes, exclusive breastfeeding, birth weight, hypothermia, tetanus vaccination status, and residence are the significant predictors of early neonatal mortality. After the reduced multivariable model, maternal undernutrition, antenatal follow-up below the focused ANC visit, birth asphyxia, low birth weight, and not exclusive breastfeeding status were the prognostic predictors of early neonatal mortality (Table 2).

Table 2 Prognostic Determinants of Early Neonatal Mortality

The probability of early neonatal mortality prediction based on the linear predictors using the regression formula is:

Linear predictor of the model (lp) = −4.27 +2.38* Not exclusive breastfeeding + 1.14 *asphyxia at five minutes + 0.97 * maternal malnourished + 1.79* low birth weight + 1.24* less than four ANC visit. Therefore, the probability of early neonatal death will be predicted by the regression formula, which is equal to P/(early neonatal death) =

Model Discrimination Probability

The AUROC was 88.7% (95% CI: 83.5–93.5) for the reduced model which is a good discriminative probability (Figure 1). The model calibration checked by comparing agreement between the predicted probability of early neonatal mortality versus observed early neonatal mortality, using the calibration plot (P-value = 0.406) (Figure 2). After internal validation by 2000 bootstrap replicates, the 95% confidence interval of the AUROC curve was 89.2% (84.5–93.5%) (Sup. Figure 1). The sensitivity and specificity of the reduced model 0.55 and 0.96, respectively. Besides, the positive and negative predictive value of the reduced model is 0.71 and 0.89, respectively.

Figure 1 Area under the receiver operating characteristics curve for the reduced model.

Figure 2 Model calibration plot for predicted early neonatal death against observed neonatal death.

Decision Curve Analysis of the Model

As shown in Figure 3, the model has the highest net benefit ratio, which has more clinical and public health importance. The prediction model gives the highest risk of disease or risk prediction probability. Therefore, primary attention should be given based on the prediction model was a higher cost-benefit ratio than not refereeing at all or referring to all regardless of the prediction probability.

Figure 3 Decision curve analysis for cost benefit analysis.

Clinical Prediction and Decision Rules for Early Neonatal Mortality

For the ease of clinical application, a score chart rule was applied for the decision to high or low risk. By this, the prediction of the risk score tool had nine scores. The AUROC of the simplified risk score was 87.8% (95% CI, 82.7–92.9%) (Figure 4). The sensitivity, specificity, positive predictive value, and negative predictive value of each risk score category were determined (Sup. Table 1).

Figure 4 Area under the receiver operating characteristics curve and Youden index cut-point for the risk score tool.

For the clinical decision rule, the risk score is categorized as low risk of mortality and high risk of mortality. The risk score cut point is declared using Youden’s index value, which is the maximum sensitivity and specificity of the risk score. At the risk, score value of 3.5 the sensitivity and specificity of the risk score ROC curve was maximized (Figure 4). Therefore, the individual prediction of early neonatal mortality was a high risk if neonates have a risk score value of more than and equal to four (after rounding the nearest integer). Based on the risk category, 315 neonates had a risk score of less than 4; of them, 18 (0.06%) experienced the incidence of early neonatal mortality. Seventy-eight neonates had a risk score of more than or equal to four, among them 42 (54%) experiences early neonatal death. The sensitivity and specificity of the risk score algorithm category was 70% and 89% respectively. The positive and negative predictive value of the risk category was 53.84% and 94.29%, respectively (Table 3). The overall true prediction accuracy of the risk algorithm to predict the early neonatal mortality was 86% and the false prediction probability was 13%.

Table 3 Prognostic Risk Classification of Early Neonatal Mortality Using Simplified Prediction Risk Score

Finally, the risk of each neonatal mortality is predicted using the score chart formula.

Discussion

This study revealed that the incidence of early neonatal mortality is 15.27 (95% CI: 12.03–19.18). We have developed and validated a simplified clinical risk score model using five maternal and neonatal prognostic determinants to predict early neonatal mortality. The risk score model discrimination performance AUROC is 88.7% and true prediction accuracy 86%, which is the most promising tool to predict early neonatal mortality. Based on the prediction model, not exclusive breastfeeding, birth asphyxia at five minutes, maternal undernutrition, low birth weight, and antenatal care visits of less than four visits were the prognostic determinants of early neonatal mortality. Previous prediction studies conducted in different settings to predict neonatal mortality but some prognostic parameters such as biomarkers were invasive in low resource settings and some of them were done on only low birth weight neonates.^21–23

This study provides an easily applicable risk score tool using the Youden index cut off point, which has the maximum sensitivity and specificity. This prediction model has an excellent negative predictive value and specificity, suggesting that a useful initial screening tool to identify the risk of the neonate for early neonatal mortality.

This is the first study that develops and validates an early warning risk score tool for early neonatal mortality prediction in a low resource setting. This prediction study provides low birth weight and not exclusively breastfed neonates have been highly predictive values for early neonatal mortality. This finding was supported by a study conducted in the United Kingdom and the Gambia,²² France,²⁴ and Ethiopia.²⁵ This might be low birth weight neonates prone to hypothermia, preterm birth, and have a greater influence on the other determinants of health.^26,27 Birth asphyxia at five minutes is another prognostic determinant for early neonatal mortality, which is similar to a study conducted in Bangladesh.²⁸ Another study evidenced that low birth weight, late admission and low APGAR scores were the best predictors of neonatal mortality.²⁹ Maternal nutrition status during pregnancy was another prognostic determinant for early neonatal mortality. This might be maternal nutrition inadequacies directly associated with gestational weight gain and adverse neonatal outcomes.^30,31

Exclusive breastfeeding for the first 6 months of life prevents around 20% of neonatal deaths;³² which is a highly prognostic predictor for this study. Additional food intake before the age of six months can also be prone to early neonatal mortality due to neonatal infections such as sepsis, pneumonia, tetanus, and diarrhea, which contributes about 36% in neonatal deaths from all causes.³³ Furthermore, antenatal follow-up during pregnancy is another prognostic determinant for early neonatal mortality. Previous studies evidenced that antenatal care visits decrease the risk of neonatal mortality,^34,35 this might be mothers have a pregnancy check-up, health education and promotion during antenatal follow-up. Besides, during antenatal follow-up, health care providers can provide information on dietary practice recommendations, postpartum care, newborn care, early initiation and exclusive breastfeeding, and might got iron-folic acid supplementations.³⁶

This study follows some limitations: the first limitation is we did not include the pre-pregnancy body mass index and gestational weight gain since it may be the predictor of early neonatal mortality.

Conclusion

In summary, we have developed and validated an easy prediction tool for early neonatal mortality in low resource-limited setting. This prediction tool could identify neonates at risk for dying in the early neonatal period, which might allow timely intervention to improve neonatal death as well under-five deaths. However, further studies need external validation (geographical validation) to improve the prediction accuracy and applicability of the risk prediction tool.