Use of a Mortality Prediction Model in Children on Mechanical Ventilation: A 5-Year Experience in a Tertiary University Hospital

Introduction

The pediatric intensive care unit (PICU) team aims to provide care to severely ill children by applying early extensive interventions and high-quality treatment to achieve better clinical results. The initial assessment of illness is a critical part of PICU evaluation and management.¹ Currently, multiple scoring systems have been developed for predicting mortality in critically ill patients who are hospitalized. However, children admitted to the PICU might develop complications that can cause rapid deterioration in health status and can lead to death despite the provision of high-quality care. Thus, the timely use of a mortality prediction tool can be extremely helpful in settings with severely ill children receiving mechanical ventilation. Preferably, the predictive tool must be established using basic clinical information and must have a high specificity to accurately identify patients with a high chance of survival. Currently, several scoring systems for predicting mortality in patients admitted to the pediatric intensive care unit (PICU) have been developed. In 1988, Adoria et al established the Pediatric Risk of Mortality (PRISM) score, which comprises 14 variables, for predicting mortality in patients admitted to the PICU.² In 1996, Pollack modified the PRISM score to PRISM III score, which has three additional variables. The efficacy of the PRISM III score (17 variables) on 11,165 patients in 32 PICUs across the United States was assessed. The results demonstrate that the tool yielded better results than PRISM in predicting mortality.³ Hence, mortality can be predicted using data obtained in 12 h (PRISM III-12) or 24 h (PRISM III-24). The PRISM III score is a mortality prediction scale with good predictive ability.^3,4 However, it requires multiple variables such as blood test results and physiological parameters.⁵ These data might not be accessible during emergency situations or might be difficult to record routinely in PICUs with different resources. In 1997, Shann et al introduced the Pediatric Index of Mortality (PIM) 2 score (updated in 2003) for predicting the outcome of children admitted to PICUs. The PIM 2 score comprises 10 variables. Yes and no responses for these variables were scored as 1 and 0, respectively. These data were then entered into the system (www.sfar.org/scores2/pim22.html) to calculate for the predicted mortality rate.^6,7 The use of the PRISM and PIM 2 scores requires rigorous and specific training and strict adherence to guidelines. Consequently, assessments should be performed by well-trained professionals; however, they can cause wide variabilities in the study.^8,9

Therefore, this study aimed to develop a simple model for predicting early mortality based on clinical data obtained in the first 24 h of admission. Moreover, the ability of the newly established model and the PRISMA III score in predicting mortality was compared.

Patients and Methods

Aim

This retrospective longitudinal study aimed to establish a simple early predictive mortality (SEPM) model with high specificity for identifying severely ill children who would benefit from extensive mechanical ventilation during admission in the PICU.

Participants

Pediatric patients aged older than two weeks who were receiving mechanical ventilation and were admitted to the PICU of King Fahd Hospital of the University (KFUH) from January 2015 to December 2019 were included in this study. The PICU of KFUH is a 10-bed multidisciplinary unit that is well equipped with advanced machines that can be used in providing care for infants and children. The unit provides specialized care to patients with complex, surgical, oncological, orthopedic, trauma, and medical care requirements. It serves a community of ~4.9 million people with ~250 admissions annually.

Patients admitted for <24 h, those who died within the first 24 h, those who experienced cardiac arrest before PICU admission, those admitted to the PICU due to burn, and those with incomplete or missing data were excluded from this study. Patients who died within the first 24 h were excluded because our intended prediction model and the PRISM III scale that was compared with our model require information gathered within the first 12–24 h of admission; thus, for patients who died within this period, the requisite data to calculate the score would not have been available. In this regard, brain death cases were not excluded because brain death cannot be declared prior to 24 h of PICU admission as per the policy in Saudi Arabia.¹⁰

Data

Data were collected within the first 24 h of PICU admission and were retrospectively extracted from the medical records. Mortality was calculated at the end of the PICU admission period.

The variables included age (infancy [≤1 year], toddlerhood [>1–2 years], early childhood [>2–5 years], middle childhood [6–11 years], and early adolescence [12–18 years]), gender, cause of PICU admission (respiratory, cardiovascular, or nervous system problems and other medical or surgical conditions), number of organ failure, and sepsis (no, yes). The definition of sepsis was adopted from the recent Surviving Sepsis Campaign guidelines, in which sepsis was defined as “severe infection leading to cardiovascular and/or noncardiovascular organ dysfunction”.¹¹

Information was obtained on the presence of comorbidities including asthma, chronic lung disease, congestive heart failure, congenital heart disease, cerebral palsy, uncontrolled seizures, genetic and metabolic disease variants with CNS involvement, type 1 diabetes mellitus, malignancy, immunodeficiency, thalassemia, sickle cell disease, chronic liver disease, and chronic renal disease, and PRISM III score with 17 physiologic variables subdivided into 26 ranges.

This study was approved by the Institutional Review Board (IRB) of Imam Abdulrahman Bin Faisal University (IRB-2020-01-213). Based on the IRB’s Policies of Imam Abdulrahman Bin Faisal University, patients’ consents for medical records review purposes are waived for retrospective studies. Furthermore, data confidentiality was ensured following the Declaration of Helsinki principles.

Analysis

The STATA (version 16) software was used for statistical analyses. The chi-square test, Fisher’s exact test, and t-test were used to compare the individual characteristics of children who survived and died. Univariate logistic regression analysis was performed to investigate significant variables that might be correlated with mortality in the final prediction model. The final logistic regression model was established using the forward stepwise approach, and the goodness of fit incidences was utilized to examine model fitting (log likelihood ratio, adjusted R2, Bayesian information criterion [BIC], Akaike information criterion [AIC], and receiver operating characteristic [ROC]). The Wald test and likelihood ratio test were used to compare the nested models. The ability of the final prediction model was examined using the ROC curve and the classification table. Moreover, the Hosmer–Lemeshow test was utilized to examine model fitting using 10 random subgroups. The collinearity among continuous and explanatory variables was examined using the correlation matrix, scatter plot, and variance inflation. The risk of mortality was calculated using the following equation:

Results

Study Participants

Of the 1319 patients admitted to the PICU, only 400 were included in the analysis; 93.70% (n = 922) of the excluded patients were not put on ventilators while the remaining 6.30% (n = 62) patients were excluded because of other exclusion criteria (Figure 1).

Figure 1 Schematic illustration of the patient selection criteria.

The incidence rate of mechanical ventilation among children admitted to the PICU within the last 5 years was 28.90% (n = 400). The mortality rate was 16.25% (n = 65), and 20% of children who died were toddlers. The major causes of PICU admission were respiratory, cardiac, and renal problems. Trauma-related admissions were mainly attributed to head injuries (n = 14, 3.6%), and 69% of children with trauma were aged below 5 years. Moreover, ~50% of deceased children had complications such as sepsis and/or multiple organ failure; 60% (n = 39) of patients admitted to the PICU died after an unsuccessful cardiac pulmonary resuscitation (CPR) and 24% (n = 16) were found to be brain dead upon evaluation after 24 h of admission to the PICU. Furthermore, the parents of 10 (15%) patients signed the do not resuscitate request form, and these patients did not further receive escalation therapy.

The mean number of PICU admission days of all participants was 11.66 (standard deviation [SD] = 7.09; max = 32, min = 2). The mean number of PICU admission days was significantly higher in the deceased children (15.77 [SD = 7.07]; max = 31, min = 5) than in those who survived (10.86 [SD = 6.82]; max = 32, min = 2; t(398) = 5.277; P < 0.001). Additionally, the mean number of total ventilation days of all participants was 7.92 (SD = 9.14; max = 46, min = 2); the mean ventilation days in the deceased children was 14.14 (SD = 6.33; max = 25, min = 4), whereas in the children who survived, it was 6.71 (SD = 2.61; max = 17, min = 2). There was a significant difference in the number of inotropes received between the deceased patients (Mode = 2; mean = 2.03 [SD = 1.16]; min = 0, max = 4) and those who survived (Mode = 1; mean = 1.06 [SD = 1.04]; min = 0, max = 3; z = −6.000; P < 0.001). The mean weight of the deceased patients was 19.6 kg (SD = 11.93; max = 43, min = 4) and that of those who survived was 23.2 kg (SD = 11.84; max = 55, min = 5; t(398) = 2.24, P = 0.026). The deceased children (mean = 17.57, SD = 2.91, max = 28, min = 6) had a significantly higher mean PRISM III score than children who survived (mean = 6.51, SD = 5.84; max = 17, min = 0; t(398) = 22.97; P < 0.001). There was no statistically significant difference between children who survived and those who did not in terms of age and sex. The individual characteristics and distribution between children who survived and those who did not are presented in Table 1.

Table 1 Distribution of Individual Characteristics in Relation to Mortality Rate

Mortality Prediction Model

Model Selection

First, a univariate logistic regression analysis was performed to assess the risk of mortality before using the forward stepwise adjustment approach in selecting study variables or interaction terms that will be included in the prediction model. The various examined interaction terms were between the following study variables: weight, age, gender, and comorbidities; however, these interaction terms were not statistically significant or did not improve the prediction model goodness-of-fit statistics.

The final model was the nested model that included four categorical variables (cause of admission, comorbidity, sepsis, and age categories) and two continuous variables (number of organ failure and weight). Sex as well as interaction terms of variables were not included in this model because they did not improve the model fitting statistics.

This chosen SEPM model had the lowest likelihood ratio (177), BIC (−2128), and AIC (212) and highest adjusted R2 (McFadden’s R2 = 0.402, Cragg–Uhler’s R2 = 0.591, and Efron’s R2 = 0.542).

Predictive Ability of the Simple Early Predictive Mortality Model (SEPM)

The ROC curves for the SEPM model and the PRISM III score is presented in Figure 2. The SEPM model showed an excellent predictive ability (area under the curve [AUC] = 0.916; 95% confidence interval [CI] = 0.872, 0.958). Although the area under the ROC curve of the SEPM model was not superior to that of the PRISM III score (AUC = 0.955; 95% CI = 0.924, 0.986), there was no significant difference between the AUC of the two models (Χ²(1) = 2.22; P = 0.136).

Figure 2 Receiver operating characteristic curve for the SEPM model and PRISM III score.

The PRISM III score showed an excellent discrimination ability using a cutoff point of ≥12 (sensitivity: 84.62%, specificity: 96.12%, classification ability: 94.25%, positive likelihood ratio: 21.80; and negative likelihood ratio: 0.16). Our SEPM model showed an excellent classification ability (92.25%) using the probability cutoff value of 0.5 (Table S1). However, the cutoff points could be changed to improve either the specificity or the sensitivity, as required (Figure 3). Using the probability cutoff of 0.5, the specificity of the SEPM model was 97.31%, sensitivity was 66.15%, positive predictive value was 82.69%, and negative predictive value was 93.68% (Table S2).

Figure 3 Line graph showing the sensitivity and specificity of the SEPM model according to several cutoff points for the predictive probability.

Table 2 presents a summary of the model’s coefficients and ORs in the chosen SEPM prediction model. This SEPM model had a statistically significant goodness of fit statistics using the log likelihood ratio test, Wald test, and Hosmer–Lemeshow test with 10 random groups.

Table 2 Summary of SEPM Model Estimates and Statistics

Discussion

Our study aimed to predict mortality among ventilated children admitted to the PICU. Mechanical ventilation is the most common technology used to support severely ill children in the PICU, and its use might imply a higher disease severity. However, the incidence rate of mechanical ventilation in our unit was 28.9%. This rate was acceptable considering that the proportion of children on mechanical ventilation in PICUs vary widely from 17% to 60%.^12–16 In this study, the mortality rate among children on mechanical ventilation in the PICU was 16.25%, and this is comparable to that reported in the literature. However, most patients were lost after an aggressive CPR or after being diagnosed as brain dead. These results differed from those of a study conducted in the United States. That is, the research showed that 70% (n = 133) of PICU patients died after withdrawal from life-sustaining treatments.¹⁷ These differences in resuscitation decisions and withdrawal from life-sustaining treatment might be attributed to our culture. That is, there is a limited understanding on Islamic laws and ethics regarding end-of-life decisions that allow withdrawal of life support machines in children who are terminally ill and who are about to die.¹⁸ In addition, as per the Saudi Policy, brain death cannot be declared in pediatric patients before 24 h of admission to the PICU.¹⁰ However, this period can be further extended depending on the underlying cause of admission and the extent of injuries, which make it difficult to make an early decision on withdrawal of life support machines.¹⁰ Furthermore, in our study, respiratory (such as pneumonia and bronchiolitis) and cardiovascular diseases were the most common causes of PICU admission and mechanical ventilation. This finding can be attributed to the fact that these conditions are prevalent among children, particularly those aged below 5 years, in Saudi Arabia. These results are similar to those of studies conducted in other countries, which showed that respiratory and cardiovascular diseases are the two major causes of admission among children in the PICU.^8,19

This study aimed to establish a simple model for predicting early mortality within 24 h of admission among severely ill children receiving mechanical ventilation in the PICU. The model used the minimum number of required clinical information to facilitate decision making within a short period. Our model included six clinical data such as weight, age categories, presence of comorbidity, cause of PICU admission, number of organ failure, and severity of sepsis. The accuracy, specificity, and sensitivity of this model were 92%, 97%, and 66%, respectively, using a cutoff probability value of 0.5. Moreover, the accuracy, specificity, and sensitivity of the PRISM III score were 95.25%, 98.51%, and 78.46%, respectively, using a cutoff point of ≥14. The results of the current study were comparable to those of published research in the literature. However, these studies revealed a lower classification ability ranging from 89.2% to 92%.^9,20–22 Furthermore, another study was conducted on patients with meningococcal infections admitted to the PICU. The results showed a higher AUC for ROC (0.94), which might indicate that the predictive ability of the PRISM III score could be enhanced by illness severity, as in our case.²³ The PRISM III score might directly reflect illness severity rather than death if it was calculated in the early period of admission to the PICU (ie, in the first 12–24 h). Thus, it has a high predictive ability among extremely sick children.⁶ Our model used clinical and demographic data obtained in the first 24 h of admission, which is the period wherein the PRISM III score is routinely calculated. However, the model depends on variables such as severity of sepsis and number of organ failure, not physiological and laboratory parameters, which may vary widely when obtained during the same time frame. Moreover, the accuracy of the PRISM III score might be influenced by study design. That is, in another PICU study, the AUC for the PRISM III score obtained 12 and 24 h after admission was 0.78 (95% CI = 0.67, 0.89). Moreover, that for the PIM2 score was 0.74 (95% CI = 0.63, 0.85) when a prospective rather than a retrospective design was used. This result might indicate that a retrospective study design could cause an overestimation of mortality prediction because of precision error in collecting data during events.²⁴ The PIM2 score, which uses both clinical and laboratory data, is usually performed within 1 h of PICU admission. However, the patient’s condition might rapidly change after providing hemodynamic support; thus, predictive ability might be affected by the level of management in PICU. The use of PIM2 and PRISM scores, unlike our model, requires training to ensure adherence to treatment guidelines. Thus, this might limit the number of professional staff who could perform the assessment using the scoring system and could enhance the wide variability in results.²⁵ The high specificity (97.31%) and negative predictive value (93.68%) could have helped in identifying children who have a high chance of survival and who are likely to benefit from therapeutic interventions. However, the model had a low sensitivity (65.6%). This outcome might be influenced by PICU interventions provided after admission. Hence, the actual mortality rate among children who had a risk of dying at the time of admission decreased. This result might indicate that the PICU team had a good performance, which led to the high possibility of false-positive rates. Nevertheless, in this context, our model might be a useful quality assessment tool for assessing care provided in PICUs.

To our knowledge, this is the first study that used a simple model for predicting mortality among severely ill children who are on mechanical ventilation in this ethnic group. The current study had several limitations. That is, it was conducted at a single center. Furthermore, only a small number of patients were included, and most patients were Arabs. Thus, the model’s performance might differ when applied to another population. However, the PICU of this university hospital receives referrals from all over the eastern province of the Kingdom of Saudi Arabia. Hence, the results could be generalized on a wide region in the country. Moreover, there were no external validation data that can be used to confirm the accuracy of our model, and this is considered another important limitation. However, if external data can show that our prediction model had inappropriate fitting, it will be difficult to assess whether this was attributed to differences in context information found in other hospital sittings or to the limited prediction ability of the model itself. Nevertheless, additional studies should be conducted to assess our model. Moreover, its prediction ability must be enhanced by modifying variable categorization using other significant clinical variables that were not included in our model. These variables include admission pathway (ie, pediatric ward and ER), pupil’s reaction to light, and length of hospital stay before starting ventilation in the PICU. Our model could accurately classify patients using a probability cutoff point of 0.5 and using only six variables. Therefore, it is comparable to the PRISM III score. However, the model was more simple in term of the number of variables used and the availability and accessibility of these variables particularly during emergency situations or family consultations. The prediction model was used on severely ill children who were receiving mechanical ventilation. However, future studies should utilize this model on other PICU patients, such as those who are not on mechanical ventilation. Finally, as previously discussed, the ability of prediction models might be affected by interventions provided in the PICU and possible measurement errors associated with a retrospective study design. Hence, in the future, a prospective study should be performed to examine the ability of prediction models during several periods in PICU admission and to evaluate whether treatments affected the model’s predictive ability. In addition, we recommend comparing the prediction ability of our model against that of other validated prediction models such as PIM2 and PIM3. We could not perform this analysis due to the retrospective nature of our study and the possibility of developing time reporting error for some variables during data extraction.

Conclusion

The SEPM model had a high specificity in predicting mortality using only six clinical predictors that might be obtained easily in the early period of PICU admission. In addition, the prediction ability of the SEPM model and the PRISM III score was comparable. That it, they had an excellent mortality prediction ability in severely ill children. Finally, further research should be conducted to validate the new SEPM model using an external dataset and a prospective longitudinal design that can consider the effect of treatment on the model’s prediction ability.