Development and Validation of a Dynamic Nomogram for Predicting in-Hospital Mortality in Patients with Acute Pancreatitis: A Retrospective Cohort Study in the Intensive Care Unit

Introduction

As one of the most common gastrointestinal disorders, acute pancreatitis (AP) is an unpredictable and potentially fatal disease. The annual incidence of AP is about 34 per 100000 person-years.¹ The majority of AP patients usually have a mild course which recovers within one week without any sequelae. However, approximately one-fifth of patients develop into moderate or severe AP due to organ failure, with a mortality rate of 20–40%.² A number of studies have found that severe patients might benefit from early adequate fluid resuscitation,³ sufficient analgesia,⁴ early enteral feeding,⁵ use of endoscopy.⁶ Therefore, it is highly significant to identify patients at high risk during the early stages of AP. This identification not only aids clinicians in making optimal clinical decisions and improving the prognosis but also facilitates patient and family counseling/explanation as well as resource allocation.⁷

Currently, there were some score systems such as Ranson, Glasgow, sequential organ failure assessment (SOFA), Acute Physiology and Chronic Health Evaluation II (APACHE II), and Bedside Index of Severity in AP (BISAP) score for risk stratification in AP. Ranson⁸ and Glasgow⁹ scores were developed for predicting the severity of AP, BISAP^10,11 and SOFA¹² scores were for the mortality of AP, APACHE II^13,14 score was for both severity and mortality. However, the wide clinical application of these score systems was restricted to their complex calculation, lagging assessment, and suboptimal accuracy.

Nomogram is a visual statistical model which was widely used to evaluate the prognosis of related diseases in medical research. In the era of “big data”, medical databases are increasingly used for scientific research. Although several studies have developed nomograms to predict in-hospital mortality of AP based on critical care databases,^15–17 small sample size and lacking further external validation were their main limitation. Moreover, the clinical applicability of traditional nomogram may be restricted to time-consuming and lacking an exact calculation. Therefore, the aim of the study was as follows: First, we developed a dynamic nomogram based on the data from the Medical Information Mart for Intensive Care-IV (MIMIC-IV) database to predict in-hospital mortality of AP patients admitted to the Intensive Care Unit (ICU) and compared the performance of the nomogram and traditional score systems. Furthermore, the data from the Electronic Intensive Care Unit Collaborative Research Database (eICU-CRD) was used for external validation to improve the level of evidence.

Materials and Methods

Data Source and Outcome

This retrospective cohort study extracted critical care data from the MIMIC-IV (v.2.0) database. It contains over 70,000 ICU patients admitted to the Beth Israel Deaconess Medical Center from 2008 to 2019.¹⁸ The MIMIC-IV, a large, single-center, and de-identified patient database, includes plenty of patient information such as demographics, bedside monitors, International Classification of Diseases codes version 9 and version 10 (ICD-9 and ICD-10) revision diagnoses, laboratory data, therapeutic measures, radiology reports, and clinical outcome.¹⁹ In addition, the eICU-CRD (version 2.0), a multicenter database composed of more than 200,000 ICU admissions in the United States, was served as an independent external validation set.²⁰ After an in-depth study of a training course “Protecting Human Research Participants”, an author (Kang Zou) obtained permission to access the two databases for research purposes (certification number: 11739201). The ethics committee of the Affiliated Hospital of Southwest Medical University, China, approved the study and waived the ethics approval as this retrospective study was based on public databases and personal information provided in the databases was de-identified. The primary outcome of this study was the all-cause in-hospital mortality of patients with AP admitted to ICU.

Study Patients

All patients in the MIMIC-IV and eICU-CRD who were diagnosed as AP were recruited (only including the ICU admission). The patients were identified by a manual search of ICD-9 (577.0) and ICD10 (K85) codes. The exclusion criteria were as follows: (1) Age below 18, (2) Repeat ICU admission, (3) Primary outcome missing. The flow chart of patient selection is shown in Figure 1.

Figure 1 Flow chart of patients selection.

Data Collection and Variable Extraction

For this study, we collected information from the electronic medical record including the general demographic, vital signs, laboratory test results, advanced life support strategy, accompanied diseases, and score systems. The process of data collection was following Cosgriff²¹ with a slight modification. The following variables were excluded: (1) Variables missing over 30%, (2) Variables that were not shared by the two databases. Finally, the following demographic data collected admission were extracted: age, gender, race, and hospital expire flag (in-hospital death). The mean over all values of the vital sign in the first 24 hours was selected: heart rate, respiratory rate, systolic blood pressure (SBP), diastolic blood pressure (DBP), oxyhemoglobin saturation (Spo2), mean blood pressure (MBP), and temperature. The maximum value within 24 hours of admission was used for the following variables: red cell distribution width (RDW), creatinine, blood urea nitrogen (BUN), total bilirubin (TBIL), aminotransferase alanine (ALT), aminotransferase aspartate (AST), and aniongap. The minimum value within the first 24 hours was extracted for bicarbonate, albumin, platelets, hemoglobin, hematocrit, red blood cell (RBC), calcium, and chloride. Both the minimum and maximum value within 24 hours after admission was selected for the following variables: the maximum of white blood cells (WBC_max), the minimum of white blood cells (WBC_min), the minimum of sodium (sodium_min), the maximum of sodium (sodium_max), the minimum of potassium (potassium_min), the maximum of potassium (potassium_max). In addition, advanced life support (mechanical ventilation and renal replacement therapy), comorbidities (hypertension and diabetes), and score systems (BISAP and APACHE II score) were also included. For missing values, a K-nearest neighbor classification algorithm was applied to minimize the impact of missing data on results.

Statistical Analysis

The continuous variables with a normal distribution were presented as the mean ± standard deviation (), and Student’s t-test was used for comparisons between groups, while those having an abnormal distribution were expressed as the median and interquartile ranges [M (Q1, Q3)] and compared with the Mann–Whitney U rank-sum test. Categorical variables were expressed by counts and percentages [n (%)], and χ2 test or Fisher’s exact test was used for comparison between groups. All patients from MIMIC-IV were randomly assigned into the development group and validation group according to a ratio of 8:2. The univariate logistic regression and least absolute shrinkage and selection operator (LASSO) regression method was used for screening the best predictors in the development group. On the basis of the screened variables, multivariate logistic regression was used to build the nomogram in the development group and then validated in the validation group. Furthermore, the data from eICU-CRD was used for external validation. As the nomogram was developed and validated in ICU patients, the results cannot be generalized to patients in a general ward well patient with mild AP. Various indicators, including the area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and Yoden index were used to assess the discrimination of this predictive model. Delong test was utilized to compare the predictive performance of the nomogram and score systems. Moreover, the calibration of nomogram was evaluated by the calibration curve, and the clinical value of the predictive model was also evaluated through decision curve analysis (DCA). All statistical analyses were carried out using R software (v.3.6.3), and statistical significance was set at p < 0.05.

Results

Baseline Data from the MIMIC-IV Database

As shown in Table 1, a total of 1030 patients extracted from the MIMIC-IV database were included in this study, in which, 133 patients were death and 897 survival during hospitalization, respectively. Among all the characteristics, age (p < 0.001), respiratory rate (p < 0.001), temperature (p < 0.001), SBP (p < 0.001), DBP (p < 0.001), MBP (p < 0.001), Spo2 (p < 0.001), WBC_max (p < 0.001), WBC_min (p = 0.023), hematocrit (p < 0.001), RDW (p < 0.001), RBC (p < 0.001), Hemoglobin (p < 0.001), platelets (p = 0.002), AST (p < 0.001), albumin (p < 0.001), TBIL (p < 0.001), BUN (p < 0.001), creatinine (p < 0.001), aniongap (p < 0.001), bicarbonate (p < 0.001), calcium (p = 0.002), potassium_min (p < 0.001), potassium_max (p < 0.001), mechanical ventilation (p < 0.001), renal replacement therapy (p < 0.001), BISAP score (p < 0.001), and APACHE II score (p < 0.001) differed significantly between the two groups.

Table 1 Summary of Demographic and Clinical Features of MIMIC-IV Patients with Acute Pancreatitis

A Predictive Nomogram Development

There were 824 and 206 patients in the development and validation group. Supplementary Table 1 displayed the baseline characteristics of the two groups. There were no significant differences between the two data sets in any variable (P > 0.05), which meant that the distribution of clinical features for the two data sets was well balanced. Univariate logistic regression analysis for variable selection in the development group is shown in Supplementary Table 2. LASSO regression was used to further screen variables with non-zero coefficients among the results of univariate logistic regression. It showed that the best predictors of in-hospital mortality in AP were age, oxygen saturation, SBP, temperature, respiratory rate, RDW, Wbc_max, bicarbonate, BNU, albumin, and TBIL (Figure 2).

Figure 2 Selection of predictive factors using the least absolute shrinkage and selection operator logistic regression algorithm. Least absolute shrinkage and selection operator coefficient profiles of the 26 candidate variables. The best value was determined by the two dashed vertical lines which were drawn according to the minimum mean-square error criterion (left dashed line) and the standard error criterion (right dashed line). In the present study, 11 predictors were selected according to the minimum standard error criterion (λ=0.028).

Based on the above variables, multivariate logistic regression was used to establish a predictive model represented by a nomogram (Figure 3A). Multivariate logistic regression showed that there were 8 independent predictors of in-hospital mortality in AP, as shown in Table 2.

Table 2 Risk Factors Selected After Multivariate Logistic Regression Analysis

Figure 3 Nomogram to estimate the risk of in-hospital mortality in acute pancreatitis. (A) Ordinary nomogram. (B) The schematic outline of dynamic nomogram. To obtain the accurate probability of in-hospital mortality, select the value of each variable and then click on the “Predict” button. The probability and 95% confidence interval will receive. For properly function of the site, please click the “Quit” button after finish predicting. Abbreviations: SBP, systolic blood pressure; RDW, red cell distribution width; TBIL, total bilirubin; BUN, blood urea nitrogen.

To obtain the in-hospital mortality risk of an individual patient, we can get an exact variable value according to clinic characteristics of the patient, and then draw a vertical line to the “Points” axis to get points of one variable. Subsequently, we can obtain total points by summing all of the points for 8 variables. Finally, the probability of in-hospital mortality for the patient can be located by drawing a vertical line from the “Total Points” axis to the “Prob of in-hospital mortality” axis. A precise probability of in-hospital mortality can be obtained from the dynamic nomogram by visiting https://model.sci-inn.com/KangZou/. For example, an 83-year-old patient with AP who was admitted in ICU, the mean SBP, respiratory rate, and Spo2 were 96mmHg, 28 breaths/min, and 87%, respectively, in the first 24 hours. The maximum value within 24 hours of RDW, TBIL and BUN were 16%, 3mg/dL, and 39mg/dL, respectively. The minimum of albumin in the first 24 hours was 2g/dL. We can obtain the exact probability and 95% confidence interval of in-hospital mortality of this patient was 0.842 (0.663–0.944), as shown in Figure 3B.

Evaluation and Validation of the Nomogram

The receiver operating characteristic (ROC) curve and AUC of the nomogram and score system in both the development and validation groups were calculated and compared (Figure 4A and B). In the development group, the AUC of the nomogram, APACHE II, and BISAP scores were 0.859, 0.781, and 0.700, respectively. The discrimination of our nomogram was significantly higher than APACHE II (p = 0.001) and BISAP (p < 0.001) score. In the validation group, the AUC was 0.871, 0.841, and 0.690, respectively. Compared to BISAP score, our nomogram performed better in the discrimination (p = 0.013). However, there was no significant difference between APACHE II score and the nomogram (p = 0.537) (Table 3).

Table 3 Comparison of the Nomogram Model and Score System for Predicting the in-Hospital Mortality of Acute Pancreatitis in the Development, Internal Validation and External Validation Set

Figure 4 The discrimination, calibration, and clinical value of the nomogram. (A and B) Receiver operating characteristic curves in the development and validation group by Nomogram and Score system. A, Development group. B, Validation group. (C) Calibration curves of development group. (D) Calibration curves of validation group (Briser Score = 0.08). (E and F) Decision curve analysis of Nomogram and Score system. E, Development group. F, Validation group. Model 1, nomogram; model 2, APACHE II score; model 3, BISAP score. Abbreviations: AUC, the area under the receiver operating characteristic curve; APACHE II, Acute Physiology and Chronic Health Evaluation II; BISAP, Bedside Index of Severity in Acute Pancreatitis.

The calibration curves constructed through 1000 bootstrap replicates is shown in Figure 4C and D. The Brier score was 0.08 in validation group. The calibration curves showed good consistency between the predicted probability and the actual probability in both the development and validation groups.

The DCA in Figure 4E and F indicated that the nomogram, APACHE II, and BISAP score showed more benefit than the treat-all-patients or treat-none scheme. Moreover, compared with the APACHE II and BISAP score, the net benefit of our nomogram was significantly higher, suggesting that our nomogram was more clinically useful.

To improve the generalization of the model, we used the data from eICU-CRD for external validation. There were 514 patients extracted from eICU-CRD, of which 39 patients were death and 475 survival. In the external validation set, age (p = 0.003), respiratory rate (p = 0.010), SBP (p < 0.001), DBP (p < 0.001), MBP (p < 0.001), WBC_max (p = 0.006), hematocrit (p = 0.001), RDW (p < 0.001), RBC (p < 0.001), hemoglobin (p < 0.001), albumin (p < 0.001), TBIL (p = 0.009), BUN (p < 0.001), creatinine (p < 0.001), aniongap (p = 0.042), potassium_max (p = 0.004), mechanical ventilation (p < 0.001), renal replacement therapy (p < 0.001) were significantly different between the survival and death patients (Supplementary Table 3). The ROC of the external cohort is shown in Figure 5A. The AUC of the nomogram in the external cohort was 0.847. The calibration curve is shown in Figure 5B. The Brier score in the external cohort was 0.077, which indicated that the predicted probability of nomogram was also highly in agreement with the actual probability in the external cohort.

Figure 5 The discrimination and calibration in external validation set based on the data of Electronic Intensive Care Unit Collaborative Research Database. (A) Receiver operating characteristic curve. (B) Calibration curve (Briser Score = 0.077).

Discussion

AP is a heterogeneous disease with a mild to severe course. Most severe AP patients require intensive care and advanced life support. Determining the prognosis of AP is the base of both patient-centered care and shared decision-making, such as choice of treatment strategies and informed consent.¹⁹ This present study constructed a dynamic nomogram with the data from MIMIC-IV database. Based on multivariate regression analyses, 8 predictors including age, oxygen saturation, SBP, respiratory rate, RDW, albumin, TBIL, and BUN were the independent prognostic factors for in-hospital mortality of AP. The nomogram showed great performance in predicting in-hospital mortality of AP and external validation data from eICU-CRD received equally satisfactory results. In addition, compared to the traditional score system, our nomogram showed better discrimination and clinical practicability.

There were several prognostic scores systems such as APACHE II, BISAP, and Ranson score performed well in predicting the mortality of AP.^22–24 However, these score systems were mainly applied to scientific research due to various limitations, which led to simple and effective tools still lacking in clinical work. The APACHE II score involved multiple variables and complex computation. In addition, the generalizability of APACHE II score across diseases was poor because it was not designed specifically for AP. The BISAP score involved a subjective assessment of mental status.¹¹ The SOFA score consisted of five variables scored within 24 hours of admission, while it was simplistic to use.²⁵

Both MIMIC and eICU-CRD are publicly accessible and well-established research databases, analysis can be accessed without restriction once a data use agreement is accepted. It is obvious that these medical critical care databases were widely used in areas of clinical research and education all over the world.²⁶ In the past several years, a large number of researchers put efforts to probe prognostic analysis,^27,28 length of stay,²⁹ prediction of severity,¹⁷ and treatment strategy^30,31 of AP based on the two databases. Currently, several studies have developed nomograms for the prediction of in-hospital mortality in AP based on critical care database. Liu¹⁵ and Li¹⁶ constructed nomograms to predict in-hospital mortality of AP based on Medical Information Mart for IntensiveCare-III (MIMIV-III) database and eICU-CRD, respectively. However, both two traditional nomograms lack external validation. A study by Xu¹⁷ also developed an ordinary nomogram combining serum TBIL and albumin with other significant features based on MIMIC-III and eICU-CRD, which also show favorable discrimination. But the predictive variables selected by Xu X mainly focused on endogenous antioxidant indexes and the SOFA score included in the nomogram needs further calculation. In addition, MIMIC-III includes data more than 50,000 patients from 2001 to 2012 while over 70,000 ICU admissions in MIMIV-IV from 2008 to 2019.^18,26 Compared with MIMIC-III, MIMIC-IV has a great boost in the number of cases and the data of MIMIC-IV are closer to the medical level of the last decade. Therefore, we used MIMIC-IV database to develop a dynamic nomogram and exerted internal validation. The predictors in our nomogram only involved vital signs and common laboratory indicators which were easily accessed after admission. It can guide physicians to exert risk stratification and make clinical decisions for AP patients.

Older age was regarded as a well-established marker of poor prognosis^32,33 in AP and was included in the APACHE II score, Ranson score, the BISAP score, and the Japanese severity score as a predictor. The likely explanation was that there were higher odds of comorbid diseases with increasing age.³⁴ Age was also demonstrated as an independent risk factor in the present study. RDW is a laboratory parameter widely used to evaluate the extent of erythrocyte anisocytosis. In addition to being a traditional marker of anemia,³⁵ studies suggested that it can be used as an indicator of inflammation and was related to multiple links of inflammatory response in recent years.^36–38 Therefore, there was a significant association between RDW and the mortality of patients with AP.^39,40 With a cutoff value of 14.8 for RDW, approximately 77% of the deaths could be correctly predicted.⁴¹ Similarly, RDW was also correlated with the in-hospital mortality of AP and weighed heavily in our nomogram. BUN, albumin, and TBIL were significant predictors for mortality of AP, which were already confirmed by numerous studies.^{11,17,42–44} Current nomograms which were established for mortality of AP by other researchers usually included one or more of them as predictors,^15,16,45 which were consistent with our nomogram. AS for SBP, respiratory rate, and oxygen saturation as a reflection of patients’ vital signs, it can surely be used as a predictor of mortality.

This retrospective analysis of a large sample from multiple centers explored risk factors of mortality in AP and developed a nomogram that can predict in-hospital mortality in patients with AP precisely. The predictive variables in the nomogram were easily accessible within 24 hours after ICU admission. In addition, we improved the limitations of lack of external validation and small sample size commonly found in previous studies. Finally, on the basis of the traditional ordinary nomogram, we further constructed a dynamic nomogram, which can facilitate rapid and accurate risk stratification and clinical decision-making for clinicians. Therefore, our nomogram model should be reliable and user-friendly. However, there were also several limitations in the present study. First, selection bias was inevitable in this retrospective study. Prospective studies are needed to improve the level of evidence. Second, we excluded the variables with more than 30% of observations missing to ensure the accuracy of our study. It led to some important laboratory index such as C-reactive protein, procalcitonin being excluded due to significant missing data. Moreover, in the acute setting, albumin may not be a reliable indicator and its levels can be influenced by nutritional or hepatic dysfunction issues, which may not necessarily reflect the severity of AP. Third, the mortality between the two databases was different. There were only about 40 deaths in the eICU-CRD. However, we included 8 variables in nomogram, which meant the principle of 10-fold Events per variable was not followed in external validation set. Finally, data was extracted from the medical records of patients and the performance of the nomogram was closely related to the accuracy of the recording.

Conclusion

We constructed a dynamic nomogram and make internal and external validation based on MIMIC-IV and eICU-CRD. The link of the dynamic nomogram is https://model.sci-inn.com/KangZou/. The nomogram showed good discrimination, calibration, and clinical practicability in predicting the in-hospital mortality of AP. However, prospective studies are still necessary in order to bring such predictive models into clinical application earlier.