Machine Learning Predictive Model for Septic Shock in Acute Pancreatitis with Sepsis

Introduction

Acute pancreatitis (AP) complicated by sepsis is a complex clinical condition often associated with high clinical instability and risks. Up to 40–70% of AP patients will develop infections related to pancreatitis in the later stages or progress to sepsis in severe cases,^1,2 which is associated with higher mortality rates and adverse prognosis,³ especially when patients progress to septic shock, the mortality rate can reach up to 80%.⁴ Early admission to Intensive Care Unit (ICU) treatment can more effectively manage high-risk patients, halt the progression of sepsis, and thereby improve the clinical prognosis of patients.⁵ Currently, the early identification of high-risk patients who may progress to septic shock remains a challenging task. Clinicians rely on symptoms, signs, and clinical examinations for initial assessments. Existing scoring systems such as the Bedside Index for Severity in Acute Pancreatitis (BISAP)^6,7 can effectively predict the severity and prognosis of AP. Other scoring systems such as the Acute Physiology and Chronic Health Evaluation (APACHE) II,⁸ Sequential Organ Failure Assessment (SOFA),⁹ and quick Sequential Organ Failure Assessment (qSOFA) should be used for risk stratification and consideration of sepsis in emergency department patients.¹⁰ However, these scoring systems have shown limited effectiveness in predicting septic shock.^11–14 Their effectiveness in identifying sepsis-complicated AP and predicting the occurrence of septic shock remains uncertain, and there are no related studies.

Machine learning (ML) techniques have shown significant potential in the medical field, particularly in early disease identification and prediction.^15–18 The study indicates that using artificial intelligence algorithms for septic shock risk identification and diagnosis can improve the early detection rate of septic shock by 32%. Furthermore, these models exhibit a high predictive accuracy in the first 12 hours of patient diagnosis (AUC 0.94, sensitivity 0.87, specificity 0.87), enabling the acquisition of a crucial window for clinical intervention.¹⁹ Similarly, the use of ML techniques also offers significant advantages in the prediction and assessment of septic shock.^17,20

Therefore, this study aims to develop an early predictive model using ML techniques that can accurately predict the occurrence of septic shock in patients with AP complicated by sepsis. It aims to provide emergency physicians with an accurate tool for more effective management of these high-risk patients, allocate intensive care resources judiciously, and prevent the occurrence of septic shock, thereby potentially improving the clinical outcomes of patients.

Materials and Methods

Study Design

This retrospective cohort study complied with the Declaration of Helsinki, and the study protocol was approved by the Human Ethical Committee of West China Hospital of Sichuan University, which is also responsible for overseeing ethical matters at Chengdu Shangjin Nanfu Hospital, an affiliated branch of West China Hospital of Sichuan University (No. 2019-334). In addition, approval has been obtained from the Ethics Committee of Chengdu First People’s Hospital (No. 2020-048).

Study Population

Patients with AP complicated by sepsis, who were admitted to the Emergency Departments of Sichuan University West China Hospital, Chengdu First People’s Hospital, and Chengdu Shangjin Nanfu Hospital between January 1, 2017, and September 30, 2019, were enrolled in the study.

Inclusion and Exclusion Criteria

Patients were included in this study if they were: 1, Age ≥ 18 years old; 2, Met the 2012 revised Atlanta criteria for acute pancreatitis diagnosis;²¹ 3, Met the diagnostic criteria for Sepsis 3.0.²² Patients were excluded if they had: 1, Chronic pancreatitis; 2, Patients with concomitant hematological disorders or terminal-stage malignant tumors; 3, AP caused by trauma; 4, Pregnancy or perinatal period; 5, Patients who required immediate cardiopulmonary resuscitation for cardiac arrest upon arrival or had experienced cardiac arrest before admission and were currently in the post-cardiac arrest syndrome (PCAS) phase; 6, Patients who were already receiving vasopressors medications to maintain blood pressure upon admission; 7, Patients with pre-existing chronic heart, lung, liver, or kidney failure (eg, severe heart failure classified as New York Heart Association Class IV, chronic respiratory failure requiring long-term home nasal cannula/face mask oxygen therapy, non-invasive positive pressure ventilation, Child Class C liver function, confirmed chronic kidney failure); 8, Patients with uncontrolled bleeding requiring blood transfusion support within the past 24 hours; 9, Incomplete clinical data or lacking follow-up information.

Diagnostic Criteria

Patients with septic shock can be identified with a clinical construct of sepsis with persisting hypotension requiring vasopressors to maintain MAP ≥65 mmHg and having a serum lactate level >2 mmol/L (18 mg/dL) despite adequate volume resuscitation.¹⁰

Data Collection

Patient demographic data, vital signs at admission, and laboratory results are collected within 24 hours of admission. Demographic data include gender, age, etiology (gallstones, alcohol, hyperlipidemia, and other types excluding trauma), and disease duration.

Vital signs at admission include temperature (T), respiratory rate (RR), heart rate (HR), mean arterial pressure (MAP), systolic blood pressure (SBP), diastolic blood pressure (DBP), peripheral oxygen saturation (SpO₂), and Consciousness status.

Laboratory results within 24 hours of admission include measurements of pH, partial pressure of carbon dioxide (PCO₂), partial pressure of oxygen (PO₂), oxygenation index (OI), lactate (LAC), base excess (BE), bicarbonate (HCO3⁻), ionized calcium, white blood cell count (WBC), platelet count (PLT), hemoglobin (Hb), hematocrit (HCT), neutrophil percentage (N%), total bilirubin (TBIL), direct bilirubin (DBIL), alanine aminotransferase (ALT), aspartate aminotransferase (AST), albumin (ALB), glucose (GLU), blood urea nitrogen (BUN), creatinine (Cr), cystatin C (Cys-C), uric acid, cholesterol (CHOL), triglycerides (TG), low-density lipoprotein cholesterol (LDL-C), high-density lipoprotein cholesterol (HDL-C), amylase (AMY), lipase (LIP), serum calcium, sodium ions (Na⁺), potassium ions (K⁺), prothrombin time (PT), activated partial thromboplastin time (APTT), fibrinogen (Fib), D-dimer and pleural effusion.

Calculate admission qSOFA score, SOFA score, APACHE II score, and BISAP score.

Quality Control

The research team conducted regular audits as part of their data quality control efforts. The data underwent double-checks, and when inconsistencies were found during data verification, the medical team conducted validation to ensure its authenticity and reliability. Subsequent follow-ups were carried out by dedicated personnel.

Study Follow-Up and Primary Endpoint

All patients with acute pancreatitis complicated by sepsis underwent a 28-day follow-up. The primary endpoint was the occurrence of septic shock during the follow-up period.

Data Preprocessing

We divided our dataset into two subsets: 70% for the training dataset and 30% for the independent testing dataset. We utilized the widely used Feature Explorer (FAE, V 0.5.4)²³ to develop our model in the training dataset and evaluate its performance in the testing dataset.

To address the class imbalance issue in the training dataset, we performed upsampling by randomly repeating cases to achieve a balanced ratio of positive to negative samples. We then applied three types of normalization to the feature matrix: Min-Max, Z-score, and Mean. Due to the high dimensionality of the feature space, we used principal component analysis (PCA) and Pearson correlation coefficient (PCC) methods for dimensionality reduction. We evaluated feature similarity and maintained feature independence by eliminating one feature from pairs with a PCC exceeding 0.80. Prior to model construction, we employed Kruskal–Wallis (KW), Analysis of Variance (ANOVA), Recursive Feature Elimination (RFE), and Relief methods for selecting the most informative features.

Classifications and Model Selection

We employed 10 ML classifiers from the Feature Explorer (FAE, V 0.5.4) software package to select features with optimal efficacy in distinguishing septic shock from non-septic shock, based on 10-fold cross-validation results. These 10 ML classifiers include support vector machine (SVM), linear discriminant analysis (LDA), auto-encoder (AE), random forests (RF), linear regression (LR), logistic regression using Lasso (LR Lasso), AdaBoost (AB), decision tree (DT), Gaussian process (GP), and naive Bayes (NB). In this step, we used bootstrap sampling with 1000 iterations on the cross-validation dataset, obtaining average classification results. To select a simpler and more generalizable model, we applied the “one-standard error” rule. Finally, the testing dataset was employed to evaluate the generalizability of results from the training dataset and estimate unbiased classification accuracy without modifying the identified model parameters.

Evaluations

We assessed the model’s performance using receiver operating characteristic curve (ROC) analysis and calculated the area under the ROC curve (AUC) for quantification. Additionally, we computed accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) at the cutoff value that maximized the Youden index. Furthermore, we estimated 95% confidence intervals using bootstrapping with 1000 samples. All of these procedures were conducted using Feature Explorer Pro (FAE, V 0.5.4) in Python (3.7.6).²³ Conducted Delong’s test using MedCalc to compare the performance of the machine learning model with scoring systems. A p value <0.05 was considered statistically significant.

Results

A total of 604 patients with AP complicated by sepsis were included in this study. In the entire cohort, 395 males and 209 females were found, with an average age of 49.45 ± 14.67 years. During hospitalization, a total of 81 cases were observed to progress to septic shock within 28 days (35 deaths, 43.21%), while 523 cases did not (12 deaths, 2.3%). The entire dataset was randomly divided in a 7:3 ratio: 423 cases for the training dataset (57 positive and 366 negative cases) and 181 cases for the independent testing dataset (24 positive and 157 negative cases).

We observed that the AE model, based on Mean normalization, PCC, and RFE feature selection, which relies on 11 selected features, achieved the highest AUC on the validation dataset, reaching a value of 0.900. Additionally, the model demonstrated an accuracy of 0.868 in this context. When applied to the testing dataset, the model maintained strong performance with an AUC of 0.879 and an accuracy of 0.790. Table 1 and Figure 1 provide details on clinical statistics for predicting the occurrence of septic shock within 28 days and the selected features, respectively. The importance ranking of the 11 selected features for ML are as follows: disease duration, heart rate (HR), respiratory rate (RR), consciousness status, lactate (LAC), base excess (BE), white blood cell count (WBC), albumin (ALB), blood urea nitrogen (BUN), cystatin C (Cys-C), high-density lipoprotein cholesterol (HDL-C).

Table 1 Clinical Statistics of the AE Model in Predicting the Occurrence of Septic Shock Within 28 Days

Figure 1 Performance of models generated using Pearson correlation coefficient (PCC) analyses, and RFE and auto-encoder (AE) classification. (A) Receiver operating characteristic (ROC) curves of this model on different datasets. (B) FeAture Explorer software suggested a candidate 11-feature model according to the “one-standard error” rule. (C) The contribution of features in the final model. Abbreviations: HR, heart rate; RR, respiratory rate; LAC, lactate; BE, base excess; WBC, white blood cell count; ALB, albumin; BUN, blood urea nitrogen; Cys-C, cystatin C; HDL-C, high-density lipoprotein cholesterol.

The AUCs for CV training, CV validation, training, and testing datasets, utilizing two-dimensional reduction methods and four feature selection techniques, all exceeded 0.600 with the AE classification (Figure 2). This study conducted an exploratory analysis to assess the impact of various feature selection methods on the performance of predictive models for different ML classifiers. The results revealed the following patterns: When ANOVA feature selection was employed, the AUCs of the 10 ML classifiers for the training, CV validation, cross-validation training, and testing datasets were all above 0.800, except for the AB and DT classifiers (Figure 3A). Similarly, when using KW feature selection, the AUCs of the 10 ML classifiers for the training, CV validation, cross-validation training, and testing datasets were all above 0.800, except for the AB and DT classifiers (Figure 3B). Utilizing Relief feature selection resulted in AUCs above 0.600 for the 10 ML classifiers in the training, CV validation, cross-validation training, and testing datasets, except for the DT classifier (Figure 3C). Finally, RFE feature selection led to AUCs above 0.800 for the 10 ML classifiers in the training, CV validation, cross-validation training, and testing datasets, except for the DT classifier (Figure 3D). The study found that the AE model demonstrated robust performance on both cross-validated training and independent testing datasets with the four mentioned feature selections. This indicates that the AE model is effective for identifying patients at high risk of septic shock.

Figure 2 Areas under the curve (AUCs) of the different datasets using Pearson correlation coefficient (PCC) and principal component analysis (PCA) methods, four feature selections, and auto-encoder (AE) classification. (A) Analysis of Variance (ANOVA), (B) Kruskal–Wallis (KW), (C) Relief, and (D) Recursive Feature Elimination (RFE).

Figure 3 Areas under the curve (AUCs) of different datasets using 10 machine learning classifications. Feature selections using (A) analysis of variance (ANOVA), (B) Kruskal–Wallis (KW), (C) Relief, and (D) Recursive Feature Elimination (RFE). Abbreviations: SVM, support vector machine; LDA, linear discriminant analysis; AE, auto-encoder; RF, random forests; LR, linear regression; LR Lasso, logistic regression using Lasso; AB, AdaBoost; DT, decision tree; GP, Gaussian process; NB, naive Bayes.

We compared the performance of the AE model with traditional scoring systems (SOFA, qSOFA, APACHE II, and BISAP scores) in the early prediction of septic shock occurrence within 28 days of admission for patients with AP complicated by sepsis using the testing dataset in this study. The results demonstrate the superior predictive performance of the AE model compared to traditional scoring systems (Table 2). The pairwise comparison results of ROC curves between the AE model and the SOFA, qSOFA, APACHE II, and BISAP scoring systems are detailed in Table 3. ROC curve comparisons are depicted in Figure 4.

Table 2 AUC for AE Model Compared to Traditional Scoring Systems

Table 3 Pairwise Comparison of ROC Curves Between the AE Model and Traditional Scoring Systems

Figure 4 The ROC curve comparison between AE model and traditional scoring systems. Abbreviations: AE, auto-encoder; APACHE II, Acute Physiology and Chronic Health Evaluation II; BISAP, Bedside Index of Severity in Acute Pancreatitis; qSOFA, quick Sequential Organ Failure Assessment; SOFA, Sequential Organ Failure Assessment.

Discussion

This study included patients with AP complicated by sepsis from three comprehensive hospitals to ensure a diverse analysis of the patient population. Through various advanced ML classifications, a predictive model for the early identification and prognosis of septic shock in patients hospitalized within 28 days of AP complicated by sepsis was developed, and promising results were demonstrated.

In this study, a comprehensive dataset comprising 604 patients was analyzed. Various feature selection techniques were employed, including dimensionality reduction using PCC and PCA, as well as feature selection methods such as ANOVA, KW, RFE, and Relief, to identify the most informative features for predicting septic shock. Ten different ML methods were utilized, with the AE classification performing the best. AE is a neural network-based algorithm for constructing predictive models.^24–26 It is an unsupervised learning model that leverages techniques like backpropagation and optimization methods like gradient descent to use input data X itself as supervision, guiding the neural network in learning a mapping relationship to obtain a reconstructed output X^R. The study results demonstrated that the predictive model using the AE algorithm achieved an AUC of up to 0.900 on the validation dataset and maintained strong performance on the test dataset with an AUC of 0.879. In another study by Ramos et al, two fully unsupervised learning approaches were employed to predict the occurrence of septic shock in the Intensive Care Unit (ICU). The results showed that the unsupervised approach using variational autoencoder achieved an AUC of 0.82 and an F1-score of 0.65.²⁷

In this study, the feature selection output of the ML model ranked the duration of illness as the most important feature. AP has a disease course that is divided into early (within 2 weeks of onset) and late stages (2 weeks or more after onset), each corresponding to distinct peaks in mortality during the course of the disease.^28,29 However, these two stages overlap and do not have a clear boundary. Patients in the late stage of the disease are more likely to progress to septic shock, which not only worsens their condition but also prolongs their hospital stay, increases the economic burden of medical care, and adversely affects their prognosis. Another multicenter retrospective study’s results demonstrated that InSight,³⁰ a gradient tree boosting ML algorithm, utilizes features including systolic blood pressure, diastolic blood pressure, heart rate, respiratory rate, peripheral capillary oxygen saturation, and temperature for prediction. The results show that InSight predicts septic shock with an AUROC of 0.96 (95% CI 0.94 to 0.98) within 4 hours before the onset of septic shock. Among the features included in the InSight algorithm, heart rate and respiratory rate, ranked second and third in the feature importance ranking of the AE classification in this study, respectively. This consistency aligns with early clinical presentations of septic shock. Furthermore, alterations in consciousness, as one of the early clinical signs of septic shock, ranked fourth in the feature importance ranking in this study. Seymour et al retrospectively analyzed 148,907 suspected infection cases in 2015 and found that factors such as a respiratory rate ≥22 breaths per minute, systolic blood pressure ≤100mmHg, and altered mental status were valuable predictors of sepsis occurrence.¹⁰ Subsequently, they proposed the qSOFA score for sepsis screening and prognostic evaluation. Lactate serves as a fundamental biomarker for septic shock, being the most widely recognized indicator of tissue hypoperfusion/shock. Its elevation precedes a drop in blood pressure, aiding in the early diagnosis of shock.^10,31 Elevated lactate levels reflect cellular dysfunction in sepsis,³² with higher levels predicting higher mortality.³³

Compared to traditional scoring systems such as SOFA, qSOFA, APACHE II, and BISAP score, the AE model exhibits a significant advantage. This indicates the model can reasonably predict the risk of patients with AP complicated by sepsis progressing to septic shock within 28 days of hospitalization.

Limitations

Firstly, despite our best efforts to minimize bias, there remains an unavoidable risk of some degree of selection bias.

Secondly, with around 70% of the data sourced from the Emergency Department of West China Hospital, Sichuan University, there is a potential for regional bias in the sample, limiting the external applicability of the model. Validation in a more diverse healthcare setting is essential.

Moreover, despite the remarkable performance of the AE model in early septic shock prediction, it should not completely replace traditional scoring systems and clinical judgment.

Furthermore, the exclusion of data on inflammatory markers, cardiac markers, and radiological results, along with the optimization and inconsistency of treatment plans, may impact the comprehensiveness and in-depth understanding of septic shock prediction by the model.

Lastly, future research could explore the integration of more advanced models or employ ensemble modeling to enhance prediction accuracy, bringing a more forward-looking and innovative perspective to the field.

Conclusions

The AE model performed well in predicting septic shock within 28 days of hospitalization for AP patients with sepsis. It outperformed traditional scoring systems, aiding physicians in early identification of high-risk patients, making timely medical decisions and enabling stratified treatment in the ED.