Development of a Nomogram for Predicting Mortality Risk in Sepsis Patients During Hospitalization: A Retrospective Study

Introduction

Sepsis is a severe and life-threatening disease that brings a tremendous public health burden. More than 1,700,000 cases are diagnosed annually across the United States.¹ Its mortality increases when sepsis is further aggravated and progresses to septic shock. Previous studies have indicated that the mortality rate of inpatients with sepsis is about 17%.² Septic shock is a subset of sepsis, with a mortality rate of more than 40%.³ Early diagnosis of patients with sepsis and active intervention measures can reduce the risk of progression to septic shock and mortality, indicating great clinical significance.

At present, the severity score and mortality estimation of sepsis patients are primarily dependent on the acute physiology and chronic health assessment (APACHE), systemic inflammatory response syndrome standard (SIRS), quick sequential organ failure assessment (qSOFA) and sequential organ failure assessment score (SOFA). These have certain deficiencies in assessing sepsis. The APACHE score established earlier can be utilized for all critically ill patients.⁴ Some studies have pointed out that the APACHE score underestimates the death risk of sepsis patients.^5–8 The SIRS standard is mainly used to determine whether patients have systemic inflammatory responses based on the changes in white blood cell count, body temperature, heart rate, and respiratory rate. Due to poor differentiation, this standard cannot effectively predict the prognosis of sepsis.⁹ The qSOFA score is simple, easy to operate, and more specific than the SIRS standard in evaluating the prognosis of sepsis. However, it has the disadvantage of low sensitivity and, therefore, cannot be a suitable tool for predicting the prognosis of sepsis.¹⁰ The development of the SOFA score is primarily used to describe the occurrence of multiple organ dysfunction. Compared with qSOFA, SOFA score has a higher accuracy in determining the mortality of sepsis.¹¹ However, the SOFA scoring system is not specifically developed to predict the mortality outcome of sepsis.¹² Thus, the accuracy of the SOFA score in predicting the short-term mortality of sepsis patients is dissatisfactory.¹³

Without active intervention, sepsis will progress to septic shock, leading to higher mortality. Consequently, developing a tool to predict the short-term mortality of patients with sepsis is necessary. Nomogram is a very friendly visualization tool to present how much the involved variables contributed to the death risk, which is more applicable and convenient than the traditional scoring system.¹⁴ This study aims to establish a reliable and suitable model for predicting the death risk of hospitalized patients with sepsis. The variables involved in the prediction model were collected within 24h following hospitalization, which could identify the patients with high death risk at an early stage.

Materials and Methods

Patient Extraction from Medical Record Information Mining Database

The inpatients’ information was collected from the medical record information mining database based on the clinical records of patients who were admitted to Affiliated Dongyang Hospital of Wenzhou Medical University, Zhejiang Province, China. The technology for database construction was supported by Le9 Healthcare Technology Co. Ltd.¹⁵ The written informed consent of the enrolled patients or their relatives was obtained after hospitalization. The personalized identification information in medical records was removed, and their clinical information was uploaded to clinical record information mining database. This study was conducted using the principles of the Declaration of Helsinki and the subsequent amendments. The present study was approved by the Ethics Committee of Affiliated Dongyang Hospital of Wenzhou Medical University (Approval Number: 2022-YX-301).

The study duration was from January 2013 and August 2022. The patients with sepsis were collected from the database based on the diagnosed diseases when they were discharged. In detail, the diagnosis of severe sepsis was based on the sepsis-2 in 2013, which was recategorized as sepsis in sepsis-3 updated in 2016 by the Society of Critical Care Medicine and the European Society of Intensive Care Medicine.^3,16 In addition, the enrolled patients were excluded with these criteria: less than 18 years; patients with advanced tumors, decompensated cirrhosis, leukemia, lymphoma, and advanced AIDS in diagnosed diseases (patients with advanced tumors often gave up treatment; patients with decompensated cirrhosis, leukemia, lymphoma and other patients indicated abnormal blood indicators; advanced AIDS patients had an abnormal immune function with multiple microbial infections). The terminal event for patients with sepsis was obtained from their clinical records at discharge: the patients were defined as alive if they recovered, or as dead if they could not survive during hospitalization.

Variables Extraction from Database

The following variables were retrospectively extracted from database based on the clinical records of hospitalized patients, including age, gender, and the levels of the following indicators collected from the first examination within 24 hours after admission: C-reactive protein, creatinine, lactic acid, prothrombin time, serum cholinesterase, procalcitonin, white blood cells, red blood cells, hemoglobin, hematocrit, platelet, pro-brain natriuretic peptide (pro-BNP), total bilirubin, albumin, alanine aminotransferase, aspartate aminotransferase, systolic blood pressure, diastolic blood pressure, mean arterial pressure, respiration rate and heart rate. Respiratory failure was defined based on partial oxygen pressure/oxygen concentration ≤300 mmHg, which was tested after admission.

Evaluating the Variables Included for Prediction Model

The R software (version 4.2.2) was used to process the data of the patients. The continuous variables were represented as the median (quartile), and the categorical variables were described by numbers (percentage). The variables with missing data more than 15% were excluded. The remaining variables with missing values ≤15% (Figure S1) were amended by multiple imputations in the “mice” function.¹⁷ The involved patients were divided into the modeling group and the verification group using the “createDataPartition” function in the cart package based on the 7:3 split ratio. The univariate analysis was conducted by the ‘twogrps’ function in the “CBCgrps” package, to evaluate the difference of baseline characteristics between the modeling and the validation groups. In the modeling group, the univariate analysis was conducted to screen the risk factors associated with death. Then, whether the death related continuous variables were linear to logitp was analyzed using the “boxTidwell” function in the “car” package. When continuous variables were not linear to logitp (P < 0.05), they were converted into categorical variables. The “VIF” function was conducted to test whether there were multiple collinearities among the included variables (VIF >5).¹⁸ Finally, a multivariate logistic regression and stepwise regression analyses (both directions) were performed to determine the independent risk factors for the final model (P < 0.05). The impact of individual variables on the death risk in the established model was presented as a nomogram by the “regplot” package.

Evaluation of the Established Prediction Model

The discrimination power of the prediction model was evaluated by the area under the curve (AUC) of the receiver-operating characteristic (ROC) curve. AUC >0.8 indicates the model with good discrimination ability.¹⁹ The calibration of the prediction model was assessed by establishing a calibration curve, and a P-value greater than 0.05 indicates excellent goodness of fit.²⁰ DCA curve was used to evaluate the clinical effect of the established model.²¹ Finally, the effectiveness of the established model was compared with the SOFA scoring system and the machine learning models established by the random forest method²² and stacking method.^23,24 The AUCs between the logistic regression model and other models were compared by “roc.test” function.

Results

Basic Data and Variable Screening of the Included Patients

A total of 1740 patients were included in this study, with 1015 females and 725 males. Among them, 421 died, with a mortality rate of 24.2% (421/1740). The population was divided into the modeling group (1218 patients with 294 deaths) and the validation group (522 patients with 127 deaths). No significant difference was observed in the baseline characteristics between the two groups (Table 1). In the modeling population, the univariate analysis demonstrated that gender, creatinine, lactic acid, serum cholinesterase, respiratory failure, total bilirubin, prothrombin time, pro-BNP, procalcitonin, and mean arterial pressure were associated with death (Table 2). The significant continuous variables in univariate analysis were linear to logitp (Table S1). Multiple collinearity tests showed that the VIF value of each variable was less than 5, so a further analysis can proceed (Table S2).

Table 1 Baseline Characteristics of the Development Group and Validation Groupa

Table 2 Univariate Analysis Between Survivors and No Survivors in Modeling Groupa

Variable Screening and Establishment of the Logistic Regression Model

The logistic regression results indicated that serum cholinesterase, total bilirubin, creatinine, lactic acid, pro-BNP and respiratory failure were independent risk factors for the death of patients with sepsis, and finally included in the modeling (Table 3, P < 0.05).

Table 3 Stepwise Regression Analysis of Involved Variables

Using the nomograph (Figure 1), a vertical line was drawn upward from each variable to the top scoring line, and the corresponding points were obtained. Then, the scores corresponding to each variable were summed up to determine the total score. Finally, the death prediction probability corresponding to the bottom of the nomograph was obtained based on the total score. For example, a patient with sepsis was admitted, and variables were collected within 24h as followed: creatinine (112 μmol/L), lactic acid (6.9 mmol/L), cholinesterase (2914 U/L), pro-BNP (17,960 pg/mL), total bilirubin (129.5 μmol/L) and with respiratory failure. The total score of this patient was 2.25, and the corresponding predicted probability of death was 0.67.

Figure 1 The nomogram of established model for predicting mortality risk in sepsis patients after hospitalization. The enrolled variables were collected within 24h after admission. A patient was displayed as an example, with detailed enrolled variables labelled by red dots. The variables labelled of asterisk indicated significance in the model, *, P<0.05, **, P<0.01, ***, P<0.001.

Discrimination Power, Fit Goodness, and Clinical Effectiveness of Prediction Model

The logistic regression based on the independent risk factors in the modeling population owned an AUC of 0.847 (95% CI: 0.821–0.872) (Figure 2A). At the threshold of 0.192, the model had a specificity of 72.8% and sensitivity of 81.6%, respectively. In the calibration chart (Figure 2B), the P-value was 0.838, with a Brier scale of 0.125, a calibration slope of 1.000, and R2 of 0.409, indicating a good fit. The DCA curve (Figure 2C) showed that the modeling curve (mod) was above the two extreme curves, depicting good clinical significance.

Figure 2 Evaluation of prediction model in modeling group. (A) the discrimination power of established model by ROC curves; (B) the agreement between predictions and observations of the predicted values by calibration chart; (C) the clinical net benefit of established model by DCA curves, red bold curve means the mean value, red curves mean 95% confidence interval.

The AUC of logistic regression model in the validation population was 0.826 (95CI: 0.786–0.866), with a specificity of 74.7% and a sensitivity of 79.5% (at the threshold of 0.220, Figure 3A). The P-value of the calibration plot was 0.771, with a Brier scale of 0.136, calibration slope of 1.000, and R2 of 0.347 (Figure 3B). The DCA curve owned good clinical significance (Figure 3C).

Figure 3 Validation of prediction model in validation group. (A) the discrimination power of established model by ROC curves; (B) the agreement between predictions and observations of the predicted values by calibration chart; (C) the clinical net benefit of established model by DCA curves, red bold curve means the mean value, red curves mean 95% confidence interval.

Modeling with SOFA Scoring System, Random Forest Method, and Stacking Method in the Validation Population

The AUC of the model established by the SOFA scoring system in the validation group was 0.777 (95% CI: 0.729–0.825, Figure 4A), significantly lower than that of the logistic prediction model (P < 0.002, Table S3). For the model established using the random forest method in the validation group, its AUC was 0.827 (95% CI: 0.788–0.866, Figure 4A), similar to the logistic prediction model (P = 0.981, Table S3). Regarding the stacking method, SVM, C5.0, and XGboost were taken for ensemble modeling, and the AUC value of the final ensemble model within the validation group was 0.832 (95% CI: 0.793–0.87, Figure 4B).

Figure 4 The discrimination powers of other models. (A) ROC curves of the models based on logistic regression, SOFA and random forest method (B) ROC curves of the models established by stacking method.

Discussion

This study showed that reduced serum cholinesterase activity, elevated total bilirubin, lactic acid, creatinine, pro-BNP levels, and respiratory failure were the independent risk factors for the death of sepsis patients. The prediction model established in this study indicated good significance in discrimination, calibration, and clinical effectiveness, showing a significantly better prediction performance than the model developed by the SOFA scoring system. The machine learning models, such as random forest modeling and stacking modeling, had no apparent advantages in performance than the model in this study. The prediction model can help clinicians visually evaluate the death risk of patients with sepsis, thereby formulating better interventions to reduce the mortality risk.

The logistic regression model established in this study was helpful in discovering independent risk factors. The continuous decline of serum cholinesterase in liver function indicators and the elevation of total bilirubin level were directly related to the death of sepsis patients. Previous studies have revealed that lower serum cholinesterase is associated with increased mortality in sepsis patients. The possible reason was that serum cholinesterase, an indicator of cholinergic activity, reduced rapidly after inflammation.^25,26 As a key indicator of a dysfunctional liver, hyper-bilirubin can also damage the bactericidal properties of neutrophils by antioxidation and stimulate the production of many inflammatory factors, leading to poor prognosis in sepsis patients.²⁷ This study also discovered that the increase in pro-BNP was closely associated with death risk for patients with sepsis, which could be explained by the close relationship between pro-BNP and heart failure.²⁸ Our study also observed that an increase in creatinine levels in the serum could help to predict the death risk of patients since sepsis causes acute renal injury. Previous studies have indicated that acute renal injury is related to the high mortality rate of sepsis.²⁹ The increase in lactic acid concentration could be a substitute indicator for the severity of hypoperfusion and sepsis.³⁰ Several studies on sepsis and septic shock have revealed that the increase in lactic acid significantly contributes to the mortality of patients,^31,32 as confirmed by our study.

Procalcitonin and C-reactive protein help clinicians to judge the severity of sepsis and guide the course of antibacterial drug usage,^33,34 in which procalcitonin is better than C-reactive protein.³⁴ However, previous studies have indicated that levels of procalcitonin and C-reactive protein cannot predict the prognosis of patients with severe infection at an early stage.³⁵ The ability of procalcitonin to stratify the death risk of sepsis was limited.³⁶ This was consistent with our results that procalcitonin and C-reactive protein were not independent risk factors for the poor prognosis of sepsis. Abnormal coagulation function is also a risk factor for poor prognosis of sepsis.³⁷ However, proper application of heparin sodium anticoagulation therapy can improve organ failure due to extensive micro-thrombosis and increased survival rate.³⁸ This study revealed that the indexes for initial coagulation dysfunction were not an independent risk factor for poor short-term prognosis of sepsis patients. Thus, it may be associated with the early intervening correction of coagulation dysfunction. The continuous decrease in mean arterial pressure and significantly increased heart rate indicated that sepsis could progress to septic shock, increasing the mortality risk. However, such patients are given more active fluid rehydration, anti-infection, and other comprehensive treatments immediately after admission. Therefore, some studies revealthat the initial mean arterial pressure value and the heart rate of newly admitted patients could not predict mortality among sepsis patients,^39,40 consistent with our study.

Over the years, many scoring tools have been used in clinical practice to predict the prognosis of sepsis patients, including the APACHE score, SIRS standard, qSOFA score, SOFA score, etc. However, they have defects in predicting the mortality of sepsis patients since these scoring systems are not specially developed for sepsis. The SOFA scoring system is the most widely used among these scoring systems, owning better prediction ability than SIRS and qSOFA as described previously.^3,11 However, SOFA score showes fluctuating, prediction effectiveness when applied in certain circumstances.^41–43 In this study, the overall accuracy of the SOFA score in predicting short-term mortality was not as good as the logistic regression model regarding the AUC. This phenomenon could be explained by the different variables enrolled in the SOFA score and the logistic regression model established in this study. For example, cholinesterase, pro-BNP and lactic acid were included in the logistic regression model but absent in the SOFA scoring system. Recent studies have found the role of pro-BNP and lactic acid in prognosis of patients with sepsis, supporting the displayed advantages of established models over SOFA score models.^28,32 Three variables (respiratory failure, creatinine and total bilirubin) were commonly included in this established model and SOFA score, indicating their significance in death risk prediction in patients with sepsis.

The model for predicting the mortality of sepsis patients established through machine learning also demonstrated good effectiveness. However, the decision support provided by the machine learning model is usually difficult to explain. The development of a clinical prediction model is needed to conform to clinical practice, which is largely limited by the complex algorithms of machine learning models.⁴⁴ In this study, the machine learning models using random forest method or stacking method had no significant advantages in convenience compared with logistic regression models. The impact of enrolled variables on the risk of death is presented as a nomogram, with advantages in clinical interpretation and application.¹⁴

The disadvantages of this study were : (1) It was a single-center study and there was lacked data for validation from other centers. (2) It was a retrospective study, and selection bias could not be avoided. During data collection of sepsis patients, the variables with missing data more than 15% were excluded, such as thrombocytocrit, prealbumin, serum phosphorus, and serum magnesium; only two statuses (recovered or dead) of patients at discharge were considered to define the terminal events in this study. Information on follow-up was unavailable for those patients recovered at discharge.

Conclusion

A prediction model is developed in this study based on the variables collected within 24h after admission and could assess the mortality risk during hospitalization among sepsis patients, thus assisting clinicians in the management of these patients. Another major advantage of the nomogram is also that it can pinpoint the variables with the highest impact on the outcome, which ultimately enables a more personalized medicine approach to each patient.