Establishment and Effectiveness Evaluation of a Scoring System-RAAS (RDW, AGE, APACHE II, SOFA) for Sepsis by a Retrospective Analysis

Background

Sepsis is a serious, life-threatening syndrome that is one of the leading causes of death in hospitals.^1,2 Sepsis was also among the leading causes of the death associated with COVID-19.³ Despite significant advances in critical care, sepsis still has a relatively high short-term mortality rate due to organ failure due to a dysregulated response to infection.⁴ Often as the first place where sepsis patients are initially treated, it is significant for the emergency department (ED) to identify patients with a high mortality rate among sepsis patients in the early stages of sepsis, in order to provide more timely and adequate interventions for these patients and to help patients be properly classified.⁵ Up to now, there is still no effective treatment and means for sepsis.⁶

Early prediction of progressive septic shock plays an important role in its clinical course and prognosis.⁷ Although several biomarkers have been studied to predict mortality in patients with sepsis, none of these biomarkers have been shown to reliably identify patients at high risk for sepsis, and tests for these biomarkers are often not widely used in clinical practice.⁸ Moreover, due to the complexity of sepsis, the diagnostic accuracy of each parameter is limited in clinical practice. As a standard to evaluate the severity of disease in intensive care unit (ICU), scoring system has become an important tool to help clinicians make decisions.⁹

A scoring system with higher prognostic accuracy has been developed that combines four clinical indicators to predict the mortality of patients with sepsis using a score of 0 to 6, with one point awarded for each value that meets the scoring requirement (Supplementary Table 1). The main objective of this study was to evaluate the effect of RAAS on short-term (30 days), medium-term (60 days, 90 days) and long-term (180 days, 365 days, 730 days) mortality in patients with sepsis using a large clinical database, and then to establish nomograms of the effective prediction period to visually predict the 30-day,60-day and 90-day mortality probability in patients with sepsis. The RAAS score has a great predictive function for the evaluation of the prognosis of death in sepsis.

Methods

Patients and Study Design

We conducted a retrospective observational study on 1066 patients with sepsis between March 2013 and April 2021. According to the time point, 529 patients were selected as the primary cohort. The remaining 537 patients were selected as validation cohort study subjects to verify the effectiveness of RAAS. As can be seen from Supplementary Table 2, this database has a high degree of fit and is suitable for the verification of the scoring system. The authors used a structured collection format to collect data from electronic medical records. The other author reviewed all the data, and any disagreements were resolved through discussion among the authors.

All hospitalized patients diagnosed with sepsis or septic shock were eligible for inclusion in the study. In order to unify the standards, we reevaluated the situation and information of all patients according to the latest sepsis standards, and eliminated the information of patients that did not meet the latest sepsis standards. Exclusion criteria included: (1) Patients younger than 18 years; (2) Patients with ICU stay less than 24 h; (3) Patients with abnormal values (values 3 times higher than the mean standard deviation (SD)). For patients admitted more than once, only the first hospitalization was included in the analysis. In the case of multiple infections, the main source of infection was dominant (the infection that been spotted first and related organ diseases).

Ethics

The research was carried out in accordance with the ethical standards of the Ethics Committee of Xinhua Hospital Affiliated to Shanghai Jiao Tong University School of Medicine (No. XHEC-D-2021-131) and the National Research Council and in accordance with the Declaration of Helsinki and its subsequent amendments or similar ethical standards. We confirm the patients were informed about the purpose of the study and we read informed consents from the participants to participate the study and to report or publish the retrospective study data.

Establishment of RAAS Scoring System for Sepsis and Investigate the Correlation Between Different RAAS Scores and Prognosis of Patients with Sepsis

Assign values for parameters according to different variable ranges and CUTOFF values of different parameters. The parameter assignment is based on the assignment method of the APACHE II scoring system and the MODS scoring system.¹⁰ To verify the prognosis evaluation effect of RAAS in patients with sepsis, The APACHE II score, SOFA score and corresponding RAAS score within 24h after admission were calculated for patients in the primary cohort and validation cohort. The AUC area of the three scoring systems and the AUC area of the single RDW were calculated, and the predictive effect of RAAS on sepsis patients was judged by comparing the AUC area of the three scoring systems.

The scores were divided into four intervals (GROUP-1:0–1, GROUP-2:2–3, GROUP-3:4–5, GROUP-4:6), and the total mortality of sepsis patients corresponding to four different RAAS scoring intervals and the mortality of patients with different RAAS scores were calculated. The survival curves at 30, 60, 90, 180, 365 and 730 days in the primary cohort and the validation cohort were used to demonstrate and verify their prognostic value.

Statistical Analysis

The normal-distributed measurement data set is represented by mean plus or minus standard deviation (mean±SD or x±S), while the mis-distributed measurement data set is represented by median (quaternary range). Counting data is expressed as a percentage. For the comparison of data between two groups, univariate statistical method is used to measure the data, two-independent-samples t-test is used for the normal distribution of the two groups of data, and the rank sum test of two independent samples is used for the normal distribution of the data.

For parameters with statistically significant differences in univariate analysis, receiver operating characteristic (ROC) curve analysis was used to determine the optimal truncation value of each independent parameter. The corresponding value of the highest Youden index calculated by specificity + sensitivity −1 was used to calculate the AUC screening RAAS parameters, and the 95% confidence interval was calculated. Log rank test was used to compare the survival curves in different intervals of RDW and the survival curves in different time periods of each rating group of RAAS. The significant 30-day, 60-day and 90-day RAAS calibration curves were plotted respectively to verify the performance of the RAAS scoring system model trained by the primary cohort in the validation cohort. Meanwhile, the overall and 30-day, 60-day and 90-day nomogram were plotted. The results of logistic regression analysis were used to construct the nomogram. The performance of the nomogram was evaluated for calibration, identification and clinical applications.

The statistical analyses were performed using SPSS software version 26.0 (SPSS Inc, Chicago, IL) and the R statistical software, version 3.2.4 (The R Foundation for Statistical Computing, Vienna, Austria).

Results

Baseline Clinical Data and Demographic Characteristics and Results of Screening the Scoring System Parameters for RAAS

There were no significant differences between the alive and dead groups for cardiovascular, kidney disease, or other comorbidities. In terms of the sources of sepsis infection, the distribution of each source of infection is roughly the same. Since it is not the key research parameter in this study, no more details will be given here (Table 1). The determined single-factor statistical analysis of RAAS parameters actually included in the statistical analysis is shown in Table 2. A total of four significantly different parameters were included in RAAS between the two groups. Through the above series of screening, this study finally determined that RDW, AGE, SOFA and APACHE II can be used as the parameters of RAAS.

Table 1 The Demographics and Baseline Clinical Data of the Sepsis Patients for RAAS Establishment

Table 2 The AUC of Each Parameter for RAAS in Different Survival Length at Two Cohorts

Establishment of the Scoring System for Sepsis and Evaluation Effectiveness of RAAS on the Prognosis of Sepsis Patients

The RAAS scoring system is a better predictor of mortality than the old scoring system. Had the highest value (Table 2), indicating that the improved scoring system was the best predictor of the three scoring methods (Figure 1). AUC of all parameters of RAAS were calculated using the database to evaluate the effectiveness of RAAS (Table 2). The prognosis of survival was different between the primary cohort and the validation cohort with different lengths of the RAAS scoring system, but was greater than 0.7 overall. In the primary cohort, AUC-30 days were 0.816, AUC-60 days were 0.815, AUC-90 days were 0.820, AUC-180 days were 0.805, AUC-365 days were 0.800, and AUC-730 days were 0.789. In the validation cohort, the values of AUC-30 days, AUC-60 days, AUC-90 days, AUC-180 days, AUC-365 days and AUC-730 days were 0.759, 0.746, 0.748, 0.746, 0.737 and 0.732. When the RAAS score was no less than 6, sepsis patients have a mortality rate of up to 80%.

Figure 1 Primary cohort ROC and validation cohort ROC. Figure 1-1, The AUC of each parameter for RAAS was calculated using the primary cohort. Figure 1-2, The ROC of each parameter for RAAS was calculated using the validation cohort. A, B and C refer to the 30 days, 60 days and 90 days, respectively. D, E and F refer to the 180 days, 365 days and 730 days, respectively. ROC (receiver operator characteristic curve). RAAS(RDW, AGE, APACHE II, SOFA).

In the Kaplan–Meier analysis, higher RAAS scores were associated with higher mortality on days 30, 60, and 90 (Figure 2). As can be seen from the statistical results (Figure 3), correction curves were drawn in the primary cohort and the validation cohort in this study on days 30, 60 and 90, and the data of the two groups of sepsis patients were highly fitted. With the improvement of RAAS score, sepsis’s mortality rate also increased and the number of patients also increased (Supplementary Figure 1, Supplementary Table 3). However, there was no significant difference between the death rates at 180 days, 365 days, and 730 days. When the RAAS≤1, the mortality rate of sepsis patients was less than 10%; when the RAAS=6, the mortality rate of sepsis patients was more than 80% (Supplementary Table 3). Line plots were made based on the overall prognosis and 30-day, 60-day, 90-day data. Four elements of the multivariate analysis were used to construct the nomogram (Figure 4).

Figure 2 Kaplan–Meier curve of 30, 60, 90, 180, 36, 730 days survival according to the four groups (group-1:0–1, group-2:2–3, group-3:4–5, group-4:6) using the RAAS scoring system. (A–F) for primary cohort, and (G–L) for validation cohort, *p<0.05. There are significant differences between the two randomly selected from 4 group.

Figure 3 Calibration plots of the primary model and the verification model. Ideally, all groups of predicted probabilities fit close to the dashed diagonal line (perfect calibration). Vertical lines in each group represent 95% confidence intervals of estimated probability. A, B and C refer to the 30 days, 60 days and 90 days calibration plot in primary cohort; D, E and F refer to the 30 days, 60 days and 90 days calibration plot in validation cohort.

Figure 4 Validated nomogram for predicting 30-day, 60-day and 90-day mortality of sepsis. When using it, drawing a vertical line from each variable upward to the points and then recording the corresponding points (ie, RDW=17=50 points). The point of each variable was then summed up to obtain a total score that corresponds to a predicted probability of 30-day, 60-day and 90-day death at the bottom of the nomogram. (A) the nomogram for the risk of 30 days mortality. (B) the nomogram for the risk of 60 days mortality. (C) the nomogram for the risk of 90 days mortality.

Discussion

Critical patient scoring system is an important method to quantitatively evaluate the severity of disease.¹¹ APACHE II scoring system is the most widely used and authoritative critical disease evaluation system. The system, designed by Professor Knaus’s team at the University of Washington in 1985, consists of three components: age, acute physiological score (APS) and chronic health score. In general, disease assessment and prognosis prediction are often quite accurate for common critical diseases, especially when the pathophysiological characteristics of the disease are similar to the included indicators in the APACHE II scoring system.^12,13 However, continuous studies have confirmed that for some diseases with strong specialty characteristics or some special populations, diseases with special organ damage or abnormal physiological indicators, the above scoring system has certain defects.¹⁴ There is study found that the APACHE II score tended to overestimate mortality in this population.¹⁵

Previous study established and evaluated of a scoring system for exertional heat stroke by retrospective analysis. As EHSS score increases, the mortality rate of EHS patients gradually increases.¹⁶ Our study refers to this study, and adopts a similar method to screen out four parameters that constitute RAAS. Through statistical analysis, four parameters were included in RAAS, which included SOFA and APACHE II, simplified the lengthy details in the scoring system, and at the same time covered most of the clinical evaluation indicators. A review of the development of previous critical illness scoring systems (APACHE II, SAPS II, and MODS) shows that they were established on a large sample size.^17,18 However, for the diseases with strong characteristics of specialization, the sample size of the scoring system based on the larger sample size cannot be obtained due to the limitation of the characteristics of primary disease and the source of cases. For example, the Ranson Scale.^19,20

The number of indicators and scoring systems that enhance sepsis prognosis has been increasing researched in previous studies, sepsis (severe infection) on hematopoietic system has a great influence, one of the directions is the diagnostic and prognostic value of red blood cell distribution width in sepsis.^21,22 In previous study, our group found the RDW fluctuation can help predict sepsis-related DIC morbidity and prognosis in patients with sepsis.^23,24 More importantly, is it the RDW or the cause of increased RDW that is associated with increase mortality, in particular, malignancy, active chemoRT, vitamin B12 deficiency and chronic liver disease.^25–28 The new research also found new biomarkers such as monocyte distribution width improve early detection of ED sepsis.²⁹ Age is also a default prognostic factor for critical illness, with sepsis severely affecting the elderly.³⁰

To our knowledge, this is the first study to assess independent risk factors associated with RAAS (RDW, AGE, APACHE II and SOFA) and establish a prediction graph for 30-day, 60-day, 90-day, 180-day, 365-day, and 730-day mortality. The overall study covered from short, to medium to long term outcomes (Figure 2). In addition, with the increase of RAAS score, the mortality rate of sepsis patients gradually increased from 20% to 80%, which has a great predictive function for the evaluation of the prognosis of death (Supplementary Table 3). The RAAS scoring system does not only apply the initial assessment of sepsis patients in ED, but also can intensively evaluate sepsis patients in the ICU, and also can do second evaluate to the latest condition of septic patients.

The difference between our RAAS score and SOFA score and APACHE II score is that, first, we obtained the score from a large number of patients with heterogeneous suspected sepsis, and we selected patients with confirmed sepsis who were admitted to the ICU of the emergency department for treatment. Second, we used only four independent predictors. In addition to pragmatic considerations, these rules may also suffer from overfitting,³¹ the performance of classifier is usually evaluated by increasing the amount of data and testing the sample set, we also calculate variance inflation factor (VIF) (variance inflation factor value) (Supplementary Table 4). The larger the VIF is, the smaller the tolerance of the independent variable is, and the more collinearity there is. This can result in poor subsequent performance of the validation queue. However, the performance of the fitting degree in the correction curve is still relatively fit. The RAAS score is simple and practical, but it also has a good sensitivity and negative predictive value, allowing us to exclude patients with an elevated risk of death.

In the medical literature, a nomogram is a graphical tool commonly used as a statistical prognostic model to assign a relative risk score for each risk factor based on its contribution to prognosis. The nomogram is primarily used for cancer prognosis and is primarily used to estimate the likelihood of an event.³² In this study, we applied the nomograms of RAAS to assess sepsis patients and explored the accuracy of its prediction in order to identify sepsis patients at high risk of death in the emergency department early in order to make the prediction model visualized and easy to use. Combined with the four RAAS parameters, a nomogram was made to predict the overall and 30-day, 60-day, and 90-day mortality of sepsis patients (Figure 4). It is essential that clinicians comprehensively assess the true risk of death and objectively assess the risks/benefits of medical interventions for patients with sepsis, enabling clinicians, patients and their families to carefully evaluate the impact of potential treatment options. Help them make medical decisions together and prevent medical disputes.

Reinforcement learning has been successfully applied to medical problems in the past, such as diabetes and mechanical ventilation in the ICU.^33,34 As the number of cases participating in the RAAS scoring system increases, the weights of the four parameters constituting RAAS can be modified or adjusted continuously through machine reinforcement learning to make the scoring system more perfect and practical. We take RAAS as the base to jointly look forward to the emergence of a more accurate prognosis system for sepsis.

Our study had several limitations. First, as retrospective observational studies are by nature, information obtained from hospital electronic medical record systems may be biased or inaccurate. Second, as others have discussed,^35,36 the total in-hospital mortality rate was used because the primary outcome of the performance evaluation may not match, because the in-hospital mortality rate may not have been caused by the initial sepsis in the ED. Finally, this was a retrospective study from a single medical center with a limited sample size. Therefore, Follow-up prospective studies are necessary to validate our findings and further confirmation of RAAS in a large sample is needed. In the following study, we plan to validate the RAAS scoring system in the MMIC III database.

Conclusions

RAAS (RDW, AGE, APACHE II, SOFA) is a scoring system for the prognosis of patients with sepsis. The establishment of this scoring system quantifies the severity of sepsis, which is of great significance to effectively determine the severity of sepsis patients in the future, improve the success rate of treatment, judge the prognosis of patients, and guide the treatment of doctors.