Prognostic Accuracy of SOFA, qSOFA, and SIRS for Mortality Among Emergency Department Patients with Infections

Background

Sepsis is a serious clinical condition with high mortality.^1,2 The incidence of sepsis has been reported to 83–731/100,000 person-years at risk depending on the definition and severity of sepsis.^1,3,4

Since 1992, Systematic Inflammatory Response Syndrome (SIRS) (Appendix-1) criteria have been used to early identification of infected patients with sepsis. Due to the low predictive validity of SIRS⁵ and a better understanding of the pathobiological mechanism, sepsis has since 2016 (Sepsis-3) been defined as a life-threatening organ dysfunction caused by a dysregulated host response to infection.⁵ Organ dysfunction is defined as an increase of two points or more from the baseline of the Sequential Organ Failure Assessment (SOFA) score (Appendix 1).⁵ Since SOFA requires laboratory tests with the risk of delaying the identification of patients with sepsis, a simple bedside test, quick-SOFA (qSOFA)⁵ (Appendix 1) was proposed as a clinical tool for early identification of infected patients with increased risk of serious outcomes.

However, data from our institution⁶ and other studies^7,8 have reported that qSOFA’s prognostic accuracy is not substantially better than SIRS, and qSOFA is nowadays considered a prognostic tool rather than a diagnostic tool.^6,9,10 Thus, we still lack a method for early identification of infectious patients with a sepsis-like condition and an increased risk of a poor outcome.

The use of SOFA may be unfeasible. Clinicians often have to await laboratory results (appendix 1), and comorbidities (lung, kidney, liver diseases, and dementia) may influence the SOFA calculations, which may delay the identification of septic patients and treatment with antibiotics. With these considerations, SOFA seems, after all, to have superior prognostic accuracy.^9,11–13 However, prospective evaluations of SOFA’s prognostic performance in patients with infections compared to other tools in the emergency department (ED) settings are scarce.^9,11,14 Therefore, the present study aimed to compare the prognostic accuracy of SOFA compared to qSOFA and the SIRS criteria for predicting 28-day mortality in a study of patients with documented or suspected infection admitted to an ED. A secondary purpose was to estimate the incidence of sepsis based on the new Sepsis-3 criteria.

Method

The present study is a secondary analysis of data from a previously published prospective observational cohort study of the prognostic accuracy of qSOFA among adult (≥ 18 years) ED patients with an infection.⁶ All patients were admitted to the ED at Slagelse Hospital, Denmark, during the period October 1, 2017, to March 31, 2018.⁶ Slagelse Hospital is a tertiary care center that serves the entire population in the catchment area, and the ED has an uptake area of 198,000 adult inhabitants with 26,500 visits annually. The Danish healthcare system offers equal access to all residents. Patients who are admitted acutely are either referred by general practitioners or by ambulance to the hospital. Danish privately funded hospitals have no acute patient intake.¹⁵

Patients with Infections

During the study period, triage records and electronic patient records for all patients (N= 12,092) were every working-day reviewed by the authors (SMOBA, RHS) for inclusion. As previously described,⁶ all patients with either documented or suspected infection were diagnosed by an ED physician in charge. If delivered either intravenous or peroral antibiotics (AB) within 24 hours from arrival (N=3176), the patients were initially registered as patients with infectious diseases.

Patients who had a qSOFA score of at least two or fulfilled the SIRS criteria for sepsis, or were critically ill, were given priority for prompt medical examination by an ED physician. Septic patients were treated according to a protocol during the study period. If a physician suspected sepsis, treatment with oxygen, lactate measurement, blood tests, an electrocardiogram (ECG), treatment with intravenous fluids, and intravenous broad-spectrum AB after blood cultures were drawn was recommended initiated within one hour from recognition of sepsis irrespective of disease severity.⁶

Inclusion and Exclusion

Patients treated with antibiotics within 24 hours after ED admission and continued for at least 48 hours were considered to have an infection and were included in the present study. Among the 3176 patients initially registered as having an infection on admission patients with foreign nationality without a Danish Civil Registration number (n=6), if AB treatment was discontinued within 48 hours (n=174), prophylactic AB treatment with surgery (n=240), transfer to or from other hospitals (n=224), if previously included during the study period (n=354), missing data (n=123) and registration errors (n=10) were excluded.

Definitions

We used the Charlson Comorbidity Index (CCI) to classify the burden of chronic disorders.¹⁶

Patients who fulfilled a SOFA score of at least two on admission were considered to have sepsis.⁵ A baseline SOFA score of zero was registered for all patients without chronic disorders. An adjusted baseline score of SOFA was calculated for patients with known chronic diseases (kidney, liver, respiratory, or dementia). SOFA scoring was not a routine in our ED during the study period. The SOFA calculation method has been described in detail in Appendix 2. The definitions of qSOFA and SIRS followed the guidelines described in Appendix 1.

The altered mental state was defined as either a Glasgow Coma Scale (GCS) value less than 15 or AVPU (Alert, Verbal, Pain, Unresponsive) other than A on admission to the ED. Positive blood culture was defined as at least one positive blood culture during the stay in the ED (coagulase-negative staphylococci excluded).

Data Collection

All patients admitted to our ED were triaged immediately after the arrival. We obtained the following triage variable, laboratory test and medical history from the triage forms and the electronic medical record daily by the authors (SMOBA, RHS): Glasgow Coma Scale (GCS) or the Alert-Voice-Pain-Unresponsive (AVPU) scale, systolic blood pressure (SBP)(mm Hg), respiratory rate (RR)(breaths/min), heart rate (HR)(beats/min), body temperature (Tp)(°C), haemoglobin (mmol/L), C-reactive protein (CRP), leucocyte count (×10⁹/L), paO_2´(kPa), peripheral oxygen saturation (%), creatinine (μmol/L), bilirubin (μmol/L), platelets count (×10⁹/L) and information regarding comorbidities (CCI).

Information on site of infection was based on a review of all electronic records at discharge with specific details on infectious sources diagnosed and documented in the records by the physicians during hospital stay. Foci of infection were specified by bacterial culturing of suspected infected tissues and body fluids. Other examinations (eg x-ray, ultrasound, computed tomography, gynecological examinations) were performed if relevant.

qSOFA and SIRS scores were calculated (appendix 1) based on the arrival variables. Survival status was obtained from the Danish Civil Registration System,¹⁷ a national registry with information on vital status among all Danish citizens. Each person has a unique personal registration number (CPR) in Denmark, making the linkage between health registries possible.¹⁷

Statistical Analysis

The primary outcome was 28-day mortality. In-hospital mortality and transfer to the ICU were secondary outcomes. Continuous data are reported as medians with interquartile ranges (IQR). Categorical variables are reported as counts and percentages with 95% confidence intervals (CI). Differences within medians with 95% CI and exact differences of proportions with 95% CI were used to compare the groups. Differences were assumed significant if the 95% CI for the median difference or the 95% CI for the difference of proportions did not include zero. We have calculated the incidence as (number of patients with sepsis defined by either SOFA, qSOFA, or SIRS/adult population in the area x 0.5) x 100.000 person-years at risk (95% CI). Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive- and negative likelihood ratio (LR+/LR-) with 95% CI and area under the receiver operating characteristic curve (AUROC) was used to assess the prognostic accuracy of the scores. The equality of the AUROC was reported by p-value. AUROC values at 0.5 suggest no ability to discriminate between patients with or without the outcome of interest, 0.6 to 0.7 was considered poor, adequate at 0.7 to 0.8, good at 0.8–0.9 and excellent at 0.9 or higher.^12,18

Mortality rates among patients fulfilling the qSOFA score and SIRS criteria, and the prognostic accuracy of the scores, have previously been published⁶ but are included in the present analyses for comparison purposes.

All statistical analyses were performed using STATA 15.1.

Results

Study Population

A total of 2045 patients (48.6% male) with a median age of 73.2 years (IQR 60.9–82.1) were included (Figure 1).

Figure 1 Study flow chart.

Baseline Characteristics

The baseline characteristics, according to the sepsis criteria (SOFA ≥2), are shown in Table 1. A total of 61 (3.0%) of the patients had systolic blood pressure ≤ 90 mmHg on admission, and 20 (1%) were treated with vasopressors (Table 1). The proportion of hypotensive (6.9% vs 0.9%) and vasopressor treated patients (2.2% vs 0.3%) was highest among sepsis patients (Table 1)

Table 1 Baseline Characteristics According to Sepsis Classification

Sepsis patients were older, more often male, and had higher comorbidity scores (Table 1). The proportion of patients with congestive heart failure, ischemic heart disease, cerebrovascular disease, diabetes, chronic kidney disease, and hypertension was higher among patients with sepsis (Table 1). C-reactive protein, leucocyte count, creatinine, bilirubin, and lactate values were higher among septic patients while platelets were lower (Table 1). Blood cultures were more often obtained among septic patients, and positive blood cultures were more common in patients with sepsis (Table 1). Pulmonary origin of infection was more frequent in septic patients, while skin and urinary infections were more common in non-sepsis patients (Table 1). A total of 139 (6.8%) had unknown sites of infections (Table 1).

Incidence of Sepsis

A total of 714 (34.9%;32.8–37.0) patients had a SOFA score of ≥2 on admission, 174 (8.5%;7.3–9.8%) had a qSOFA score of ≥2, and 999 (48.9%; 46.6–51.0%) fulfilled at least two SIRS criteria (Table 2). The incidence of sepsis based on SOFA was 721 (663–768) per 100,000 person-years, based on qSOFA 176 (149–202) per 100,000 person-years, and based on SIRS criteria 1009 (938–1062) per 100,000 person-years (Table 2).

Table 2 Incidence, Transfer to ICU and Mortality According to Different Sepsis Criteria

Mortality

A total of 158 (7.7%) patients died within 28 days, and the mortality in patients with a SOFA score ≥2, qSOFA ≥2, and SIRS ≥2 was 13.6% (11.2−16.3), 17.8% (12.4− 24.3), and 8.3% (6.7–10.2), respectively. The 28-day mortality increased with increasing SOFA scores (Table 3) and was significantly higher (13.6% vs 4.5%; 6.3–11.8%) in patients with sepsis (SOFA ≥ 2) compared to patients without sepsis (SOFA ≤ 2) (Table 2).

Table 3 28-Day Mortality According to the SOFA Scores

The 28-day mortality was highest among patients meeting a SOFA score ≥2 compared to SIRS ≥2 (13.6 vs 8.3%; 2.3−8.3%) on admission (Table 2). There was no difference in 28-day mortality between patients with SOFA ≥2 and qSOFA ≥ 2 (13.6 vs 17.8%; −2.0 − 10.4%) (Table 2).

The in-hospital mortality was also higher among patients with SOFA ≥2 than SIRS ≥2, (7.3% vs 4.2%; 0.8−5.4%) (Table 2). However, there was no difference in in-hospital mortality between patients with SOFA ≥2 and qSOFA ≥2 (7.3% vs 9.2%; −6.5 − 2.8) (Table 2).

Transfer to ICU

The proportion of patients transferred to the ICU was higher among patients with a SOFA score ≥2 than patients with SIRS ≥2 (13.5 vs 10.0%; 0.3−6.6%) (Table 2). However, there was no difference between patients with a SOFA score ≥2 and qSOFA ≥2 (13.5 vs 16.1%; -8.6−3.4%) (Table 2).

Prognostic Accuracy for 28-Day Mortality

Patients with SOFA ≥2 had a sensitivity of 61.4% (53.3−69.0) and a specificity of 67.3% (65.1−69.4) (Table 4). Patients meeting qSOFA ≥2 had a sensitivity of 19.6% (13.7−26.7) and a specificity of 92.4% (91.1−93.6) (Table 4). Patients with SIRS ≥2 had a sensitivity of 52.5% (44.4−60.5) and a specificity of 51.5% (49.2−53.7) (Table 4). Overall AUROC for SOFA was significantly higher compared to qSOFA (0.68 vs 0.63; P=0.018) and SIRS (0.68 vs 0.52; p<0.001) (Table 4). Likelihood ratios are close to 1.0 (for both LR+ and LR-) for SOFA and qSOFA (Table 4). PPV and NPV for the scores are shown in Table 4. The AUROC curves and corresponding AUROC values showed that SOFA and qSOFA had poor prognostic ability to predict 28-day mortality (Figure 2). SIRS could not predict mortality (Figure 2).

Table 4 Prognostic Accuracy of SOFA, qSOFA, and SIRS for 28-Day Mortality

Figure 2 The area under the receiver operating characteristics (AUROC) curves. Abbreviations: qSOFA, quick sequential organ failure assessment; ROC, receiver operating curve; SIRS, Systemic inflammatory response syndrome; SOFA, sequential organ failure assessment

Supplementary Analyses

We found no significant changes for AUROC values according to the three scores after excluding patients with systolic blood pressure ≤ 90 mmHg or need of vasopressor treatment (Supplement Table 1).

Prognostic Accuracy for in-Hospital Mortality and Transfer to ICU

The sensitivity for SOFA ≥2, qSOFA ≥2, SIRS ≥2 in predicting in-hospital mortality was 67.5% (55.9−77.8), 20.8% (12.4−31.5), 54.5% (42.8−65.9), respectively. The specificity was 67.5 (55.9−77.8), 92.0% (90.7–93.1), 51.4% (49.1−53.6), respectively. Analyses applying ICU-admission as endpoint did not show any significant differences across the scores (data not shown).

Patients without Chronic Disorders

In a sensitivity analysis excluding patients with chronic disorders and leaving patients with a baseline SOFA score of zero, we found a sensitivity of 64.5% (51.3−76.3), a specificity of 70.7% (67.9−73.4), and AUROC of 0.70 (0.63–0.77) for SOFA, which is not notable deviating values compared to the total group of patients (Table 4). The sensitivity analyses were repeated for qSOFA and SIRS without significant new findings (data not shown).

Discussion

In this study, we found that a SOFA score of at least two in patients with infectious diseases on admission to an ED had better prognostic accuracy than SIRS and qSOFA to predict 28-day mortality. However, the overall accuracy of SOFA was poor for the prediction of 28-day mortality. Our study also revealed that sepsis incidence based on the SOFA criteria was 721/100,000 person-years, which was considerably higher compared to estimates based on qSOFA, and lower compared to the SIRS criteria.

The Sepsis-3 guidelines were based on a retrospective study of both ICU and non-ICU patients.¹² Since the introduction of SOFA in 2016, only one prospective study of SOFA’s prognostic accuracy in ED patients has been published.¹¹ Freund et al¹¹ found in a multicentre study of 879 ED patients with infectious diseases that SOFA outperformed SIRS; however, SOFA was not better than qSOFA. The sensitivity and specificity of SOFA for predicting in-hospital mortality were 73% and 70%, respectively.¹¹ The AUROC values for SOFA, qSOFA, and SIRS for in-hospital mortality were 0.77, 0.80, and 0.65, respectively.¹¹ The higher values of sensitivity, specificity, and AUROC compared to our study can be explained by differences in the SOFA measurement. We have used clinical and laboratory values on arrival to calculate the SOFA score. In contrast, Freund et al used the worst values during the ED stay, explaining the different performances of the scores. However, in our study, we have taken chronic disorders into account and examined the validity of the SOFA scores as a secondary data analysis of prospectively collected data. Freund et al did not describe the method behind the SOFA calculations.

A retrospective study by Boillat-Blanco et al of 519 infected patients admitted to an ED in Dar es Salaam concluded that SOFA had a prognostic accuracy for 28-day mortality similar to qSOFA but superior to SIRS.¹⁹ The study reported AUROC values for 28-day mortality for SOFA, qSOFA, and SIRS of 0.79, 0.80, and 0.61, respectively, which were also higher than our study. The higher discriminatory ability of SOFA and qSOFA was probably due to the high prevalence of various infectious diseases (Malaria, Dengue, HIV, etc.) among febrile adult patients.¹⁹

A screening tool for sepsis requires adequate sensitivity and specificity to identify sepsis patients. However, the study by Seymour et al¹², which was the basis for the Sepsis-3 definition of sepsis, did not report sensitivity or specificity for SOFA. The sensitivity and specificity of SOFA score were higher in our study in comparison to SIRS. However, it was slightly lower than prior studies conducted among infected patients in the ED.^11,19 Furthermore, we found that the sensitivity of the SOFA was substantially higher than qSOFA.

Few studies found that the prognostic accuracy (measured by the AUROC) of qSOFA was better than SOFA for non-ICU patients.^11,19 However, recent prospective studies^6–8 and a meta-analysis²⁰ of qSOFA concluded that qSOFA cannot stand alone as a tool to identify infectious patients with a high risk of serious outcomes due to its low sensitivity.

AUROC for the SOFA was in our study lower than in prior studies (Table 4).^{9,11–14,19,21–25} The different findings can be explained by differences in methodology. Several studies^{9,11,12,22,23} have used the worst values of variables from the ED stay to calculate SOFA. Many studies were undertaken among patients with critical clinical conditions and admitted to the ICU^21,22 or among mixed cohorts of ICU and non-ICU.^13,14 Further on, the definition of infection varied across the studies.^{9,11–14,19,21–24} All these studies reported better prognostic accuracy of SOFA than SIRS, and seven out of ten studies reported slightly better prognostic performance of SOFA than qSOFA.

The apparent poor performance of the tools to identify patients with sepsis and the use of prognosticators rather than specific diagnostic tools calls for the development of other and better methods for early identification of sepsis patients. Two meta-analyses examining the role of more than 200 hundreds biomarkers have shown that the precise role of biomarkers in the management of sepsis patients has not been well-defined.^26,27 The methodology used in the studies are not adequate, and most of the studies are based on old sepsis definition and are performed in different settings.^26,27 The biomarkers with the highest AUROC’s were described in studies with limited sample sizes.²⁷ Both meta-analyses call for new studies with a standardized methodology to evaluate the biomarker’s ability to identify septic patients.^26,27 With rapid technological advancement within the point-of-care diagnostic tests of different biomarkers, it is desirable to look into their usefulness together with different scores of sepsis and, for example, combining biomarkers as Interleukin-6 (IL-6) and procalcitonin (PCT) with SOFA or qSOFA. IL-6 nor PCT cannot identify patients with sepsis alone; however, the combination of these with clinical criteria may improve their prognostic accuracy.^26,28 There are also ongoing promising research initiatives in developing host-based molecular assays to detect sepsis at an early stage of the disease.²⁹

The incidence of sepsis in our study was lower when the new sepsis-3 (SOFA) criteria were used compared to the previously used SIRS criteria, which is consistent with another large retrospective study conducted by Donnelly et al among non-ICU patients.²³ Donnelly et al reported the sepsis incidence according to SOFA and SIRS criteria to 580- and 820 pr. 100.000 person-years at risk, respectively.²³

Strength and Limitation

The main strength of this study was its prospective design of data collection. Although the SOFA calculations were performed as secondary analyses, the analyses were based on prospective data collection. The study was designed to identify all infected patients based on well-defined infection criteria and consecutively admitted to our ED. A complete follow-up was ensured by the CPR number of all the included patients, which reduced the risk of selection bias. Furthermore, we assume no referral bias as all acute patients from the uptake area were referred to our ED.

The study has potential limitations. First, the median age in our cohort seems higher compared to other studies (Table 5), which should be taken into account when comparing mortality rates.

Table 5 Prognostic Performance of SOFA, qSOFA and SIRS Criteria

Second, different definitions of infection should be considered when comparing prognostic performances of the scores across the studies. Other definitions used are suspicion of infection by the on-duty physician,^11,14,24 infection criteria derived from national Infectious Diseases Societies^19,21 and a combination of AB and culture obtained^12,22,30 (Table 5). Third, the method used to adjust the SOFA score for chronic diseases with potential impact on the baseline SOFA value was not protocolized before the study start. The SOFA calculations were performed as secondary analyses, and the information on chronic diseases was based on diseases that were included in the CCI classification. We did not have precise information on the severity of chronic diseases before admission to ED, and our method used to adjust the SOFA values has not been validated. Therefore, we cannot exclude the risk of misclassification of septic patients based on the SOFA score. However, we have reanalyzed the prognostic accuracy of SOFA after excluding patients with chronic diseases with a potential impact on the baseline SOFA value and did not find any important differences. Furthermore, we only used admission variables to calculate the SOFA score. Serial SOFA measurements during the first hours of admission may have detected some patients with clinical deterioration and fulfilling the sepsis criteria during the initial ED stay. Fourth, sepsis treatment was primarily guided by a sepsis treatment protocol for patients fulfilling the qSOFA≥2 criteria, SIRS≥2 criteria, or if the patients were critically ill. According to the protocol, some of the septic patients identified by the SOFA criteria in our study did not fulfill the qSOFA and SIRS criteria, and sepsis treatment according to the protocol may therefore have been opted-out. The extent to which this may have affected the outcomes in patients only fulfilling the SOFA criteria is difficult to explore due to the observational study design and risk of bias. Fifth, the accuracy of our estimates could have been increased with larger sample size. Finally, the study was undertaken at a single Danish center and may not be generalizable to other centers in different countries.

Conclusion

A SOFA score of at least two among infectious patients admitted to an ED had better prognostic accuracy than SIRS and qSOFA in predicting 28-day mortality. However, the overall results of the scores indicate that SOFA is not an ideal sepsis screening tool and cannot stand alone in the identification of infected patients with the risk of serious outcomes. The incidence of sepsis depends on the definition used, and the findings from our and other studies indicate a need to rewrite parts of the epidemiology of sepsis.