Validation of an ICD-Based Algorithm to Identify Sepsis: A Retrospective Study

Background

As one of the leading causes of death and a major contribution to preventable mortality, sepsis was extrapolated to affect 48.9 million patients and contribute to 11 million deaths each year.¹ Although great efforts had been made to estimate the true burden of sepsis worldwide, most epidemiology studies of sepsis were conducted in high-income countries.^1–4 The scarcity of high-quality data in the rest of the world hampered the accuracy and generalizability of those estimates.⁵ Meanwhile, it was estimated that 85.0% of incidences and 84.8% of case fatalities occurred in low- and middle-income countries (LMICs), which highlighted the priorities of sepsis epidemiology research in LMICs, particularly in China, the most populous country in the world.¹

As per the most recent definition of Sepsis-3, sepsis was defined as acute change in Sequential Organ Failure Assessment (SOFA) score ≥2 points consequent to any infections.⁶ However, the components of SOFA excluded the feasibility of sepsis surveillance, especially in LMICs.⁷ In 2018, the Centers for Disease Control and Prevention (CDC) developed a hospital toolkit, known as eSOFA, using simplified clinical data obtained from the electronic health records (EHRs) for sepsis surveillance in hospital.⁸ However, the limited availability and accessibility of EHRs in LMICs in China remained a challenge.⁹ Thereby, healthcare administrative claims data might be the most feasible way to estimate burden of sepsis in China.

As a common data source, the healthcare administrative claims were widely used for sepsis surveillance.¹⁰ The most cited international classification of diseases (ICD) identification algorithm was the combination of ICD codes for both infection and organ dysfunction.¹¹ In the era of Sepsis-1, mainstream identification strategies including Angus’s criteria and Martin’s criteria focused mostly on narrow diagnoses of infection and organ failure. Validation of these algorithms showed a good specificity, but a suboptimal sensitivity using medical record reviews as a gold standard.^11–13 Similar findings were reported in the validation of the Sepsis-3 based algorithm.⁷ Nevertheless, a retrospective medical record review was limited to several methodological mistakes,¹⁴ inadequate data collection, inter-rater or intra-rater unreliability, and compromised data abstraction.

The Chinese version of the International Classification of Diseases, Tenth Revision (ICD-10) diagnosis codes was expanded to a 10-digit version from the 4-digit WHO version.¹⁵ There were explicit sepsis cases that had an ICD-10 code that explicitly referenced sepsis, which differed from the traditional criteria.¹⁶ For instance, the category code 008 (3-digit) refers to “complications following ectopic and molar pregnancy”, and its secondary category code 008.2 (4-digit) pertains to the specific condition “embolism following ectopic and molar pregnancy”, while its tertiary category code 008.200x006 (10-digit) pertains to the particular condition “embolism following abortion and ectopic and molar pregnancy (septicopyaemic)” and was thus labeled as an explicit sepsis code. However, 008.200x006 was not included in the traditional criteria.

Similar to previous studies (Supplementary Table S1), the aim of this study was, therefore, using a prospective cohort of hospitalized patients with SOFA score confirmed sepsis or non-sepsis, to validate a modified Sepsis-3 based ICD identification algorithm in ICU and non-ICU settings, hopefully to provide a validated ICD-based algorithm for sepsis surveillance in China.

Methods

Study Design and Data Collection

This is a retrospective analysis of a single-center, cohort study (NCT 02930070 registered in clinicaltrials.gov), which was designed to assess the diagnostic value of qSOFA for sepsis in a general ward.¹⁷ During the study period from October 1, 2016, to March 31, 2017, all adult patients admitted to one medical ICU ward and ten non-ICU wards with suspected or confirmed infection were eligible for enrollment. Patients were excluded if their age were less than 18 years or hospital length of stay (LOS) less than 24 hours. All patients were followed up until hospital discharge, death, or end of 28-day period, whichever occurred first.

The retrospective analysis was performed based on two databases. We retrieved the following data from the original trial’s dataset: (1) Demographic data, chronic comorbidities, information of infections; (2) systolic and diastolic blood pressure, heart rate, saturation of pulse oxygen, respiratory rate, body temperature, urinary output; (3) laboratory results required for SOFA scores, microbiology findings, arterial blood gases; and (4) requirement for critical care resources, ie, ICU admission, respiratory support, vasopressors, and renal replacement therapy. The ICD codes were extracted from the discharge abstract dataset containing ICD-10 diagnosis codes and ICD-9 procedure codes. The protocol was approved by the institutional review board of Peking Union Medical College Hospital, and informed consent was waived (ZS-1142).

Definitions and Sepsis Identification Criteria

Sepsis-3 Criteria

The Third International Consensus Definitions for Sepsis (Sepsis-3) were considered as the gold standard and sepsis was defined as a change in SOFA score ≥2 points consequent to an infection⁶ (Supplementary Table S2). Infection was defined according to the predefined criteria (Supplementary Table S3). Baseline SOFA was defined as the estimated SOFA score on the hospital admission day. The baseline SOFA score was assumed to be zero if no history of organ dysfunction was present. The variables required for SOFA score were collected daily, including platelet count, serum bilirubin, serum creatinine, Glasgow Coma Score, saturation of pulse oxygen (SpO₂) or partial pressure of arterial oxygen (PaO₂), Fraction of Inspired Oxygen (FIO₂), blood pressure, and vasopressors. We preferentially used the PaO₂ to FIO₂ ratio (PaO₂/FIO₂) when arterial blood gases were obtained. If not available, the PaO₂ was estimated from the SpO₂.¹⁸

Modified ICD-10 Criteria

Similar to previous studies,^11,13 our modified ICD-10 identification algorithm was based on a combination of ICD codes for both acute infection (Supplementary Table S4) and organ dysfunction. Organ dysfunction was defined based on organ dysfunction related ICD-10 diagnosis codes and ICD-9 procedure codes (Supplementary Tables S5 and S6). The identification algorithms of Angus and Martin were converted from ICD-9 to ICD-10 codes (Supplementary Tables S7–S10) according to the tool of Centers for Medicare & Medicaid Services.¹⁹

ESOFA Criteria

Based on the simplified organ dysfunction criteria (eSOFA) proposed by the US Centers for Disease Control and Prevention (CDC), sepsis was identified if there was presumed infection and concurrent eSOFA score ≥1 points.^7,8,20,21 The presumed infection was defined as a medical record of both blood culture and new administration of antibiotics for at least 4 consecutive days. The eSOFA criteria was optimized for electronic health records including: 1) initiation of vasopressors; 2) initiation of mechanical ventilation; 3) doubling of serum creatinine or decrease by 50% of estimated glomerular filtration rate (eGFR) relative to baseline; 4) total bilirubin ≥2.0 mg/dL and increase by 100% from baseline; 5) platelet count <100 cells/μL and ≥50% decline from baseline; and 6) serum lactate ≥2.0 mmol/L (Supplementary Table S2).

Statistical Analysis

Continuous variables were presented as medians (with interquartile range) and compared by Mann–Whitney U-test. Categorical variables were compared by Chi-square test or Fisher’s exact test.

The Charlson Comorbidity Index was computed according to Supplementary Table S11. The accuracy of our ICD identification algorithm was evaluated by sensitivity (SE), specificity (SP), positive predictive value (PPV), negative predictive value (NPV), and Area Under the Receiver Operating Characteristic curve (AUROC). Confidence intervals (CI) were calculated by delta method. P-value <0.05 was considered as statistically significant. All statistical analyses were performed by statistical software IBM SPSS Statistics for Windows (IBM Corp. Released 2011, Version 20.0. Armonk, NY: IBM Corp.).

Results

A total of 547 patients were enrolled in the final cohort, including 138 patients in the ICU and 409 patients in the general wards (Figure 1). As shown in Table 1, in the ICU setting, there were 103 (75%) patients meeting Sepsis-3 criteria and 91 (66%) meeting modified ICD-10 criteria. In general wards, there were 229 (56%) patients meeting Sepsis-3 criteria and 183 (45%) meeting modified ICD-10 criteria. Clinical characteristics and outcomes were similar between patients meeting Sepsis-3 and modified ICD-10 criteria in both ICU and non-ICU settings. Compared with non-ICU sepsis patients, ICU patients had higher hospital acquired infections (54% vs 31%, p<0.01), blood stream infections (44% vs 20%, p<0.01), multiple infection rates (40% vs 24%, p<0.01), and expenses (142.06 vs 77.51 1000 CNY, p<0.01). Respiratory, blood stream and abdominal infection were the most common infection site.

Table 1 Clinical Characteristics and Outcomes of Patients Enrolled in ICU and Non-ICU Settings

Figure 1 Flowchart of the patients admitted to the non-ICU and ICU. Abbreviations: LOS, Length of stay; ICU, intensive care unit.

Comparison of modified ICD-10, Angus’s, Martin’s, and eSOFA criteria for identification of sepsis were shown in Table 2. For both ICU and non-ICU setting, the sensitivity of modified ICD-10 (70.48%, 95% CI=65.20–75.27%) was higher than Angus’s and Martin’s criteria (30.12%, 95% CI=25.29–35.42% and 25.60%, 95% CI=21.07–30.72%, respectively), lower than eSOFA (80.12%, 95% CI=75.33–84.20%). On the contrary, the specificity of modified ICD-10 (81.39%, 95% CI=75.41–86.23%) was lower than Angus’s and Martin’s criteria (93.49%, 95% CI=89.09–96.26% and 98.14%, 95% CI=94.99–99.40%, respectively) and higher than eSOFA (69.77%, 95% CI=0.63–0.76). The corresponding AUROC value of modified ICD-10 criteria was the highest (0.76, 95% CI= 0.72–0.80). The modified ICD-10 criteria in the ICU setting had a higher sensitivity (84.47%, 95% CI=75.70–90.59% vs 64.19%, 95% CI=57.57–70.33%) and specificity (88.57%, 95% CI=72.32–96.27% vs 80.00%, 95% CI=73.67–85.68%) than th non-ICU setting. Positive and negative likelihood ratios of different identification strategies are shown in Supplementary Table S12.

Table 2 Comparison of Different ICD-Coded Algorithms and eSOFA Criteria for Identification of Sepsis in ICU and Non-ICU Settings

Characteristics Sepsis‑3 (+)/ Modified ICD-10 (+), Sepsis‑3 (+)/ Modified ICD-10 (-), Sepsis‑3 (-)/ Modified ICD-10 (+), and Sepsis‑3 (-)/ Modified ICD-10 (-) Patients

There were 234 (43%) patients who met both Sepsis-3 and modified ICD-10 criteria. In comparison, both the Sepsis-3 (+)/modified ICD-10(-) (n=98) and the Sepsis-3 (-)/modified ICD-10 (+) (N=40) (shown in Table 3) groups were less likely to have respiratory, blood stream, and two or more infections. Also, the SOFA score, qSOFA, eSOFA score, proportion of mechanical ventilation support and vasoactive agents, and mortality rates in both of the groups were significantly lower than the Sepsis‑3+/modified ICD-10+ group. Among 98 Sepsis-3 (+)/modified ICD-10 (-) patients, 29 (30%) patients had abdominal infection and 43 (44%) patients had coagulation dysfunction. 82 of 98 Sepsis-3 (+)/modified ICD-10 (-) patients were not identified as sepsis by the modified ICD-10 algorithm because organ dysfunction ICD codes were not assigned.

Table 3 Characteristics of Infected Patients With or Without Sepsis Defined by Sepsis-3 or Modified ICD-10 Criteria

Discussion

Reliable surveillance of sepsis was important for the estimation of disease burden, prevention initiatives. and national sepsis quality measures.²² In this study, we validated a sepsis surveillance algorithm based on ICD-10 codes. Our findings showed that the modified ICD-10 based algorithm could identify more than 95% of sepsis patients in the ICU and more than 80% of sepsis patients outside the ICU in a single medical center of China. Considering the availability and accessibility of ICD codes in China,²³ a modified ICD-10 based algorithm was a feasible way for population-based sepsis surveillance in China.

Recent published studies have demonstrated that EHRs-based criteria were more accurate than administrative claims in identifying patients with sepsis.^24–27 EHRs-based criteria had several advantages such as objectivity, quantization, and real-time monitoring for hospitalized patients.²⁸ CDC developed a Toolkit based on clinical data directly obtained from the EHRs, eSOFA, to track hospital-level sepsis incidence and outcome.^8,21 In our study, a high eSOFA diagnosis proportion in the Sepsis-3 (+)/modified ICD-10 (-) group showed the validity of EHRs. Although our previous validation study confirmed the performance of the toolkit in a hospitalized Chinese population, the feasibility to use the CDC sepsis surveillance toolkit was limited in most resource-limited areas where EHRs were not available or accessible.²⁹ Administrative claims based modified ICD-10 criteria offered a feasible low-cost approach to extrapolate population-based estimates of sepsis burden in LMICs, where most of the septic patients were admitted in non-ICU settings.³⁰ In addition, our study showed the overall accuracy in sepsis identification was similar between the EHRs algorithm and modified ICD-10 algorithm. Even so, most patients were not identified by modified ICD-10 criteria because of the absence of the corresponding organ dysfunction codes, which could be explained by insufficient coding training. With improving coding practices and more standardized coding procedure,⁷ modified ICD-10 criteria might have better performance in sepsis surveillance.

A major strength of our study was the use of a prospective cohort which could accurately identify patients with sepsis and collect all components of SOFA score and other detailed clinical data. In 2019, Rhee et al²⁶ evaluated the accuracy of sepsis identification strategy based on claims data and clinical data from EHRs. The sensitivity of administrative claims strategy ranged from 5% to 54% for explicit sepsis codes, and from 42% to 80% for implicit codes. However, the robustness of the findings in the study was limited to the retrospectively confirmed diagnosis of sepsis, especially in non-ICU settings. Also, previous studies^7,24–26 validating administrative data algorithms were based on retrospective cohorts which were inevitably influenced by missing data including Glasgow score, laboratory findings, and urinary output. Another strength of our study was the inclusion of patients with sepsis admitted to non-ICU settings,³¹ allowing generalization of our findings to the hospitalized populations.

The study also had several limitations. First, our study was based on a cohort from a single center. Although the ICD coding practice might be different in hospitals across different regions, the data qualities of ICD coding were ensured by manual review (Supplementary Figure S1), quality control meetings, courses of staff training, and regulations for inspection in all public hospitals in China.³² Further validation was warranted for generalization of our algorithms in other hospitals in China. Second, we could not differentiate sepsis-related or non-sepsis related organ dysfunctions based on the ICD codes, which would lead to an overestimation of the sepsis rate. Lastly, we only enrolled patients with infection or suspected infection. Further investigation was needed for population with extremely low incidence of infection.^24,33

Conclusions

This study demonstrated that modified ICD-10 criteria had higher validity compared with Angus’s criteria and Martin’s criteria. Validity of the modified ICD-10 criteria was similar to eSOFA criteria. Modified ICD-10 algorithm can be used to provide an accurate estimate of population-based sepsis burden of China.