Validation of a clinical risk-scoring algorithm for severe scrub typhus

Introduction

“Scrub typhus,” one of several potentially fatal tropical rickettsial infections, is caused by Orientia tsutsugamushi and presents as a systemic, vasculitis-like infection. Clinical manifestations are typically fever, eschar, generalized or regional lymphadenopathy, maculopapular rash, severe headache, and myalgia. Nausea and vomiting, diarrhea, and conjunctival injection can also occur.^1,2 Without appropriate treatment, infected patients usually experience complications in the second week. Those especially at risk of infection are the elderly, those without previous immunity,^2–4 and those with risk factors such as hypoalbuminemia, hyperbilirubinemia, high levels of aspartate aminotransferase (AST) and/or creatinine, and those with lung crepitations.^3,5–10 Patients with complications who are admitted to intensive care units with a high Acute Physiology and Chronic Health Evaluation II (APACHE II) score are at high risk of mortality.⁵ In rural areas where effective treatment is unavailable or delayed, mortality rates have been reported as high as 35%.²

The cost of treatment for scrub typhus patients with complications is much higher due to hospital admissions and more intensive treatments.¹¹ Prognostication of disease severity may help with the identification of disease severity and prompt initiation of treatments.

Prognostication of disease severity has been considered by researchers of various conditions. Examples include a clinical prediction rule for severe community-acquired pneumonia,¹² Mortality in Emergency Department Sepsis score,¹³ a prediction rule for risk of mortality in Clostridium difficile infection,¹⁴ and decision-tree algorithms for dengue fever¹⁵ and dengue hemorrhagic fever.¹⁶ Some of these tools were found to have both advantages and disadvantages when validated.

Previously, we developed a clinical risk-scoring algorithm for scrub typhus severity comprising six predictors from routine practice.¹⁷ The objective of the study reported here was to validate the risk-scoring algorithm developed earlier with an independent dataset obtained from similar patients from a few years later.

Materials and methods

Patients

Data were retrieved retrospectively from the medical files of scrub typhus patients. The criteria for inclusion were similar to those used in the previous study¹⁷ and were based on history of disease exposure and presentation with acute fever and at least one of the following signs and symptoms: myalgia, headache, conjunctival injection, cough, profuse sweating, maculopapular rash, and/or lymphadenopathy accompanied by the presence of eschar, and/or a positive immuno-chromatographic test for scrub typhus.¹⁸

Patients were classified into three groups:

1. non-severe – patients without any complications
2. severe – patients with complications involving at least one of the following organ systems:
• cardiovascular system – presence of any of any of the following: systolic blood pressure <90 mmHg, myocarditis (defined as elevated creatine kinase-myocardial band isoenzyme above baseline), or abnormal cardiac arrhythmia with no previous history of atrial fibrillation, supra ventricular tachycardia, or frequent premature ventricular tachycardia
• respiratory system – presence of acute respiratory distress syndrome, defined as the ratio of partial pressure arterial oxygen and fraction of inspired oxygen (PaO₂/FiO₂) <200 mmHg, with bilateral interstitial infiltration on chest film with normal cardio/thoracic ratio, or no volume overload of central venous pressure from central venous catheter
• central nervous system – presence of any of the following: Glasgow Coma Scale ≤12 without other causes, seizure without other causes, or meningoencephalitis
• hematological system – platelet count ≤20,000/mm³
• urinary system – presence of acute renal failure, defined as creatinine ≥2mg/dL or creatinine change of >0.5 mg/dL/day
• gastrointestinal and hepatobiliary system – presence of hepatitis, defined as elevated AST or alanine aminotransferase (ALT) more than fivefold normal levels.
3. fatal – patients who died in hospital from scrub typhus.

Patients in any intervention trials during the same period were excluded from the data analysis.

Development data

The development data were for patients admitted to two university-affiliated referral hospitals – Nakornping Hospital, Chiang Mai and Chiangrai Prachanukroh Hospital – in the north of Thailand from 2004 to 2010 (n=526).

Validation data

The validation data were for patients admitted to Chiangrai Prachanukroh Hospital from 2011 to 2012 (n=257).

Data analysis

The characteristics of the development data and the validation data were compared using an exact probability test, Student’s t-test, or Wilcoxon rank-sum test as appropriate. The patient risk scores were assigned following the scoring scheme previously developed (Table 1).¹⁷ Patients were categorized into three severity groups: 1) non-severe, 2) severe, and 3) fatal, as described. The predictive performance of the scores was compared by analysis of the area under the receiver-operating characteristic curve (AuROC). Over- and underestimated proportions were calculated and compared.

Table 1 Significant predictors and assigned item scores Abbreviation: AST, aspartate aminotransferase.

Ethical approval

The study was approved by the Ethics Committee for Research in Patients, Faculty of Medicine, Chiang Mai University, and the research ethical committees of the two hospitals.

Results

Data for 257 patients comprised the validation dataset. Of those, 160 cases were defined as non-severe, 62 as severe, and 35 as fatal. The distribution of patients into the three severity groups of the development and the validation datasets was similar. The clinical characteristics of the patients were also similar except in terms of 1) sex, 2) symptoms or signs of conjunctival injection, maculopapular rash, lymphadenopathy, eschar, abdominal pain, stiff neck, and crepitation, 3) vital signs of systolic blood pressure, and diastolic blood pressure (Table 2). The laboratory investigation data also showed similar profiles for both datasets, except in terms of platelet count and AST, alanine aminotransferase, alkaline phosphatase, albumin, and potassium levels (Table 3).

Table 2 Demographic and clinical characteristics of scrub typhus patients in the development and validation datasets Abbreviations: DBP, diastolic blood pressure; SBP, systolic blood pressure; SD, standard deviation.

Table 3 Laboratory test findings Notes: The data were missing for some parameters: in the development set, urine albumin and urine sugar =83 cases; in the validation set, AST =42 cases, albumin =68 cases, creatinine =48 cases, and urine albumin and urine sugar =28 cases. Abbreviations: ALP, alkaline phosphatase; ALT, alanine aminotransferase; AST, aspartate aminotransferase; BUN, blood urea nitrogen; Cl2, chloride; CO2, carbon dioxide; K, potassium; Na, sodium; SD, standard deviation; WBC, white blood cell.

The risk score was assigned to patients with six complete clinical parameters (n=167, 65%). The mean risk scores of patients in the validation data were different to those of patients in the development dataset (mean ± sd 7.1±3.9 and 6.0±3.9, respectively). The proportion of patients classified into the three severity levels was also different (P=0.004; Table 4).

Table 4 Score-classified scrub typhus severity levels in the development and validation datasets Abbreviation: SD, standard deviation.

In the validation dataset, the score correctly predicted 31.7% (53 out of 78) of patients into the non-severe group (score ≤5) with one-level underestimation of 6.6% (eleven patients) and no two-level underestimation, while 14.4% (24 out of 55) of patients were correctly classified into the severe group (score 6–9), with an underestimation of 5.4% (nine patients) and an overestimation of 13.8% (23 patients). Finally, for the fatal severity group (score ≥10), the score was predicted correctly for 14.9% (25 of 34) of patients, with one-level overestimation of 12.0% (20 patients) and two-level overestimation of 1.2% (2 patients), giving a total overestimation of 13.2% (Table 5).

Table 5 Score-classified severity levels, criterion-classified severity levels, and risk estimation validity Note: *Percentage of total patients. Abbreviations: IQR, interquartile range; SD, standard deviation.

The overall predictive performance (as assessed by the AuROCs) of the validation dataset was insignificantly lower than that of the development dataset, both in the discrimination between non-severe versus (vs) severe and fatal (AuROC =88.74% vs 91.48%, P=0.324) and between fatal versus non-severe and severe (AuROC =88.66% vs 91.22%, P=0.407; Table 6 and Figure 1).

Table 6 Discriminative performance of scrub typhus severity score in the development and validation datasets Abbreviations: AuROC, area under the receiver-operating characteristic curve; CI, confidence interval.

Figure 1 Score-predicted probability of severity in the development data (solid lines) and the validation data (dashed lines). Note: Vertical dotted lines represent score-derived criteria for classifying patients into non-severe, severe, and fatal groups.

Discussion

Validation is needed for any clinical prediction rules before they can be adopted in clinical practice, either internally or externally. The Mortality in Emergency Department Sepsis score, for example, was developed to evaluate the risk of death in patients in emergency departments, in which validation in one-third of the sample yielded similar discrimination (AuROC =0.82 vs 0.78).¹³ Further, a clinical prediction rule for severe community-acquired pneumonia was developed to screen patients for initial treatments in emergency departments. The rule performed similarly to the development data when prospectively validated in patients from the same hospitals (AuROC =0.92). However, the performance was poorer when externally validated in patients from other hospitals (AuROC =0.80).¹² External validation of a prediction rule for risk of mortality in Clostridium difficile infection also yielded poorer performance (AuROC =0.704 vs 0.653).¹⁴ It is very common for the performance of the validation data to be poorer than that of the development data.

Our clinical risk-scoring algorithm for severe scrub typhus was validated against patients admitted in only one hospital some years after the patients used in the development dataset. When only patients with complete information on the six predictors (n=167) were selected from the validation dataset, the performance of the algorithm was somewhat poorer than with the development data, both in terms of differentiating non-severe from severe and fatal (AuROC =88.74% vs 91.48%), and fatal from non-severe and severe (AuROC =88.66% vs 91.22%). However, these differences were not statistically or clinically significant. The overall proportion of correct classifications was somewhat lower (61.0% vs 68.3%), but overestimation was higher (27.0% vs 25.8%), which may be considered more beneficial to the patients.¹⁷

There were incomplete data on important predictors for 90 of 257 patients (35%). The missing data were AST (n=42, 24.1%), albumin (n=68, 26.5%), and creatinine (n=48, 18.7%). These patients were in the non-severe group, which did not require organ-specific investigations. We did the sensitivity analysis by replacing the missing data by the mean values calculated from existing nonmissing data for each of the corresponding variables. When this was done, the discrimination between non-severe and severe and fatal was reduced (AuROC =84.41%) but was still considered excellent,¹⁹ while the discrimination between fatal and non-severe and severe was unchanged. However, sensitivity analysis by multiple imputation of the missing data yielded almost identical performances (data not shown).

These sensitivity analyses imply that the derived risk-scoring algorithm might be successfully and safely applied to classify future patients, especially those with severe and fatal risk of scrub typhus. The score should help physicians in charge determine the need for further investigations and start the necessary treatments for those at these risk levels.

The main advantage of the risk-scoring algorithm is that it requires routinely available patient information, comprising demographic, physical examination, and laboratory test data.¹⁷ In endemic areas with restricted health resources,^20–22 the score may help inform clinical decision making with regard to both investigations and treatments.

In the present study, data on the validation patients were collected retrospectively and were based on provisional diagnosis made as part of routine clinical practice, in which scrub typhus cases may reflect “suspected” rather than “confirmed” cases, by definition. Therefore, prospective validation to patients with a theoretically defined diagnosis of scrub typhus¹⁸ would be helpful to confirm the algorithm’s feasibility for future use in routine practice.

Conclusion

A risk-scoring algorithm developed previously to classify scrub typhus patients into the three severity levels was validated using a different set of patients. The algorithm demonstrated good performance in this subsequent set of patients. The algorithm was clinically acceptable and could be applied in routine clinical practice to help identify patients with severe prognosis in order to judge the need for more intensive investigation and treatments.

Acknowledgments

The authors wish to thank the authorities of Chiangrai Prachanukroh Hospital and Nakornping Hospital, Chiang Mai for their permission to collect data and wish to thank Chiang Mai University for its financial support.

Disclosure

The study was partially granted by the Faculty of Graduate, Chiang Mai University. The authors declare no other conflicts of interest in this work.

---

References

1.	Lee N, Ip M, Wong B, et al. Risk factors associated with life-threatening rickettsial infections. Am J Trop Med Hyg. 2008;78(6):973–978.
2.	Day NP, Newton P, Parola P, Raoult D. Scrub typhus and other tropical rickettsioses. In: Cohen J, Powderly WG, Opal SM, editors. Infectious Diseases. 3rd ed. United States. Elsevier; 2010;2:1231–1237.
3.	Kim DM, Kim SW, Choi SH, Yun NR. Clinical and laboratory findings associated with severe scrub typhus. BMC Infect Dis. 2010;10:108.
4.	Wu KM, Wu ZW, Peng GQ, Wu JL, Lee SY. Radiologic pulmonary findings, clinical manifestations and serious complications in scrub typhus: experiences from a teaching hospital in eastern Taiwan. Int J Gerontol. 2009;3(4):223–232.
5.	Lee CS, Hwang JH, Lee HB, Kwon KS. Risk factors leading to fatal outcome in scrub typhus patients. Am J Trop Med Hyg. 2009;81(3):484–488.
6.	Sriwongpan P, Krittigamas P, Kantipong P, Kunyanone N, Patumanond J, Namwongprom S. Clinical indicators for severe prognosis of scrub typhus. Risk Manag Healthc Policy. 2013;6:43–49.
7.	Varghese GM, Abraham OC, Mathai D, et al. Scrub typhus among hospitalised patients with febrile illness in South India: magnitude and clinical predictors. J Infect. 2006;52(1):56–60.
8.	Chanta C, Triratanapa K, Ratanasirichup P, Mahaprom W. Hepatic dysfunction in pediatric scrub typhus: role of liver function test in diagnosis and marker of disease severity. J Med Assoc Thai. 2007;90(11):2366–2369.
9.	Thap LC, Supanaranond W, Treeprasertsuk S, Kitvatanachai S, Chinprasatsak S, Phonrat B. Septic shock secondary to scrub typhus: characteristics and complications. Southeast Asian J Trop Med Public Health. 2002;33(4):780–786.
10.	Hu ML, Liu JW, Wu KL, et al. Short report: Abnormal liver function in scrub typhus. Am J Trop Med Hyg. 2005;73(4):667–668.
11.	Lee CS, Min IS, Hwang JH, Kwon KS, Lee HB. Clinical significance of hypoalbuminemia in outcome of patients with scrub typhus. BMC Infect Dis. 2010;10(1):216.
12.	España PP, Capelastegui A, Gorordo I, et al. Development and validation of a clinical prediction rule for severe community-acquired pneumonia. Am J Respir Crit Care Med. 2006;174(11):1249–1256.
13.	Shapiro NI, Wolfe RE, Moore RB, Smith E, Burdick E, Bates DW. Mortality in Emergency Department Sepsis (MEDS) score: a prospectively derived and validated clinical prediction rule. Crit Care Med. 2003;31(3):670–675.
14.	Butt E, Foster JA, Keedwell E, et al. Derivation and validation of a simple, accurate and robust prediction rule for risk of mortality in patients with Clostridium difficile infection. BMC Infect Dis. 2013; 13(1):316.
15.	Tanner L, Schreiber M, Low JG, et al. Decision tree algorithms predict the diagnosis and outcome of dengue fever in the early phase of illness. PLoS Negl Trop Dis. 2008;2(3):e196.
16.	Lee VJ, Lye DC, Sun Y, Leo YS. Decision tree algorithm in deciding hospitalization for adult patients with dengue haemorrhagic fever in Singapore. Trop Med Int Health. 2009;14(9):1154–1159.
17.	Sriwongpan P, Krittigamas P, Tantipong H, Patumanond J, Tawichasri C, Namwongprom S. Clinical risk scoring algorithm to forecast scrub typhus severity. Risk Manag Healthc Policy. 2014;7:11–17.
18.	World Health Organization (WHO). WHO Recommended Surveillance Standards. 2nd ed. WHO/CDS/CSR/ISR/99.2. Geneva: WHO; nd. Available from: http://www.who.int/csr/resources/publications/surveillance/whocdscsrisr992.pdf. Accessed October 5, 2011.
19.	Hosmer DW Jr, Lemeshow S, Sturdivant RX. Applied Logistic Regression. 3rd ed. New York, NY: Willey; 2013.
20.	Abrahamsen SK, Haugen CN, Rupali P, et al. Fever in the tropics: aetiology and case-fatality – a prospective observational study in a tertiary care hospital in South India. BMC Infect Dis. 2013;30(13):355.
21.	Lee JH, Chung KM, Kim ES, et al. Dynamics of clinical symptoms in patients with scrub typhus. Jpn J Infect Dis. 2013;66(2):155–157.
22.	Vikrant S, Dheer SK, Parashar A, et al. Scrub typhus associated acute kidney injury – a study from a tertiary care hospital from western Himalayan state of India. Ren Fail. 2013;35(10):1338–1343.