Establishment and Validation of a Risk Prediction Model for Mortality in Patients with Acinetobacter baumannii Infection: A Retrospective Study

Introduction

Nosocomial infection caused by A. baumannii, especially multi-drug resistant strains, has become a growing concern in recent years. A. baumannii can cause various types of hospital infections, with the most common infection site being respiratory tract, such as ventilator-associated pneumonia (VAP). It can also cause bacteremia, meningitis, endocarditis, peritonitis, and skin and urinary infections. Patients in intensive care unit (ICU), newborns, and patients with low immune function are at high risk.^1–5 Due to prolonged exposure to antibiotics over the last decade, A. baumannii has developed strong resistance and can survive in harsh conditions, and carbapenem-resistant A. baumannii (CRAB) has become hyperly prevalent in hospitals.⁶ Several risk factors for A. baumannii infection-related mortality have been identified, including underlying diseases, mechanical ventilation (MV), ICU admission, RBC transfusion, SOFA score and CRAB.^1,7–10 Patients with CRAB bacteraemia also have a higher risk of early death, regardless of the source of infection or underlying diseases.¹¹ Shi et al found that in the gram-negative bloodstream infection (BSI) cases, A. baumannii was the most common carbapenem-resistant (CR) isolate (46.15%).¹² However, more and more studies are about the analysis of risk factors of A. baumannii infection in a single-system, such as respiratory system, blood stream infection and wound surface. Many patients with basic diseases have a significantly higher mortality rate after infection with A. baumannii.^8,13–15 Early diagnosis and prediction of the risk of death are crucial for clinical management and reducing mortality in patients with A. baumannii infection. Therefore, there is a need to establish an early death prediction model to predict the mortality of A. baumannii infected patients, providing timely warning and guiding the optimization of treatment plans.

The initial research of our team has successfully established a reliable prediction model for death of A. baumannii infection patients.¹⁶ The aim of this study is to further refine and verify the accuracy and reliability of the risk prediction model to provide more credible insights into the high risk of death caused by A. baumannii infection, and to guide the clinicians to select appropriate treatment plans.

Patients and Methods

Patient Selection

A study cohort and a validation cohort were included in this study. Patients with A. baumannii infection admitted at the First Affiliated Hospital of Anhui Medical University from January 2013 to December 2018 were continuously enrolled as the study cohort. The validation cohort included patients with A. baumannii infection from the 901st Hospital between January 2016 and December 2021. According to the formula for calculating the sample size when comparing the limited population percentage in descriptive research,¹⁷ the mortality value comes from the previous research report of 27%,⁷ the total sample size in this study is 1099. The calculated sample size is about 238, and the sample size of study group in this study is 622, which exceeds the calculated sample size requirement. The clinical data of all patients were analyzed retrospectively. The study was in line with the Guidelines for Transparent Reporting of a multivariable model for Individual Prognosis or Diagnosis (The TRIPOD statement).¹⁷

Inclusion Criteria

The inclusion criteria were as follows:

1. A. baumannii-infected patients hospitalized in the 901st Hospital from January 2016 to December 2021 and patients hospitalized with A. baumannii infection in the First Affiliated Hospital of Anhui Medical University from January 2013 to December 2018.
2. All patients were over 18 years old.

1) The clinical diagnosis of A. baumannii infection conformed to the standards of the Centers for Disease Control and Prevention.¹⁸ The definition of A. baumannii infection needs to be combined with clinical evidence, such as chills and fever, procalcitonin elevation and leukocytosis besides laboratory and imaging results. Specimens with positive culture results come from different infected sites, such as cerebrospinal fluid (CSF), respiratory tract, soft tissue, urine, skin, pleural and ascites fluid, and blood. This study only included patients infected with A. baumannii for the first time.

2) Once A. baumannii is cultivated, the samples obtained from blood, CSF, pleural effusion and ascites will be regarded as infection. However, non-sterile samples, such as sputum, secretions and urine, must meet quality control standards and quantitative culture methods should be used. The following sputum samples were qualified: WBC/microscope field ≥25, epithelial cells <10. The following results are considered to be of great significance: when the number of bacteria in bronchoalveolar lavage fluid and tracheal aspirate is ≥10⁴ CFU/mL and ≥10⁶ CFU/mL respectively, it is diagnosed as lower respiratory tract infection. Catheter culture threshold ≥10³CFU/mL or middle urine ≥10⁵ CFU/mL.

Exclusion Criteria

Patients with the following situations will be ruled out:

1. Contaminated samples isolated from patients.
2. Strains cultivated from the same patient.
3. Asymptomatic infection or colonization patients, including patients with recessive infection, no abnormal auxiliary examination results and no antibacterial treatment.
4. Patients infected with pathogens other than A. baumannii.
5. Patients with incomplete data.

Data Collection

Data of each patient were collected for statistical analysis, including age, sex, infectious source (CSF, blood, respiratory tract, pleural fluid and skin, ascites, soft tissue, and urine), CRAB, serum albumin value, Charlson comorbidity index (CCI) on the day (or within 24 hours) with A. baumannii cultivated,¹⁹ ICU admission history, MV, anemia, serum albumin value, and hospitalization history. Hospitalization within 3 months before A. baumannii infection is defined as hospitalization history. The diagnostic criteria for anemia are hemoglobin levels of female and male patient on the day (or within 24 hours) with positive microbial culture lower than 110 g/L and 120 g/L, respectively. Mortality within 30 days after A. baumannii infection was calculated.

Statistical Analysis

Multiple regression logistic analysis (Statistical software package R and Empower stats (http://www.empowerstats.com)) was performed in this study, with the dependent variable being the possibility of death within 30 days after A. baumannii infection and the independent variables being the 10 variables mentioned above. The risk factors for death caused by A. baumannii infection were identified based on univariate and multivariate analyses. Considering the relatively small number of death cases, LASSO regression was performed to minimize the number of variables,²⁰ so that to reduce the risk of over-fitting and increase the EPV (events per variable).

Using multiple regression logistic analysis, we analyzed that the dependent variable was the possibility of death of patients within 30 days after CRAB infection, and the independent variable is the above 10 variables. Univariate and multivariate analysis were used to find the risk factors of death caused by CRAB infection. However, in view of the relatively few deaths, in order to reduce the risk of over-fitting, we have to improve EPV (events per variable) as much as possible, so instead of selecting which variables to model based on P value, we have selected the predictor through LASSO regression to minimize the number of variables to model.²⁰

Statistical software package R and Empower stats (http://www.empowerstats.com) are used for statistical analysis and drawing LASSO coefficient path diagram, Cross-validation curve, ROC curve, calibration chart and DCA curve. P value <0.05 indicates that the difference between the obtained results is statistically significant.

According to the regression coefficient of the selected independent variables, the nomogram prediction model is established using the study cohort data. Full model and stepwise selection model are used to establish the prediction model. The performance of the model is evaluated by the two indicators: discrimination for assessing the prediction accuracy of a single result and calibration degree for assessing the accuracy of function point estimation. The area under the curve (AUC) of the receiver’s operating characteristic (ROC) curve was calculated.²¹ AUC between 0.5 and 0.75 indicates that the model has a general discrimination, while AUC >0.75 indicates that the model has a good discrimination.²² The validation cohort data were applied to verify the accuracy and reliability of the risk prediction model used.

The decision curve analysis (DCA) is usually used to evaluate the net benefit of the model in clinical practicability. The reliability of nomogram in clinical practice is evaluated by DCA.²³ The “decision curve” is drawn according to the threshold probability, so it is necessary to calculate the difference between the true positive rate and the weighted false-positive rate of different threshold probabilities in the verification queue and get the net income.

The calibration degree of the prediction model is represented by a calibration chart, which is a graphical representation of the correlation between the actual result frequency and the probability predicted by the model. The probability values predicted by a well-calibrated prediction model should all fall near the 45-degree diagonal.

Ethics Statement

The ethics committees of the 901st Hospital (The 901st Hospital of PLA 202302001) and the First Affiliated Hospital of Anhui Medical University (Quick-PJ 2020-03-10) jointly approved the study. Because it is a retrospective study, the ethics Committee can approve it without informed consent. All research data are anonymous. This study follows the Helsinki Declaration.

Results

Patient Demographics

In this study, 1099 patients with A. baumannii infection were recruited, including 622 patients in the study cohort and 477 patients in the validation cohort, respectively (Table 1). After comparing the baseline data of both cohorts, it was found that there were significant differences in age, sex, hospitalization history, ICU admission history, anemia, MV, serum albumin value, and CCI, whereas no significant differences were observed in terms of the source of infection and CRAB.

Table 1 Clinical Characteristics and Demographic of Two Groups

Nomogram Development

Based on LASSO regression analysis, according to the cross-validation curve, five variables corresponding to λ-se equation have been selected for modeling initially (Figure 1). However, three variables, CRAB, CCI, and MV, were finally selected according to the non-zero compression coefficient value in the analysis results, which were selected for subsequent single-factor and multi-factor regression analysis (Table 2). The other variables were discarded. There is a significant correlation of the three variables with the death of patients with A. baumannii infection (P<0.001) (Table 3). The mortality rate of patients infected with CRAB was 34.4% which was much higher than that of patients infected with carbapenem-sensitive A. baumannii (CSAB) (7.4%). Similarly, the mortality rate of patients receiving MV treatment is much higher than that of patients not receiving MV treatment ((40.6%) and (12.4%) respectively). At the same time, the CCI OR value is obviously greater than 1, which fully indicates that it has a strong correlation with the death of A. baumannii infection. Based on the above results, we established a personalized nomogram to predict the death risk of patients infected with A. baumannii. (Table 3).

Table 2 LASSO Regression Analysis of Modeling Group

Table 3 Univariate and Multivariate Logistic Regression Results of the Study Cohort

Figure 1 LASSO coefficient profiles of the 10 risk factors (A). Three risk factors selected using LASSO Cox regression analysis (B).

In order to draw the nomogram, the point index corresponding to each predictive variable was first obtained, and the total score is counted. The total risk score was used to calculate the probability related to the death risk of A. baumannii infection (Figure 2). By vertical alignment of each variable element with a point on the “point” axis, numerical values are obtained. The corresponding point on the axis of the total point were selected, which was then plotted on the death axis to determine the patient’s probability of death due to A. baumannii infection. For instance, if a patient is infected with CRAB and has a CCI score of 6, and receives MV treatment, resulting in scores of 52, 75, and 52.5, respectively, the total score is 52+75+52.5 = 179.5. Based on this model, the predicted risk of death is 0.72 (72%) (Figure 3).

Figure 2 Nomogram for predicting the risk probability of death due to A. baumannii infection. Abbreviations: CRAB, carbapenem-resistant A. baumannii; CCI, Charlson comorbidity index; MV, mechanical ventilation.

Figure 3 Predicting the risk probability a patient’s death after A. baumannii infection by nomogram.

We present a specific regression model as the basis of nomogram.

Nomogram Validation

The prediction model is verified by internal verification and external verification. Internal verification is done through bootstrapping, times=500. External verification is implemented with validation cohort data. After discrimination and correction, the AUC value of the death risk using the study cohort and the validation cohort data was 0.75 and 0.69, respectively (Figure 4). The ROC curve analysis results of the two groups are shown in Table 4. DCA curve analysis shows that when the probability falls between 30% and 80%, the net income of the two groups is relatively high (Figure 5). The calibration curves swinging around the 45 diagonal line, indicating high calibration and discrimination ability (Figure 6).

Table 4 ROC Curve Analysis Results of Two Group

Figure 4 ROC curve of nomogram division with binomial verification of the study cohort (A) and validation cohort (B). The AUC of the study cohort and the validation cohort was 0.75 and 0.698, respectively.

Figure 5 Decision curve analysis (DCA) curves of the study cohort (A) and validation cohort (B). Verify the net income of Nomogram was verified by DCA curves of the two cohorts.

Figure 6 Calibration curve verifies the calibration of nomogram.

Discussion

A. baumannii infection, especially that caused by multi-drug resistant strains, has raised worldwide concern due to its increasing mortality rate. Our analysis found that the mortality rate of patients in the study cohort and the validation cohort was 27.0% (168/622) and 33.3% (159/477), respectively. The results are similar to those of previous studies.^11,22–25 Improper empirical antibiotic treatment and the severity of the disease may play a major role. Some scholars found that the exposure of carbapenems increased the risk of VAP resistance of A. baumannii.¹ Carbapenem resistance is an independent risk factor for the increase of mortality of A. baumannii infected people, as well as for the increase of ICU utilization, hospitalization time, medical expenses, and readmission rate of A. baumannii infection.²² This research found that approximately 80% of A. baumannii infections originate from the respiratory tract, with an MV rate of about 60%. Therefore, respiratory tract infection caused by MV is considered as one of the most important risk factors for A. baumannii infection.

According to the monitoring, though the proportion of A. baumannii does not significantly change, the rate of multi-drug resistance increases over time. According to the monitoring report of CHINET, the resistance rate of A. baumannii to carbapenems increased from 31%~39% in 2005 to 77.1%~78.1% in 2018.^26,27

A study showed that high CCI value is one of the risk factors leading to death in elderly patients with nosocomial urinary tract infection.²⁸ Therefore, it is considered that CCI is one of the independent risk factors of A. baumannii infection death. This is consistent with our study. However, different from the previous model, in order to make the CCI value play a better role in the prediction model, continuous variables are used in this study, instead of being artificially decomposed into binary variables with CCI value ≥4 and less than 4. For example, when the CCI value is equal to 3, the value is about 12 more than when the CCI value is 2, but if it is split into two categories, then there is no difference. In addition to the above situation, we divide the source of infection into respiratory tract and non-respiratory tract when processing the data. Considering that the total number of other infected sites is too low to be representative, and the overall frequency of death events is not high, it is likely that the mortality rate of patients in some infected sites will be overestimated or underestimated (for instance, the mortality rate of urinary tract infections is 0). Therefore, we believe that the current model is more accurate.

The nomogram prediction model is a relatively new approach for predicting death, which is both practical and convenient. With advancements in sample size and data from two centers, the predictive ability of this model can be continuously improved. The objective of this study is to enhance the accuracy and practicality of the original prediction model. Compared to the previous model, there are several advantages in this study. The cases in this study come from different clinical settings. Internal and external verification were performed using a larger sample size to further verify the stability, clinical feasibility, and reliability of the model. Besides, a new statistical analysis method using LASSO to select independent variables was applied, avoiding the drawbacks of previous modeling method and reducing the risk of over-fitting.

Limitation

Although the prediction model shows promise, there are certain limitations that must be taken into consideration. Firstly, the sample source is derived from two medical institutions, further data collection from additional centers will be necessary. Secondly, due to the omission of certain variables to avoid the risk of over-fitting, potentially significant predictors of mortality may have been disregarded. To address this issue, future research should focus on conducting repeated variable screening to refine the model’s predictive capability. Thirdly, in our statistical analysis, we set samples obtained from the respiratory tract and those from other sites of infection as binary variables, which may have minimized over-fitting but also overlooked the potential impact of non-respiratory infections on patient outcomes. Therefore, it is important to continue investigating the influence of different types of infections on patient death.

Conclusion

In conclusion, this study has successfully established a relatively optimistic prediction model for assessing the death risk of patients with A. baumannii infection. The model’s stability has been affirmed through external validation, and the sensitivity can be accepted, but the specificity is not optimistic. If this prediction model is wanted to be applied to clinical practice, more sample sizes will be used to improve the performance of the prediction model, especially samples from different medical centers.