Predicting Antibiotic Resistance in ICUs Patients by Applying Machine Learning in Vietnam

Introduction

Antibiotic resistance (AMR) is one of ten major global health challenge facing both high-income countries (HICs) and low- and middle-income countries (LMICs).¹ Two main factors, included the inappropriate use of antibiotics in the community and the misprescription of broad-spectrum antibiotics in health-care settings, are contributing to the development of AMR.^2,3 In hospitals, intensive care units (ICUs), where the mortality rate of infections could be as high as 80%, and the overuse of antibiotics is common, are the greatest threat of AMR.⁴ In previous study conducted in 1265 ICUs from 75 countries, nearly half of the patients admitted to these units found hospital-acquired infections.⁵ In Vietnam, a high incidence of nosocomial infections and AMR were found among hospitalized patients, especially in ICUs.^6–8 Antibiotic susceptibility tests in hospitals usually require 36–72 hours for sample cultivation and identification of bacteria.⁹ Therefore, ICU physicians often prescribe antibiotics for prophylaxis based on their experience and the patient’s clinical symptoms rather than laboratory test results as known as empiric therapy.¹⁰ However, during empiric therapy, two main types of errors might occur, including the prescription of inefficient antibiotics and the prescription of antibiotics with coverage that is too broad.

In recent years, artificial Intelligence (AI) and machine learning (ML) are used extensively in HICs to detect and control AMR in laboratories and clinical institutions.¹¹ ML is designed to predict outcome variables using an algorithm to enable “machines” to learn the “rules” from the data.^12–14 The parameters of the models are first customized with a training dataset and then applied to the test dataset to evaluate the prediction performance.¹⁴ ML is increasingly being applied in ICUs to identify AMR and to assist empiric antibiotic therapy. In a review of 60 clinical decision support systems using machine learning, 24 systems (or 40%) were applied in ICUs for diagnosing or treating infectious diseases.¹² However, few studies have been conducted in LMICs using ML to detect AMR and prescribe antibiotics in ICUs.¹¹

The present study used electronic medical records (EMR) of ICU patients at Phu Tho Hospital and Military Hospital 175 (Vietnam). Then, different machine learning algorithms were used to develop AMR prediction models of target bacteria for the most frequently used antibiotic families in AST results in two Vietnamese hospitals. This study aimed to evaluate the performance of ML models for predicting AMR bacteria and resistance to antibiotics in two Vietnamese hospitals. Findings from this study are expected to assist clinicians in making better choices of empiric antibiotic therapy for ICU patients in Vietnam and other LMICs.

Materials and Methods

Study Area

The study was implemented at Phu Tho Hospital (Phu Tho province) and 175 Hospital (Ho Chi Minh City) in Vietnam. These two hospitals were selected based on purposive sampling. Phu Tho Hospital is a provincial-level hospital, whereas 175 Hospital is a central level hospital. The first one is in a rural, mountainous, and midland province; the second is in an urban, populated, and delta city in Vietnam.

Study Design

This study applied the retrospective cross-sectional design. The data were extracted from the EMRs of patients admitted to the ICUs of Phu Tho Hospital and 175 Hospital in Vietnam between January 1st, 2020, and June 30th, 2022. The criteria for selection were the medical records of all patients who were 18 years of age or older on admission (in 175 Hospital) or at the time of taking the AST test (in Phu Tho Hospital) and had positive bacterial cultures.

Data Collection

We retrieved EMRs of patients who had positive bacterial culture results from the ICUs in Phu Tho Hospital (Phu Tho province) and Military Hospital 175 (Ho Chi Minh City) in Vietnam between January 1st, 2020, and June 30th, 2022. The total of 3326 specimens were positive for bacteria in the two hospitals (1121 in Phu Tho Hospital and 2205 in Military Hospital 175). After checking for duplicates, 2432 specimens, of which 856 were from Phu Tho Hospital, and 1576 were from Military Hospital 175, were included in the final data analysis of the present study. 1625 medical records were collected, of which 643 were from Phu Tho Hospital and 982 from Military Hospital 175. The patients from both hospitals were 1296 people, including 600 patients in Phu Tho Hospital and 696 in Military Hospital 175.

The demographic and clinical information (age, gender, place of residence (province and town/district), occupation, having insurance or not, diagnosis at admission by the International Code of Diseases (ICD-10), dates of admission and discharge, treatment results, and complication) were collected from the patient’s medical history.

Antimicrobial Resistance Patterns of Isolated Bacteria

Different types of specimens, such as blood, cerebrospinal fluid (CSF), tracheobronchial/bronchoalveolar fluid, urine, skin/wound/tissue specimens, catheters, pleural and peritoneal fluid, were collected and used for the Antimicrobial Susceptibility Testing (AST). The antibiotic susceptibility of isolated bacteria was detected by VITEK 2 Compact System (bioMérieux) at the Testing Center in Phu Tho Hospital and BD Phoenix 100 system (Becton Dickinson, USA) at the Department of Microbiology in the Military Hospital 175. All tests were standardized and performed following the criteria of the Clinical and Laboratory Standards Institute (CLSI).¹⁵

Development and Evaluation of ML Models

At first, all input data were preprocessed to be compatible with machine learning models by addressing issues of variations in data format and high dimensionality. The data, then, were split into a training set (80% of the data) to fit the model to the data and a testing set (20% of the data) to evaluate the model’s ability to predict new data accurately, using k-fold cross-validation.¹⁶ The training set was used to fit the model to the data, while the test set was used to evaluate the model’s ability to predict new data accurately. This process helps to prevent overfitting, where the model is overly tailored to the training data and may not generalize extensively to new data. In the present study, we developed five models using five supervised ML algorithms, including regularized logistic regression (Ridge logistic regression), adaptive boosting decision trees (AdaBoost), random forest, XGBoost, and LightGBM.

Development and evaluation of a predictive model were performed on a three-phase process. The initial phase determined the presence of five targeted bacterial strains. The second phase predicted the top six antibiotic families most used in AST tests and prescriptions in the hospital, specifically, aminoglycosides, fluoroquinolones, polymyxins, carbapenem, fourth-generation cephalosporin, and trimethoprim derivatives. Finally, the combination of bacterial strains and the corresponding antibiotic resistance profiles were identified.

Eleven combinations were selected based on the prevalence of antibiotics used in AST and the resistance level of each kind of bacteria, including (i) four bacteria against aminoglycosides (Klebsiella spp, Pseudomonas aeruginosa, Escherichia, Staphylococcus aureus), (ii) one bacterium against polymyxins (Klebsiella spp), (iii) one bacterium against fluoroquinolones (Staphylococcus aureus), (iv) one bacterium against carbapenems (Escherichia coli), (v) two bacteria against fourth-generation cephalosporin (Escherichia coli and Pseudomonas aeruginosa), and (vi) two bacteria against trimethoprim derivatives (Staphylococcus aureus and Escherichia coli). Each dataset was run and evaluated for 22 targeted variables (including five bacteria, six antibiotics, and eleven combinations).

The performance of the various machine learning models was evaluated using a range of metrics, including sensitivity, specificity, precision, accuracy, the area under the receiver operating characteristic curve (AuROC), the harmonic mean of precision and recall (F1-score), normalized Matthew Correlation Coefficient (normMCC), Precision–Recall Curve (PRC). For binary classification and imbalanced datasets, norm MCC and PRC are more informative and valuable besides the indicators of accuracy, F1-score, and AuROC Plot. SHAP (SHApley Additive Explanations) package was also used to assess the impact of each feature on the final prediction of the machine learning models. This allowed the contribution of each variable to the model’s output to be determined.

Two datasets were prepared to predict AMR bacteria and antibiotics with ML models. The first dataset (dataset 1) included demographic information, clinical diagnoses, treatment results, and AST results, and complete blood count data. The second dataset (dataset 2) included demographic information, clinical diagnoses, treatment results, and AST results, complete blood count and biochemical data. The performance of the prediction models was evaluated by various indicators (sensitivity, specificity, precision, accuracy, F1-score, PRC, AuROC, and NormMCC) to determine the optimal time point for data selection.^17–20 A reference dataset was created by combining the data from Military Hospital 175 and Phu Tho Hospital based on available standard variables.

Statistical Analysis

The data were cleaned and checked for missing values, then reshaped and analyzed using Python 3.8 with various packages, including Pandas, Numpy, Statsmodels, Matplotlib, and Seaborn.

Results

The Performance of the Predictive Model of the Five Targeted Bacteria in the Dataset 1 and Dataset 2

Table 1 presents summary statistics of the datasets used for machine learning predictive models. In the dataset 1 (without biochemical test results), 684 patients were selected. A total of 953 health records, 1537 AST results, and 7538 complete blood tests were included (Table 1). In dataset 2 (with biochemical test results), 560 patients with 755 health records were selected. A total of 1182 AST test, 5924 complete blood count test and 4058 biochemical test results were analyzed (Table 1). In the reference dataset, all patients and health records of two hospitals were included in the predictive models.

Table 1 Description of Datasets Used for Machine Learning Predictive Models

The sensitivity, specificity, precision, accuracy, F1-score, AuROC, PRC, and normMCC were compared among the five machine learning models in the dataset 1 and dataset 2 and shown in Table 2 and Table 3, respectively. The accuracy, F1-score, AuROC, and normMMC of LightGBM, XGBoost, and Random Forest models were higher than those of other models in both datasets. In both datasets 1 and 2, accuracy, F1-score, AuROC and normMCC of the XGBoost model were the highest among five models (from 0.890 to 1.000). For database 1, XGBoost was the most accurate model in predicting all five targeted bacteria except Pseudomonas aeruginosa (Table 2). Random Forest models gave the highest specificity score for predicting Escherichia coli (0.941) and Staphylococcus aureus (0.932). In dataset 2, the high sensitivity and precision were determined for all the models, but the specificity was still low, particularly for predicting Escherichia coli and Staphylococcus aureus (Table 3). Only Random Forest models had specificity scores higher than 0.850 (Table 3).

Table 2 Comparing the Performance of Different Machine Learning Algorithms on the Dataset 1

Table 3 Comparing the Performance of Different Machine Learning Algorithms on the Dataset 2

The Performance of the Predictive Model of the Resistance to Antibiotic Families and Its Combinations of Bacteria

The predictive model of the resistance to antibiotic families was performed with the isolated bacterial species. Overall, high scores of sensitivity, accuracy, precision, F1-score, and normMCC indicated that the models were making accurate predictions for datasets 1 and 2. Tables 4–6 showed the sensitivity, specificity, precision, accuracy, F1-score, AuROC, PRC, and normMCC of the performance of LightGBM model to predict resistance to antibiotic families and the combinations between selected bacteria and antibiotic families in the dataset 1, 2, and reference, respectively.

Table 4 Performance of LightGBM Model (5-Fold Cross-Validation) to Predict Resistance to Antibiotic Families and Between Bacteria and Antibiotic Families in Dataset 1

Table 5 Performance of LightGBM Model (5-Fold Cross-Validation) to Predict Resistance to Antibiotic Families and Between Bacteria and Antibiotic Families in Dataset 2

Table 6 Performance of LightGBM Model (5-Fold Cross-Validation) to Predict Resistance to Antibiotic Families and Between Bacteria and Antibiotic Families in Reference Dataset

Discussion

In the present study, we used five machine learning models across three datasets with more than 110 variables to predict the AMR of bacteria and antibiotic families. Although no single model showed superiority for all bacteria or performance metric, the study yielded highly reliable results with AUROC scores of over 0.850 in 3 algorithms LightGBM, XGBoost, and Random Forest, which were higher than the results of the recent studies in Greece and Israel on the prediction of AMR of bacteria, antibiotics, or bloodstream infection.^4,21–23 The normMCC value reached above 0.750 for the LightGBM algorithm in predicting bacteria and AMR and predicting combinations of bacteria against antibiotic families. All machine learning models in this study were evaluated using this indicator, which is more reliable and informative than accuracy or F1-score for the binary classification.²⁰ LightGBM model was fast and gave high-performance results,^21,24 but its specificity score was not as high as those of other algorithms like Random Forest and XGBoost.

In the two datasets, Random Forest showed the highest specificity results for minority bacterial incidences (Pseudomonas aeruginosa, Escherichia coli, and Staphylococcus aureus). In dataset 2, XGBoost and LightGBM generated results with high sensitivity (0.950) but average specificity (0.650), whereas Random Forest produced high sensitivity and specificity results (0.840 to 0.990). Although no studies on AMR prediction showed the excel of Random Forest in specificity index, the application of k-fold validation when preparing data may have significantly increased accuracy, reliability, and specificity compared to ordinary Random Forest alone.²⁵ The limitations and advantages of the LightGBM, XGBoost, and Random Forest algorithms should be considered in clinical applications and specific cases. The selection of models should depend on whether clinicians prioritize detecting more positive cases or limiting false positives.

Clinical patient evaluation indicators such as Acute Physiology and Chronic Health Evaluation (APACHE II), Simplified Acute Physiology Score (SAP) 3, or indicators of catheter transmission use, surgical, or comorbidities,^4,24 were not used in our study. However, the performance of predictive models was still high with dataset 2, which contained complete blood tests and biochemical tests results. This result suggests other hospitals to determine the most appropriate model and indicators for their facilities. In addition, several potentially important predictors for resistance, such as residency location,²⁶ antibiotic use outside the hospital,^27–29 microbiome composition, diet, and exercise,^30–33 which were not included in the present study, should be consider in future studies.

Limitations

Although the study obtained high-performing machine learning models, some limitations must be addressed. The first limitation was related to the data availability and the period when data were selected. In this study, there was only one hospital with enough data on blood tests and biochemical tests. The extrapolation to other hospitals or other departments should be done with caution. Because the data were collected during the COVID-19 pandemic in Vietnam (from 2020 to 2022), they could be analyzed and interpreted with bias when extrapolated to other periods or hospitals. Furthermore, this study did not distinguish bacterial infections based on the time they occurred after admission, so it was impossible to determine whether the infections were nosocomial. The second limitation was that no algorithm was better than another, and every algorithm had its trade-offs. The results from machine learning will not be optimal and effective without the coordination among clinicians, hematology and biology technicians, and information technology staff. The interdisciplinary collaboration will ensure that professional requirements are evaluated and fully reflected in the choice of algorithms. Lastly, many recommended features in the previous study were not considered, which may influence current output values if included. Machine learning is driven by data; therefore, it is crucial to be cautious and prepare sufficient data for different objects. Data cannot be arbitrarily applied from one department to another, from one hospital to another, or from one group of patients to another without considering scientific evidence and comprehensive protocols.

Conclusion

In the present study, we present machine learning models to predict antibiotic resistance of bacterial infections of ICUs patients using the patients’ EMRs. The best-performed machine learning models were achieved using XGBoost, LightGBM, and Random Forest, and the dataset included complete blood count and biochemical test results (accuracy, F1-score, AUROC, normMCC ≥ 0.90). The performance of machine learning models varied depending on the data set, selected variables, and algorithms.