Optimized Machine Learning Models to Predict In-Hospital Mortality for Patients with ST-Segment Elevation Myocardial Infarction

Introduction

As a well-known dangerous disease, acute myocardial infarction (AMI) is the leading cause of global morbidity and mortality.^1–4 AMI is traditionally classified as ST-segment (STEMI) or non-ST-segment elevation myocardial infarction (NSTEMI) based on electrocardiography (ECG) characteristics.⁵ Compared with NSTEMI patients, STEMI patients suffer from a higher risk of short-term mortality.⁶ Clinical guidelines for the treatment of STEMI patients recommend an invasive strategy with cardiac catheterization, revascularization when clinically appropriate, and thrombolysis.⁷ Especially for high-risk patients, more-aggressive interventional and pharmacologic therapy should be targeted. However, physicians tend to adopt non-invasive treatment strategies rather than aggressive treatment strategies in clinical practice (the “treatment-risk paradox”).⁸ To alleviate this problem, risk assessment is applied to clinical practice as an effective method. Clinically, the most commonly used risk assessment tools are the Thrombolysis in Myocardial Infarction (TIMI) risk score and the Global Registry of Acute Coronary Events (GRACE) risk score.^9–11 However, the selected population of the GRACE risk score includes STEMI and NSTEMI patients. Therefore, it is not a model targeting at STEMI patients.¹⁰ Although the selected population for the TIMI risk score is STEMI patients, thrombolytic therapy is the main treatment strategy for the enrolled patients,⁹ while percutaneous coronary intervention (PCI) is the main treatment strategy for STEMI patients in China.¹² Therefore, the TIMI risk score might be inadequate for current clinical practice. Furthermore, the TIMI risk score requires the history of diabetes, hypertension or angina.⁹ However, among hypertensive patients, only 42.6% are aware of their high blood pressure.¹³ In summary, existing tools are far from perfect. It is necessary to develop new risk prediction models using a dataset that can represent the real world.

The blossom of artificial intelligence (AI), such as machine learning, has provided a possible solution to this unmet need. In recent years, many ML models have been successfully applied to cardiovascular field.¹⁴ In Al-Zaiti et al,¹⁵ an ML model using 554 temporal-spatial features of the 12-lead ECG to predict underlying acute myocardial ischemia in patients with chest pain has been developed. It achieves 76% specificity and 77% sensitivity, exceeding commercial interpretation software (25%) and experienced clinicians (40%).¹⁵ In Ragunath et al,¹⁶ a deep neural network (DNN) has been developed to predict 1-year all-cause mortality by using ECG voltage–time traces. It achieves an area under the curve (AUC) of 0.88 on a held-out test set, which includes 168,914 patients and 14,207 events occurred.¹⁶ However, ECG cannot fully reflect the clinical situation of patients, a number of models based on laboratory and clinical features were developed. In Kwon et al,¹⁷ the performance of six typical ML models in predicting the 1-year mortality rate of anterior ST-segment elevation myocardial infarction patients was compared.² It shows that the XGBoost performs best (92% accuracy, 99% specificity, 0.74 f1 score and 0.942 AUC); however, the sensitivity is only 60%. Another research developed a deep-learning-based risk stratification for the mortality of patients with AMI (DAMI).¹⁷ It achieved an AUC of 0.905 in predicting the in-hospital mortality of STEMI patients. However, the key performance metric-sensitivity that reflects the model’s capability to identify high-risk patients is not reported. Furthermore, researches have shown that the sensitivity is not high enough for some models with high AUC.¹⁸

To the best of our knowledge, existing models were trained with imbalanced data, that is there are far more negative samples (ie, survival patients) than positive samples (ie, dead patients), and the ratio is greater than 10:1. When trained with imbalanced data, ML models tend to predict the samples under test as the majority class.¹⁹ Consequently, many high-risk patients may be missed and cause disaster results. To mitigate the effect of data imbalance, two strategies are commonly used: cost-sensitive learning and data preprocessing. The latter is preferred as it is easy to implement. Among data preprocessing, the most versatile and effective method is random under-sampling (RUS), which involves the random elimination of majority class examples.²⁰ In this study, we applied RUS to balance classes in the training data to improve the performance of models. Besides, existing models were trained either with full set of features containing all variables during hospitalization or simplified set of features selected by feature importance ranking. However, some high-ranking features may not be obtained prior to reperfusion therapy, which means the existing models were not able to make predictions at the time of initial evaluation and treatment. In this work, a simplified set of features which only included variables prior to reperfusion therapy was proposed, and models using these variables to predict high-risk patients were explored.

The aims of this work lied in 1) to propose a dataset that can represent the real-world STEMI patients, 2) to apply RUS to optimize prediction models, 3) to develop models that can predict high-risk patients at the time of initial evaluation and treatment.

Materials and Methods

This study was conducted with a hospital-based dataset. We consecutively enrolled 5708 STEMI patients in Tianjin Chest Hospital from January 2015 to April 2020. The enrollment criteria for patients are as follows: 1) persistent chest discomfort or other symptoms suggestive of ischemia; 2) ST-segment elevation (≥0.1 mV) in at least two contiguous leads; 3) an elevation of cardiac troponin values with at least one value above the 99th percentile upper reference limit.⁷ To be noted, some of the enrollment patients complicated with atrial fibrillation whose pulse rate was less than heart rate. The categories of patient data collection included demographics, presentation characteristics, admission pathway, treatment and hospital outcomes.

Since the cardiac function of STEMI patients often changes in short term, we selected two indicators to evaluate the cardiac function of patients. One was direct Killip classification, which was evaluated immediately after entering the emergency room, and the other was heart failure, which included Killip classification II/III/IV and was evaluated again before reperfusion therapy.

Categorical variables were reported as counts and percentages. Chi-squared test was used to evaluate the differences in categorical variables between groups. For continuous variables, the Kolmogorov–Smirnov test was used to test the normality of distribution. We utilized the mean ± standard deviation (SD) to present the normally distributed variables, while the median and interquartile ranges (IQR) to present the variables whose distributions were not normal. Mann–Whitney-U test was used to evaluate the differences in continuous variables between groups. All the tests were two-tailed.

In this study, the entire data were divided into survival groups and death groups according to patients’ hospital outcomes. Both groups were randomly split into a training set (75%) and a testing set (25%). The models were optimized with the training set only, using 10-fold Cross-Validation. In this process, the training set was divided into 10 parts, and each of these parts was used to train models and evaluate their performance. The performance of the final models was evaluated on the testing set, which was not used for training process at all. Besides, we considered two sets of features in chronological order: a full set and a simplified set. The full set contained all variables during hospitalization, while the simplified set contained only included variables prior to reperfusion therapy. The variables used in both sets are shown in Table 1. Four commonly used ML models (logistic regression (LR), support vector machine (SVM), decision tree (DT), random forest (RF)) were selected to predict the mortality of STEMI patients, while RUS was applied as data preprocessing to optimize the models. Furthermore, to explore the possibility of developing models that could identify the high-risk patients at the time of initial evaluation and treatment, the performance of the predictive models trained using simplified set is evaluated.

Table 1 Simplified Set and Full Set of Features

The procedure for RUS was described as follows. First, the training set was split into majority class and minority class. Second, a number of samples in the majority class were removed randomly. The reduced majority class was then combined with the minority class, resulting in a balanced training set. Finally, the models were trained with the balanced training set.

The LR is a linear model that assumes a Bernoulli distribution of the outcome and a log-linear relationship with the predictors.²¹ LR predicts the binary response probability for the outcome class given the predictor values. In contrast with the three subsequent algorithms, LR lacks tuning parameters.

The SVM is a typical kernel-based supervised learning algorithm, and it aims to create a hyperplane among data points to maximize the classification interval. The kernel method enables us to model higher dimensional, non-linear models. In a non-linear problem, a kernel function can be used to add additional dimensions to the raw data and thus make it a linear problem in the resulting higher dimensional space. However, the choice of kernel function could greatly affect the performance of the SVM model.²² In this work, the optimal kernel function was selected by 10-fold cross-validation.

The DT is a supervised algorithm based on tree structure where each node indicates a feature, each branch represents decision rule and each leaf depicts an outcome. DT uses a divide-and-conquer algorithm to split a node into two sub-nodes, which is done with respect to outcome. On each iteration of the algorithm, it iterates the unused feature of the dataset and calculates entropy and information gain (IG) of this feature. The feature with highest IG forms the root node, which is partitioned into other sub-nodes and tested with another feature recursively until the leaf (outcome) is reached.²³ However, a decision tree with full depth is vulnerable to overfitting. In this work, the optimal depth was determined by 10-fold cross-validation.

The RF is a supervised algorithm that constructs an ensemble of decision trees. Specifically, the RF combines bootstrap sampling of data for constructing each tree, and random selection of variables at each decision node, and finally majority voting on outcomes. The RF feature importance scores are determined by the Gini importance index. Specifically, the variable chosen as root split for many trees gets a higher Gini importance than a variable chosen less frequently or for descendant nodes.²¹ To be noted, the number of decision trees could greatly affect the performance of the RF model. In this work, the optimal number was determined by 10-fold cross-validation.

All descriptive statistics were performed using the SPASS software. The ML model selection, fitting and optimization were performed using Python (version 3.7) with the scientific libraries “scikit-learn”.

Four performance metrics: sensitivity, specificity, G-mean and AUC are used in this study. Sensitivity refers to the models’ ability to correctly identify high-risk patients, defined as

(1)

where TP is the number of true positives and FN is the number of false negatives.

Specificity reflects the models’ ability to correctly reject low-risk patients, defined as:

(2)

where TN is the number of true negatives and FP is the number of false positives.

The area under the curve (AUC) of the receiver operating characteristic curve (ROC) is used to evaluate the overall performance. However, some researches have shown that AUC could lead to an overoptimistic estimation of the algorithm’s performance in the case of highly skewed datasets.¹⁸ Therefore, G-mean is employed as performance metrics, defined as

(3)

Its key idea is to maximize the recognition of each of minority and majority classes while keeping these accuracies balanced.

Results

From January 2015 to April 2020, a total 5708 patients were included in our study, including 4198 men (73.55%) and 1510 women (26.45%). The median age of them was 63 years. In our study, 87.95% of them received reperfusion treatment and the main reperfusion strategy was primary PCI (77.73%). The overall in-hospital mortality rate was 2.70% (n = 154). The basic descriptive statistics and the differences between patients died and survived are summarized in Tables 2 and 3. The nonsurvivors were more likely to have lower systolic (SBP) and diastolic blood pressure (DBP), higher pulse, higher heart rate (HR), higher cardiac troponin I (cTnI), higher Killip classification and higher age. Compared with survivors, the nonsurvivors were mainly admitted through emergency medical services (EMS) or transferred from other hospitals, while had a lower proportion of reperfusion treatment. Besides, nonsurvivors experienced heart failure and unconscious more often, and the mortality was higher in women compared with men.

Table 2 Basic Characteristics of Continuous Numerical Variables

Table 3 Basic Characteristics of Categorical Variables

The performance of the traditional and optimized ML models trained using the full set is shown in Table 4. Although the models trained without RUS had higher accuracy, 96.99% vs 77.35% on average, and higher specificity, 99.37% vs 77.16% on average, the sensitivity of them was extremely lower than models trained with RUS, 9.87% vs 84.21% on average. The AUC of the models trained with or without RUS did not have significant difference, 0.864 vs 0.841 on average and the ROC analysis results are presented in Figures 1 and 2. The G-mean of models trained with RUS was far higher than those without, 80.54% VS 23.31% on average. The SVM trained with RUS achieved the best performance in terms of G-mean (84.93%) and AUC (0.919). Specifically, it achieved 85.62% accuracy, 84.21% sensitivity, 85.66% specificity, and the confusion matrix is shown in Figure 3. It selected 32 out of 38 high-risk patients. Among the models trained without RUS, the SVM classifier also achieved best performance according to G-mean (45.52%). Specifically, it achieved 96.42% accuracy, 23.68% sensitivity, 98.41% specificity, and the confusion matrix is shown in Figure 4. It selected 9 out of 38 high-risk patients. The SVM trained with RUS selected 23 more high-risk patients than the SVM trained without RUS.

Table 4 Performance Comparison of the Traditional and Optimized ML Models Using the Full Set

Figure 1 ROC analysis results of models trained without RUS using the full set. Abbreviation: ROC curve, receiver operating characteristic curve.

Figure 2 ROC analysis results of models trained with RUS using the full set. Abbreviation: ROC curve, receiver operating characteristic curve.

Figure 3 The confusion matrix of the SVM trained with RUS using the full set. Abbreviations: SVM, support vector machine; RUS, random under-sampling.

Figure 4 The confusion matrix of the SVM trained without RUS using the full set. Abbreviations: SVM, support vector machine; RUS, random under-sampling.

The performance of the traditional and optimized ML models trained using the simplified set is shown in Table 5. The results were similar to those with the full set, the models trained with RUS had lower accuracy, 81.72% vs 96.79% on average, lower specificity, 82.61% vs 99.06% on average, higher sensitivity, 84.21% vs 9.87% on average, higher G-mean, 75.72% vs 35.76% on average and similar AUC, 0.753 vs 0.761 on average. As can be seen from Tables 4 and 5, the SVM trained with RUS using the simplified set achieved 83.48% G-mean, which was comparable to the models trained using the full set. For sensitivity, the SVM trained using the simplified set achieved 89.47%, which even exceeded the SVM (84.21%), DT (81.58%) and RF (81.58%) trained using the full set. The confusion matrix of the SVM trained with RUS using the simplified set is shown in Figure 5. It selected 34 out of 38 high-risk patients, which is 2 more than the SVM trained with RUS using the full set. Besides, the feature importance scores for predicting the in-hospital mortality were provided by the RF trained with RUS, as shown in Figure 6, where the cTnI had the highest importance score.

Table 5 Performance Comparison of the Traditional and Optimized ML Models Using the Simplified Set

Figure 5 The confusion matrix of the SVM trained with RUS using the simplified set. Abbreviations: SVM, support vector machine; RUS, random under-sampling.

Figure 6 The feature importance scores for predicting the in-hospital mortality provided by the RF. Abbreviation: RF, random forest.

Discussion

Although the mortality of the STEMI patients is high, the risk assessment tools widely used in clinical practice are far from perfect. With the thriving of ML, there were many researches that developed risk assessment models based on ML and the performance of their models is shown in Table 6.^24–28 However, these models only focused on accuracy or AUC without reporting sensitivity and none of these researches dealt with the imbalanced data problem. When facing imbalanced scenarios, the models tended to predict the samples as majority class (survival patients) in order to maximize the accuracy, and the AUC would lead to an overoptimistic estimation of the models. Therefore, although these models had high accuracy or AUC, they might have low sensitivity. The sensitivity of prediction models is highly related to the prognosis of the patients. The models with low sensitivity tend to ignore the high-risk patients who require special attention, which would lead to an increased risk of death. For these reasons, we applied RUS to mitigate the effect of samples imbalance and used G-mean as another evaluation metrics.

Table 6 Best Predictive Performance Results of the Models Developed by Other Articles

The results showed that the overall performance of the models trained with RUS had been greatly improved. Whether in the full set or the simplified set, although the specificity of the models was slightly decreased, the sensitivity was greatly improved, which means more high-risk patients could be found. It is more important in clinical practice. However, there is little difference in the AUC of the models trained with or without RUS. It is obvious that AUC cannot reflect the performance of the models objectively. G-mean which takes both specificity and sensitivity into account can comprehensively evaluate the performance of models. The G-mean of the models trained with RUS was significantly higher than that of those without. This result further proved the effectiveness of RUS.

From a clinical perspective, the models trained using the full set of features have high sensitivity, they can identify high-risk patients accurately. In this way, those high-risk patients can be paid more attention after being escorted to the cardiac intensive care unit, which has an positive effect on reducing in-hospital mortality. Although the overall performance of the models trained using the features obtained prior to reperfusion therapy has been weakened compared with the models trained using the full set, the sensitivity is high enough to support the decision-making. According to the 2017 ESC Guidelines, the aggressive treatment strategies were recommended for high-risk patients. However, in clinical practice, the physicians and the relatives of patients tend to choose the non-invasive treatment due to overestimating the risk of reperfusion therapy and ignoring the risk of death from ischemia. It hinders delivery of effective care to the patients who need it most, leading to the mortality of a large number of high-risk patients (the “treatment-risk paradox”). The proposed models can identify high-risk patients at the time of initial evaluation, which allows the patients to receive reperfusion therapy in the early stage of myocardial infarction, decreasing the risk of therapy. Therefore, the proposed models can support the decision-making about intensive treatment for high-risk patients, which can improve treatment compliance.

The cTnI level was the most significant risk factor given by RF, and many studies^29–31 have also reported that the level of cTnI has a high predictive value of the AMI patients’ outcome, which further demonstrates the reliability of our models. We believe that ML might better handle the correlation between the variables and RF is a particular expert in creating conditional relations. Moreover, our models were trained with a dataset that can reflect real-world STEMI patients and could give predictive results without medical history. Therefore, our models avoided collecting medical history, resulting in the improvement of usability.

However, our work still has the following two limitations: 1) Our work is a single-center study, only including the patients who were treated in Tianjin Chest Hospital. 2) It is inevitable that the patient enrollment and the data measurement are accompanied by bias. Nonetheless, the result of our study still provides an effective and robust method for predicting in-hospital mortality of STEMI patients.

Conclusion

Many medical problems are characterized by a highly imbalanced distribution of examples in classes. In this paper, we applied RUS to alleviate the effect of data imbalanced, which improved the performance of models. In the management of STEMI patients, the models trained using the full set of features can be applied to select high-risk patients after being escorted to the cardiac intensive care unit, and the models trained using the simplified set of features can be applied to assist physicians in choosing the optimal treatment. In summary, this study demonstrated the feasibility and effectiveness of the model with RUS, which can improve prognosis of patients.