Machine Learning-Based Prediction of Subsequent Vascular Events After 6 Months in Chinese Patients with Minor Ischemic Stroke

Introduction

Among stroke patients in China, patients with minor ischemic stroke (MIS) account for as much as 30%,¹ and these patients generally receive less attention and active treatment due to their mild symptoms. Studies have shown that patients with MIS have a high risk of subsequent vascular events such as recurrent stroke. Stroke recurrence rate within the first 90 days after stroke is 10.4%, and 12.8% to 18.9% at one year.^2,3 Active treatment and secondary prevention can effectively reduce the risk of stroke recurrence and subsequent vascular events.^4,5 Therefore, it is important to identify patients at risk of poor prognosis after MIS in the early stages of clinical management. Since the degree of MIS is relatively mild, and the symptoms are not obvious, the commonly used models for predicting ischemic stroke may not be sufficient to predict the prognosis of patients with MIS.⁶ Thus, developing a separate predictive model for MIS patients is necessary.

Machine learning (ML) is a kind of artificial intelligence that can provide accurate and fast prediction results and has been widely used in the medical field.⁷ ML uses computer algorithms to build models from labeled data and make data-driven predictions. It can handle the complex nonlinear relationship between variables and results. These advantages improve the accuracy of the prediction model. So far, ML technology has been used in the studies of multiple cerebrovascular diseases.^8–10 George et al propose an externally validated machine-learning-derived model which includes readily available parameters and can be used for the estimation of cardiovascular risk in ischemic stroke patients.¹¹ Xie et al. Integrating common stroke biomarkers and predicting patient recovery outcomes at 90 days using a machine learning approach and finding that a decision tree-based gradient boosting machine (GBM) can predict recovery outcomes for stroke patients on admission.¹² Carlos et al used machine learning techniques to identify the main variables of a machine-learned random forest (RF) and generated a predictive model to predict mortality and morbidity 3 months after admission. Results show that machine learning algorithm RF can be effectively used for long-term outcome prediction of mortality and morbidity in stroke patients.¹³ ML includes a variety of algorithms, mainly including supervised learning and unsupervised learning, among which supervised learning algorithms are usually used for predictive analysis, including diagnosis and prediction results.¹⁴ Common supervised learning algorithms include logistic regression (LR), linear regression, support vector machine (SVM), decision tree, ensemble method, k-nearest neighbor, and naive bayes classifier, of which LR and linear regression are the most commonly used two predictive models.¹⁵ Previous studies on stroke have shown that SVM can be used as the best model for prediction.¹⁶ Can SVM and LR predict minor strokes? Which model is more advantageous?

In this study, we used two representative machine learning algorithms, LR and SVM, to establish and validate the prediction model of subsequent vascular events after 6 months following MIS in Chinese patients, and select the best prediction model.

Materials and Methods

Material

In this retrospective cohort study, we initially screened 750 patients who visited the Neurology Clinic of the Second People’s Hospital of Kunshan from January 2017 to December 2020. The study was conducted following the principles of the Declaration of Helsinki and was approved by the institutional ethics committee of the Second People’s Hospital of Kunshan for retrospective analysis. The data are all anonymous, and the requirement for informed consent was waived. The entry criteria are as follows: (1) The initial visit showed mild brain symptoms, with or without slight positive signs of stroke, and the NIHSS score ≤ 3 points; (2) The initial visit and the brain has been examined by magnetic resonance imaging (MRI). MRI showed small infarction, or the use of diffusion-weighted imaging (DWI) or fluid-attenuated inversion recovery (FLAIR) to find lacunar lesions or enhanced brain signals, and was located in the subcortical white matter, basal ganglia, or brainstem; (3) Imaging examination to exclude patients with intracranial hemorrhage or symptoms of non-vascular etiology; (4) Stroke onset time was less than 72 hours; (5) The patient was above 18 years old; (6) Patients with complete height, weight, waist circumference, blood pressure, blood lipids, fasting blood glucose data, and complete past medical history data; (7) All patients received aspirin treatment (100 mg daily) after consultation. (8) Patients can accept follow-up evaluation at 6 months. The exclusion criteria were as follows: (1) Clear history of ischemic stroke; (2) A history of cerebral hemorrhage or other active hemorrhagic diseases, a history of brain tumors, brain trauma, or other brain injuries; (3) Dementia or mental illness; (4) Lack of MRI examination; (5) NIHSS score>3 at the first visit. The research flow is shown in Figure 1.

Figure 1 Flowchart of patient selection.

Data Collection

The following data were collected: basic data of patients (age, height, weight, waist circumference, SBP, DBP, FBG level, blood lipid level), past medical history (hypertension, diabetes, dyslipidemia, metabolic syndrome), years of hypertension, initial NIHSS score, symptoms [headache, dizziness, mild cognitive impairment (MCI)], obstruction sites of MIS [anterior circulation artery (ACA), posterior cerebral artery (PCA)], MRI results (number of lesions, the maximum diameters of lesions, size of infarcts). The height, weight, and waist circumference of patients were measured by standard methods. Fasting blood glucose and blood lipids were measured in the laboratory. The NIHSS score was evaluated by a neurologist. All patients underwent brain MRI within 24 hours of their first visit. MRI is performed using 1.5T equipment (Siemens). All MRIs were diagnosed by the same neuroradiologist and the same neurologist, who was not informed of the study. The examiner used FLAIR or DWI to assess whether there are high-intensity lesions specifically. The maximum diameter of each patient’s lesion and the number, location, and lesion area are recorded in detail.

The patients were followed up 6 months later to assess the occurrence of SVE. A neurologist collected the information over the phone. For patients who died during follow-up, the data was obtained from their relatives or hospital records.

Diagnostic Criteria

Hypertension: patients diagnosed with hypertension with clear medical records, or the blood pressure measured in two quiet states was greater than or equal to 140/90mmHg.

Diabetes: patients with a history of diabetes confirmed by medical records or had symptoms of diabetes with fasting blood glucose ≥7.0mmol/l or random blood glucose ≥11.1mmol/l.

Dyslipidemia: total plasma cholesterol level>5.2mmol/l, or low-density lipoprotein cholesterol>3.4mmol/l, or triglyceride level>1.7mmol/l, or high-density lipoprotein cholesterol (HDL-C) <1.03mmol/l for male, Or HDL-C<1.3mmol/l for female.

Metabolic syndrome: The subjects can be diagnosed if they meet three of the following five items: (1) male waist circumference ≥90cm, female waist circumference ≥80cm; (2) triglyceride >1.7mmol/l; (3) male HDL-C <1.03mmol/l or female HDL-C<1.3mmol/l; (4) systolic blood pressure ≥130 mmHg, diastolic blood pressure ≥80 mmHg; (5) fasting blood glucose ≥5.6 mmol/l.

SVE: Transient ischemic attack (TIA), transient non-focal cerebrovascular symptoms, worsening cerebrovascular symptoms or recurrence of infarction, and modified Rankin scale score ≥2 during follow-up.

Laboratory Examination

Triacylglycerol (TG, reference range: <1.70mmol/L), low-density lipoprotein (LDL, reference range: <3.36mmol/L), high-density lipoprotein (HDL, reference range: male, 0.78–1.81mmol/L; female, 0.78–2.20mmol/L), fasting blood glucose (FBG, reference range: 3.9–6.0mmol/L). Biochemical blood lipids were measured using BECKMAN COULTER AU5800 (Brea, CA, USA).

Statistical Analysis

The R software (version 3.4.3; http://www.R-project.org/; software package: glmnet, pROC, rms, dca.R) was used to analyze the data. Continuous variables are expressed as mean ± standard deviation (SD) when they follow a normal distribution and are expressed as P50 (P25, P75) when non-normally distributed. Categorical variables are expressed in frequency (%). Unpaired Student-t test or Mann–Whitney nonparametric test was used for continuous variables in comparison between groups, and Pearson chi-square test or Fisher’s exact test was used for categorical variables.

Our feature selection and classification analysis were performed using the internally developed Python framework in the open-source Python library Scikit-Learn. The data set was divided into training set and testing set with a ratio of 7:3 by using stratified random sampling. The recursive feature elimination (RFECV) method was used on the standardized training set data, and cross-validation was used to retain the features with the best performance (accuracy). The different feature combinations were cross-validated based on RFE, and the learner remains unchanged. By calculating the sum of its decision coefficients, the importance of different features to the score was finally obtained, and then the best feature combination was retained.

The machine learning models, including LR and SVM classifiers, were trained using the filtered predictive features. Five-fold stratified cross-validation from scikit-learn was used to determine the model parameters in the training set. Stratified K-Fold is a variant of k-fold cross-validation, ensuring that each set contains approximately the same percentage of samples of each target class as the entire training set.

The predictive model’s receiver operating characteristic (ROC) curve was drawn, and the area under the curve (AUC) and its 95% CI were obtained. The confusion matrix was provided, and the sensitivity, specificity, accuracy, positive predictive value (PPV), and negative predictive value (NPV) were calculated to measure the model’s performance. The AUC of the models was compared in pairs using the method proposed by DeLong et al. A calibration curve was drawn to compare the prediction accuracy of the models, and decision curve analysis (DCA) was used to quantify and compare the clinical effectiveness of different models. By subtracting the proportion of false positives from the proportion of true positives, and then weighing the relative harm of false positives and false negatives, we could get the quantification for net benefit. The formula is as follows:

Where n is the total number of patients in the study, and Pt is the given threshold probability. All statistical tests were two-sided, and P<0.05 was considered statistically different.

Results

Description of the Study Population

Four hundred ninety-five patients were finally enrolled, including 258 males and 237 females, with an average age of 58.2 ± 10.3 years (range 37–85 years). SVE occurred in 56 cases (11.3%) during the 6-month follow-up. The comparison of baseline data between the two groups was shown in Table 1. The age, fasting blood glucose, diabetes, hyperlipidemia, metabolic syndrome, years of hypertension, NIHSS score, mRS score, anterior circulation artery lesions, posterior cerebral artery lesions, maximum diameter of the lesions, MRI infarct area, abdominal obesity, and the elevated TG between two groups were significantly different (all P<0.001).

Table 1 Baseline Characteristics of SVE in Patients with Mild Cerebral Infarction After 6 Months

Screening of the Best Model Predictors

Seventeen possible features were selected with P=0.1 as the cutoff value, including age, BMI, systolic blood pressure, fasting blood glucose, diabetes, hyperlipidemia, metabolic syndrome, years of hypertension, NIHSS score, mRS score, number of MR lesions, maximum lesion diameter mm, MRI infarct area, anterior circulation artery, posterior cerebral artery, abdominal obesity, and elevated TG. According to the ratio of 7:3, the dataset was divided into a training set (307 negative cases and 39 positive cases) and a testing set (132 negative cases and 17 positive cases) using stratified random sampling. There was no significant difference for 17 features between the training and testing sets (all P>0.05, Table S1).

RFECV was further used in the training set to filter the feature combinations from the standardized data. The learning curve of specific parameters and feature numbers were shown in Figure S1. The best 15 predictive features were finally selected: age (X3), BMI (X4), systolic blood pressure (X5), fasting blood glucose (X7), diabetes (X9), metabolic syndrome (X11), years of hypertension (X12), NIHSS score (X13), mRS score (X14), maximum lesion diameter (X19), MRI infarct area (X20), anterior circulation artery (X21), posterior cerebral artery (X22), abdominal obesity (X23), elevated TG (X24). The weight map of 15 predictive features was drawn (Figure 2), and the top three predictive features were diabetes (X9), posterior cerebral artery (X22), and fasting blood glucose (X7) in order. The Spearman correlation matrix heat map of the selected 15 features was drawn (Figure 3), and it was found that there were no redundant features (|r|>0.85).

Figure 2 The weight map of the features (after feature standardized). Note: Age (X3), BMI (X4), systolic blood pressure (X5), fasting blood glucose (X7), diabetes (X9), metabolic syndrome (X11), years of hypertension (X12), NIHSS score (X13), mRS score (X14), maximum lesion diameter (X19), MRI infarct area (X20), anterior circulation artery (X21), posterior cerebral artery (X22), abdominal obesity (X23), elevated TG (X24).

Figure 3 The correlation matrix heat map of the selected 15 features. Note: Age (X3), BMI (X4), systolic blood pressure (X5), fasting blood glucose (X7), diabetes (X9), metabolic syndrome (X11), years of hypertension (X12), NIHSS score (X13), mRS score (X14), maximum lesion diameter mm (X19), MRI infarct area (X20), anterior circulation artery (X21), posterior cerebral artery (X22), abdominal obesity (X23), elevated TG (X24).

Construction of Predictive Model

LR Modeling

In order to improve the robustness of the model, a 5-fold cross-validation was performed on the training set, and the average AUC was 0.975 ± 0.024, and its accuracy, sensitivity, specificity, PPV, and NPV were 0.908 ± 0.033, 0.925 ± 0.168, 0.905 ± 0.029, 0.558 ± 0.101, and 0.990 ± 0.023, respectively (Figure 4A). In testing set, AUC was 0.929 (95% CI: 0.875–0.964) (Figure 4B), its accuracy, sensitivity, specificity, PPV, NPV were 0.832, 0.765, 0.841, 0.382, and 0.965, respectively. The confusion matrix of the LR model was shown in Table 2. The calibration curve of the LR model in training set and testing set (Figure 4C and D) was drawn, and it was found that this model overestimated the incidence of SVE in both data sets.

Table 2 Confusion Matrix of LR Model and SVM Model

Figure 4 (A) ROC curve in 5-fold cross-validation for LR model in training set, (B) ROC curve of LR model in testing set, (C) The calibration curve of LR model in training set, (D) The calibration curve of LR model in testing set.

SVM Modeling

In order to improve the robustness of the model, 5-fold cross-validation was performed for the training set, the average AUC was 0.999 ± 0.002, and its accuracy, sensitivity, specificity, PPV and NPV were 0.991 ± 0.013, 0.975 ± 0.056, 0.994 ± 0.009, 0.953 ± 0.065, 0.997 ± 0.007, respectively (Figure 5A). The AUC in testing set model was 0.992 (95% CI: 0.962–1.000) (Figure 5B), and its accuracy, sensitivity, specificity, PPV, and NPV were 0.966, 0.824, 0.985, 0.875, 0.977, respectively. The confusion matrix of the SVM model was shown in Table 2. The calibration curve of the SVM model in training set and testing set was drawn (Figure 5C and D). As was shown, there was good consistency between the predicted probability and the actual probability in testing set.

Figure 5 (A) SVM’s ROC curve of 5-fold cross-validation in training set. (B) ROC curve of SVM model in testing set. (C) The calibration curve of SVM model in the training set. (D) The calibration curve of SVM model in testing set.

Comparison of ROC Curves and DCA Between LR Model and SVM Model in Testing Set

The ROC curves of the LR model and the SVM model were compared in testing set (Figure 6A), and the DeLong test showed that the AUC of the SVM model was significantly higher than that of the LR model (0.992 vs 0.929, Z=2.858, P=0.004). In most cases, the SVM model’s net benefit was better than that of the LR model based on DCA (Figure 6B).

Figure 6 (A) ROC curves of LR model and SVM model in testing set. (B) DCA curves of LR model and SVM model in testing set.

Discussion

In this study, two machine learning algorithms, LR and SVM, were used to develop and validate the prediction model of SVE 6-month later after MIS in Chinese patients. Both models showed high AUC values (both>0.9). However, the SVM model was better than the LR model regarding discrimination, calibration, and clinical validity. Our results showed that the SVM model had a good predictive capability, which will be clinically helpful in identifying the risk of SVE 6 months later after MIS.

Among the 15 best predictive features, finally selected diabetes, posterior cerebral artery disease, and fasting blood glucose were the three strongest predictors. Diabetes is a well-known vascular risk factor. It is known that chronic hyperglycemia can lead to vascular endothelial dysfunction, early arterial stiffness increase, systemic inflammation, and thickening of the capillary basement membrane. These different mechanisms can affect the structure and function of blood vessels and regulate immunity, thereby increasing the risk of cardiovascular disease.^17,18 Blood glucose control is extremely important for secondary stroke prevention. Studies have found that every 1% increase in hemoglobin A1c can increase the chance of death after stroke by approximately 33%.¹⁹ Compared to the chronic damage caused by diabetes to blood vessels, the damage to blood vessels caused by acute hyperglycemia cannot be underestimated. Many studies have shown that poor fasting blood glucose control is an important predictor of increased risk of adverse outcomes in acute ischemic stroke,^20–22 which was consistent with this study that high fasting blood glucose after stroke was associated with poor prognosis. At present, the mechanism of the association between hyperglycemia and poor clinical prognosis of stroke patients is still unclear. Some researchers considered that the presence of hyperglycemia during cerebral ischemia would promote the production of lactic acid and aggravate tissue acidosis.^23,24 Some scholars believe that acute hyperglycemia-mediated impairment of the brain autoregulation may be the key compensatory mechanism leading to the progression of ischemic penumbral tissue to infarcted tissue,^25,26 and its specific mechanism needs to be further studied.

Current clinical scoring assessments used to predict recurrence after cerebral ischemia have limited efficacy.²⁷ Among them, the Stroke Prognostic Indicators (SPI-II) and the Essen Stroke Risk Score (ESRS) were used to predict the long-term (up to 2 years) recurrence risk after ischemic stroke.^28,29 SPI-II can be used in patients with transient ischemic attack (TIA) and minor stroke. However, studies have shown that the performance of the SPI-II score mainly depends on its ability to predict mortality rather than recurrence, and there is no better predictor for the high recurrence rate of minor ischemic stroke.^30,31 The emergence of ML has changed this situation, although, ML technology has been used in the study of multiple cerebrovascular diseases, there are few studies on the prediction model of SVE risk in MIS patients. Compared to the machine learning prediction model developed by Du³² and others, we found that in addition to age, BMI, blood sugar, metabolic syndrome, infarct size and number, and other common risk factors, the location of the lesion played an important role in the occurrence of SVE, and posterior cerebral artery lesions showed a higher incidence of SVE compared to anterior circulation lesions. Since only 5% to 10% of patients with TIA or minor stroke have ischemic sites located in the area of PCA,³³ and the number is small, so its clinical features and causes have not been studied as extensively as other vascular areas. Through verification, we found that posterior cerebral artery lesion was an important predictor of SVE risk in MIS patients. Compared with the study of Du et al,³² our study included more patients in the group. In addition to the LR algorithm, we also developed the SVM algorithm to build a predictive model. Our results showed that both algorithms had high prediction efficiency, but the prediction accuracy and clinical effectiveness of SVM were better.

This study has the following limitations. The first is the inherent limitations of a retrospective study. During the follow-up period, we cannot monitor the medication and dietary status of the patients. These unknown confounding may have a certain impact on the experimental results. Secondly, our model is a single-center study with a limited sample size and no external validation, the results need to be further validated externally by multi-center, large-scale studies. The application of machine learning in the medical field is expanding, since machine learning often requires a huge number of datasets, the types of which may not be readily available, in the future we may need a collaborative approach across multiple institutions to build robust datasets and develop more models to improve future ML-based stroke projects.

Conclusions

In summary, we used two machine learning algorithms, LR and SVM, to build and validate a prediction model that predicts the SVE incidence 6 months after MIS in Chinese patients. SVM showed high accuracy and applicability, and it can be used to predict the SVE risk after 6 months following MIS in Chinese patients.