Deep Learning in Prediction of Late Major Bleeding After Transcatheter Aortic Valve Replacement

Introduction

Transcatheter aortic valve replacement (TAVR) has emerged as an alternative treatment option for severe aortic stenosis patients in the last decade.^1,2 However, bleeding is still the main post-procedure complication that largely affects patients’ short- and long-term prognoses.^1,3,4 Early bleedings that occur immediately after TAVR were mainly related to procedural factors, and the rate of this type of bleeding had gradually decreased with technical advances.^1,3–5 Late-onset bleeding (bleeding occurs 30 days after the procedure), however, has a close relationship with patients’ risk profile, such as age, co-existing comorbidities, and antithrombotic regimens.^5,6

The difficulty of balancing bleeding risk against thrombotic risk after TAVR requires clinicians to be more rigorous in deciding on the antithrombotic plans. A few attempts have been made to predict post-TAVR bleeding. Honda et al⁷ had applied the HAS-BLED score, which was initially designed for patients with atrial fibrillation, to predict bleeding in the TAVR population. Navarese et al⁸ developed a 6-item bleeding risk score, mainly targeting early bleeding after TAVR, with great discriminative power. However, there still lacks a specifically designed risk model for late major bleedings, whose risk profile is considered to be different from early bleedings post-TAVR.

Machine learning has drawn significant attention in clinical prediction models in recent years, due to its ability to effectively model linear and non-linear relationships and interactions.⁹ These techniques have been applied to predict in-hospital outcomes,^10,11 short-term prognoses^12,13 and long-term mortality¹⁴ after TAVR. Despite the great performance of traditional machine learning methods (naive Bayes, random forest, support vector machine, etc) in predicting category outcomes (eg dead or alive), it is challenging for them to deal with time-to-event outcomes as in survival analysis.

Deep learning (DL)¹⁵ is a branch of machine learning methods that transform information through multiple layers of nodes to learn intricate patterns and connections. DL can more flexibly process high-dimensional datasets and perform time-to-event analysis. Multiple networks, such as Cox-nnet^16,17 and DeepSurv,¹⁸ have been proposed to predict survival outcomes from various forms of input data. Therefore, we hypothesized that a properly trained DL model could also accurately predict late major bleeding risk after TAVR from a comprehensive set of clinical features. In this study, we developed and internally validated a DL model named BLeNet to evaluate patients’ late major bleeding risk after TAVR. We compared our model with two standard survival analysis models, the traditional Cox proportional hazard (Cox-PH) and the random survival forest models. The study was conducted in accordance with the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) statement.¹⁹

Patients and Methods

Study Population

The study population was drawn from West China Hospital of Sichuan University Transcatheter Aortic Valve Replacement Registry (WATCH, ChiCTR2000033419). This was a prospectively designed registry that sequentially recruited all aortic stenosis patients undergoing TAVR in West China Hospital. Eligibility for TAVR was evaluated in all patients by the multidisciplinary heart team. The valves were chosen according to the availability at the time of treatment. Follow-up was scheduled at 30 days, 3 months, 6 months, 12 months, and yearly thereafter at clinic visits or telephone interviews. Unscheduled visits at the hospital due to adverse events were also recorded. Data were collected and managed through case record form (CRF) and electronic data capture system. A dedicated data management group was responsible for verifying the data.

From the registry, we included patients who underwent TAVR from April 17, 2012 to May 27, 2020 and followed up till January 30, 2021. Exclusion criteria include 1) patients who were died within 30 days after the procedure and 2) patients who were lost to follow-up within the 6 months after the procedure. The study was conducted in accordance with the Declaration of Helsinki and approved by the Ethics Committee on Biomedical Research of West China Hospital.

Endpoint Definition

The primary endpoints were major or life-threatening bleeding complications (MLBCs), which were defined according to the Valve Academic Research Consortium-2 (VARC-2)²⁰ definition and restricted to life-threatening and major bleeding events that occurred after 30 days. Bleeding events that occurred within 30 days after the procedure were recorded as postoperative complications. The multidisciplinary heart team was responsible for adjudicating the bleeding events.

Data Collection and Preparation

From the West China Hospital of Sichuan University Transcatheter Aortic Valve Replacement Registry, we collected the patients’ baseline characteristics, imaging features, procedural details, post-procedural complications, and discharge antithrombotic medications. Lab tests were obtained from the electronic West China Laboratory Information System. Candidate chemical indicators were chosen according to previous studies^6,21 and those summarized in clinical practice. Society of Thoracic Surgeons (STS) scores and New York Heart Association (NYHA) class were measured within 1 week before the procedure. All other preprocedural features were obtained within 1 week before the procedure, and postprocedural features were obtained within 1 week after the procedure. Variables with more than 20% of missingness were excluded from the analysis.

Categorical and ordinal variables were coded with discrete values. For continuous variables, the original values were retained. Missingness was imputed with missForest algorithm.²² After data coding and imputation, each patient’s characteristics were summarized as a multi-dimensional vector.

Overall, a triplet value was used to represent a patient’s complete information. Here indicates the total number of patients, , which represents the -dimensional vector containing the whole clinical features of the patient . represents the endpoint event (MLBCs is 1 and no MLBCs is 0), and represents the time when patient encountered MLBCs or became censored.

Finally, we adopted the synthetic minority over-sampling technique (SMOTE)²³ to preprocess the highly imbalanced data and Z-score normalization to normalize the input features.

BLeNet Model

In this study, a deep learning model, namely BLeNet, was proposed to estimate the risk of MLBCs after TAVR, as illustrated in Figure 1. The proposed BLeNet model consisted of an input layer, two fully connected feed-forward hidden layers, and an output layer. Each of the first three layers was followed by a batch-normalization layer and a Tanh activation function. The output layer consisted of a single node to estimate the risk index, which corresponded to the log hazard ratio in the Cox-PH model. To train the proposed BLeNet model, we adopted the partial likelihood function as the loss function:

Figure 1 Overview of study design and the architecture of BLeNet model. Input data included TAVR patients’ baseline characteristics, lab tests, CT characteristics, echocardiographic features, procedural details and antithrombotic medications. After data preparation, each patient information was summarized as a triplet value , where indicates the vector containing the clinical features of the patient (grey dots), indicates the outcome event (yellow dots), indicates the time when patient encountered MLBCs or became censored (Orange dots). The diagonal striated grey dots indicate the missing data. The BLeNet model has a hierarchical structure of an input layer, two fully connected feed-forward hidden layers and an output layer with one neuron to estimate the risk index. We evaluated the model’s performance using optimism-adjusted c-index, ROC curve, calibration curve and K-M curve. Abbreviations: CT, computerized tomography; K-M, Kaplan–Meier; MLBCs, major or life-threatening bleeding complications; ROC, receiver operating characteristics; TAVR, transcatheter aortic valve replacement.

where is the number of patients, indicates the feature vector of patient , is the survival time, indicates patient ‘s status (0 = no MLBCs, 1 = MLBCs) at time , and indicates patients who had longer MLBCs-free time than patient . In this loss function, is the hyperparameter that the network needs to learn, and is the output risk index.

The training of the BLeNet model is to optimize the loss function by using the Adam optimization algorithm²⁴ with a learning rate of 10⁻³ and a weight decay of 10⁻⁴. Besides, the training epoch and batch size are 100 and 64, respectively.

In an attempt to better interpret the BLeNet model, we applied the model interpretability library Captum²⁵ to identify feature importance.

Model Validation and Evaluation

To fully evaluate the performance of the proposed BLeNet model, we compared it with two survival analysis methods. One was the traditional Cox-PH model, and the other was the random survival forest model. Feature selection was performed using the least absolute shrinkage and selection operator (LASSO), starting with all candidate variables. We also did a comparison study with these two models retaining all the features as in the BLeNet model.

All models were evaluated in the following metrics. Harrell’s concordance index (c-index) and receiver operating characteristics (ROC) curve were used to evaluate the discriminative performance. Sensitivity and specificity were calculated based on the cutoff point on the ROC curve. Calibration curve was used to evaluate the agreement between predicted risk and observed risk,^26,27 in which integrated calibration index (ICI) and E50 were used as numerical measures. ICI indicates the weighted difference between observed and predicted probabilities, and E50 indicates the median percentile of absolute difference.²⁷ Hosmer–Lemeshow test was also used to test the goodness of fit. We then stratified the patients into high- and low-bleeding risk groups according to the best cutoff point in the ROC curve. Kaplan–Meier estimate was performed to analyze the cumulative incidence of MLBCs in these two groups, and the Log rank test was used to compare the difference.

All of these models were validated by bootstrapping,^28,29 which was recommended by the TRIPOD statement as a preferable validation method in small datasets.¹⁹ The key concept of bootstrapping is to approximate the variation in the general population by the variation in bootstrap samples. This method yields an optimism-adjusted estimate of the overall accuracy through multiple re-sampling process.

Statistical and Technical Specifications

In univariate statistical analysis, continuous variables were expressed as mean ± SD or median (interquartile range) and compared using Student’s t test or Wilcoxon rank-sum test. Categorical variables were expressed as numbers (percentages) and compared using the chi-square test or Fisher’s exact test. Two-tailed p-values were reported, with the p-value <0.05 being considered statistically significant. The above analyses were performed using Stata/SE version 15.1.

Features were standardized with the Z-score normalization method. The time-to-event outcome was analyzed based on our available follow-up data using the Kaplan–Meier estimate and compared using the Log rank test. ROC curve was drawn based on the true- and false-positive rates. The above analyses were performed using the python package “sklearn”. The calibration curve was performed based on the predicted risk against the observed risk, with Hosmer–Lemeshow test applied to test the goodness of fit, and this was done using the python package “lifelines”.

Results

Baseline Characteristics and Univariate Analysis

A total of 718 TAVR patients were retrospectively enrolled in the study, 31 patients (4.32%) died within the first month, 19 patients (1.59%) were lost to follow-up within the 6 months after the procedure, and these patients were excluded from the analysis (Figure 2). In the final study cohort of 668 patients, 64 patients (9.58%) died within a median follow-up time of 25 months (interquartile range: 12.7–39.9 months; cardiovascular death: 24 patients), of which 3 patients died from life-threatening bleeding events (intracranial hemorrhage).

Figure 2 Flow chart of the study population. Abbreviations: MLBCs, major or life-threatening bleeding complications; TAVR, transcatheter aortic valve replacement.

Forty patients (5.99%) were presented with MLBCs within the follow-up period. The median time to the occurrence of MLBCs was 5.5 months (interquartile range: 2.25–14.75 months). Gastrointestinal bleeding was the most frequent type of bleeding (n=32, 80%). Others included intracranial bleeding, intraocular bleeding, and subcutaneous bleeding.

Demographic, echocardiographic, computerized tomographic (CT), biochemical, procedural features and comorbidities of patients stratified according to the occurrence of MLBCs within 3 years are shown in Tables 1 and 2. Univariate analyses in our study population had indicated that patients who developed MLBCs within 3 years were more likely to be of older age (p = 0.003), and had prior peripheral vascular disease (PVD) (p = 0.010) and percutaneous coronary intervention (PCI) history (p = 0.007). They also had relatively lower peri-procedural hemoglobin level (pre-procedural: p = 0.033; post-procedural: p <0.001), higher creatine level (p = 0.046) and higher post-procedural aortic valve (AV) maximal velocity (p = 0.002).

Table 1 Baseline Characteristics Stratified According to the Occurrence of MLBCs Within 3 Years

Table 2 Procedural and Post-Procedural Characteristics Stratified According to the Occurrence of MLBCs Within 3 Years

BLeNet Model Performance

After data preparation and imputation, 56 variables were selected to build the model (Tables 1 and 2). All variables had less than 20% of missing rate. The details of missing data are shown in Supplementary Figure 1. In total, 24 variables (42.8%) and 173 patients (25.9%) had at least one missing value.

The proposed BLeNet model achieved great performance both in discrimination and calibration. In terms of discrimination, the optimism-corrected c-index of the BLeNet model was 0.81 (95% CI: 0.79–0.92) (Table 3), and the AUC of this model was 0.84 (95% CI: 0.76–0.91) (Figure 3). In terms of calibration, the predicted risks of BLeNet were largely consistent with the actual results with an ICI of 0.007 and E50 of 0.005 (Figure 4). Then, we stratified the patients into high- and low-risk groups. The cumulative incidence of MLBCs incrementally increases in the high-risk group compared with that in low-risk patients stratified by the BLeNet model (p < 0.0001) (Figure 5A).

Table 3 Comparison of Cox-PH Model and Random Survival Forest Model with the BLeNet Model

Figure 3 Receiver operating characteristics curve based on 3-year incidence of MLBCs. Compared with the Cox-PH model (Orange line) and random survival forest model (green line), the proposed BLeNet model (blue line) reached a higher AUC (0.84, 95% CI: 0.76–0.91), indicating that this model had better discriminative performance than the Cox-PH and random survival forest models. Abbreviations: AUC, area under the curve; Cox-PH, Cox proportional hazard; RSF, random survival forest.

Figure 4 Calibration curve for 3-year incidence of MLBCs after TAVR. The calibration curve demonstrates the agreement between predicted risk (x-axis) and observed risk (y-axis). The diagonal line demonstrates perfect calibration. BLeNet model (blue line) was better calibrated than the Cox-PH model (Orange line) and random survival forest model (green line) with a calibration curve closer to the perfect line. ICI indicates the weighted difference between observed and predicted probabilities, and E50 indicates the median percentile of absolute difference. The bar histogram indicates the counts of patients with predicted risk on the x-axis. Abbreviations: Cox-PH, Cox proportional hazard; ICI, integrated calibration index; RSF, random survival forest.

Figure 5 Cumulative incidence of MLBCs in high- and low-risk patients stratified by BLeNet, Cox-PH and random survival forest model. (A) Cumulative incidence of MLBCs in high- and low-bleeding risk patients stratified by BLeNet model according to the best cutoff point in the ROC curve. (B) Cumulative incidence of MLBCs in high- and low-bleeding risk patients stratified by Cox-PH model according to the best cutoff point in the ROC curve. (C) Cumulative incidence of MLBCs in high- and low-bleeding risk patients stratified by random survival forest model according to the best cutoff point in the ROC curve. Abbreviations: Cox-PH, Cox proportional hazard; MLBCs, major or life-threatening bleeding complications; ROC, receiver operating characteristics.

To interpret the BleNet model, we applied the Captum interpretation library for pytorch to evaluate feature importance. The top 10 features identified by the Captum interpretation network were cancer history, preprocedural platelet level, post-procedural international normalized ratio (INR), coronary artery disease, preprocedural INR, postprocedural AV mean gradient, postprocedural aortic regurgitation, postprocedural LV ejection fraction, and preprocedural activated partial thromboplastin time (APTT). The average feature importance of all variables is shown in Supplementary Figure 2.

Comparison of BLeNet Model with Cox-PH and Random Survival Forest Model

We compared the BLeNet model with the Cox-PH and random survival forest models in the metrics mentioned above. Feature selection was performed using LASSO regression for these two models, and the selected predictors are listed in Table 4.

Table 4 Variables Selected to Build Cox-PH Model and Random Survival Forest Model

The results demonstrated that the proposed BLeNet model outperformed the Cox-PH and random survival forest models significantly. In terms of discrimination, BLeNet reached a higher c-index (BLeNet: 0.81; Cox-PH: 0.72; random survival forest: 0.70) (Table 3) and higher AUC (BLeNet: 0.84; Cox-PH: 0.72; random survival forest: 0.70) than other two models (Figure 3). In terms of calibration, BLeNet was better calibrated than Cox-PH and random survival forest models with smaller ICI and E50 (BLeNet: ICI = 0.007, E50 = 0.005; Cox-PH: ICI = 0.015, E50 = 0.011; random survival forest: ICI = 0.019, E50 = 0.041) (Figure 4). BLeNet model also achieved better performance than the Cox-PH model in stratifying patients into high- and low-risk groups, as represented in the Kaplan–Meier estimate (Figure 5). In comparison with Cox-PH and random survival forest models built with all candidate features without selection, the BLeNet model also had better performance (Supplementary Figure 3).

Discussion

We developed and internal validated a DL bleeding risk model to accurately evaluate TAVR patients’ late and major bleeding risk after the procedure. The results of our study demonstrated that deep learning might be a promising method to develop prognostic models from readily available clinical data. This model had incorporated baseline, procedural and post-procedural characteristics to give a comprehensive evaluation of patients’ bleeding susceptibility after TAVR. It outperformed the Cox-PH and random survival forest models significantly in discrimination and calibration, and may serve as a useful tool for major bleeding risk evaluation after TAVR.

Late major bleeding has long been identified as a severe post-TAVR complication. Previous studies^6,21,30 found that MLBCs are associated with a two- to fivefold increase in overall mortality, which suggested that controlling bleeding is an important area of improving post-TAVR management.

MLBCs arise from all kinds of complex situations after TAVR. Both the unique hemodynamic features of TAVR patients and the special technical features of TAVR have a role in the pathogenesis pathways.^5,31 Previous studies have identified possible risk factors associated with MLBCs after TAVR, such as age, female, chronic kidney disease, paravalvular leak, baseline hemoglobin, atrial fibrillation.^{6,21,30,32,33} Marion et al revealed CT-ADP, a marker of vWF multimeter proteolysis, as a strong predictor of early and late bleeding events after TAVR. However, the application of CT-ADP is still in the research stage and has not routinely tested in most hospitals yet. Navarese et al⁸ recently developed a 6-item bleeding risk score named PREDICT-TAVR, mainly targeting early bleedings after TAVR. This score consists of hemoglobin, serum iron, anticoagulation and dual antiplatelet therapy, common femoral artery, and creatinine clearance. It had good discriminative power for early bleedings with an AUC of 0.80. However, the risk profile of late major bleedings is not quite the same as early bleedings, and evaluation of such risk is crucial to determine the long-term management plan.

To the best of our knowledge, the BLeNet model is the first model developed with the DL algorithm to analyze the late major bleeding risk after TAVR. We adopted the time-to-event outcome as the prediction target rather than simple classification, which enable the model to give a more accurate quantification of patients’ late major bleeding risk. The model assimilated risk factors identified by previous studies and combined a large set of periprocedural characteristics, so that a more comprehensive evaluation could be obtained.

Machine learning has drawn great attention in TAVR field in these years. Different types of machine learning techniques have been applied to predict in-hospital outcomes,^10,11 short-term prognosis^12,13 and long-term mortality¹⁴ after TAVR. Evidences have shown that machine learning outperformed traditional linear regression models in these classification tasks. However, most of the traditional machine learning classifiers were not designed to handle risk estimation in right-censored survival data. DL possesses more flexibility in dealing with time-to-event outcomes, as it was already been applied for survival analysis in the cancer research field. She et al³⁴ had combined the Cox-PH model with the DL algorithm to predict survival of non-small cell lung cancer (NSCLC) patients from 127 features. It showed significantly better performance than the traditional TNM (tumour, node, metastases) model. Matsuo et al³⁵ compared the DL algorithm with the Cox-PH model to predict progression-free survival and overall survival in cervical cancer patients and got exceedingly good results. The astonishing performance of the DL method may be explained by its inherent advantages. DL can process the raw clinical data directly without the need for complex feature engineering. It can extract intricate patterns and non-linear interactions through multiple layers of transformation and abstraction, and allow more flexible relationships between the variables and outcomes.^15,36 This method can be further extended by combining multi-modality input data. For example, the CT measurements and echocardiographic features in our model can be replaced by raw CT images and echocardiographic videos with sufficient computational power, and surprising findings might be found.

Although interpretation of DL models has always been challenging, we made an attempt with the Captum interpretation library, and some unexpected factors were discovered. Apart from those predictors that have already been found (platelet level, postprocedural AR), cancer history, coronary artery disease and postprocedural ejection fraction were identified as high weight features. However, how specifically these factors contribute to the MLBCs after TAVR still requires further research, both in the clinical aspects and in the interpretation of DL models.

In comparing studies with Cox-PH and random survival forest models, we applied LASSO regression to select high-weight features. Interestingly, atrial fibrillation (AF) was found to be negatively related to MLBCs in this cohort. We speculated that this resulted from the extra attention we paid to those AF patients, since anticoagulation therapy was an established risk factor for bleeding. These patients had a more frequent follow-up schedule, with their antithrombotic regimen consistently adjusted.

Compared with the 6-item PREDICT-TAVR bleeding score,⁸ the BLeNet model has a comparable performance (optimism-corrected AUC of PREDICT-TAVR vs BLeNet: 0.79 vs 0.84). The major potential utility of this model lies in the accurate quantification of patients’ bleeding risk. The output of the model corresponds to the log hazard ratio in the Cox-PH model, which makes the results interpretable to clinicians. This model can also more accurately stratify low bleeding risk patients than the Cox-PH model, even using a relatively crude stratification method based on the cutoff point in the ROC curve. For these low-bleeding risk patients, dual-antiplatelet therapy or vitamin K antagonist could be considered to reduce the rate of thromboembolic events.

Limitations and Expectations

Our study was based on a single-center registry. Though the West China Hospital of Sichuan University Transcatheter Aortic Valve Replacement Registry represents the largest single-center registry in China, we still need more centres’ participation to update and external validate our model. We also recognized that this was a retrospective observational study. Further validation from prospective studies is required to fully evaluate the model’s utility. There is still room for improvement of this model by expanding the dataset, including more patients with the region and background diversity, and incorporating more potential predictors. We also intend to conduct prospective studies to validate the clinical utility of this model.

Conclusion

Deep learning is a feasible way to build prediction models concerning TAVR prognosis. Based on the largest single-center TAVR registry in China, we developed and internally validated a deep learning risk model named BLeNet to comprehensively evaluate patients’ late major bleeding risk after TAVR. This model outperformed the traditional Cox-PH and random survival forest models in discrimination and calibration. It may serve as a useful evaluative tool to facilitate well-informed clinical decisions.