A Deep Learning Model Combining Multimodal Factors to Predict the Overall Survival of Transarterial Chemoembolization

Introduction

Liver cancer is the third leading cause of cancer-related deaths worldwide.^1,2 Hepatocellular carcinoma (HCC) is the most common type of liver cancer.³ Patients with early HCC are therapeutic with transplantation, resection, and ablation, and have an overall 5-year survival rate reaching 70–80%.⁴ However, approximately 80% of patients are in the Barcelona Clinic Liver Cancer staging classification B (BCLC B) at the time of diagnosis, with significantly decreased survival.⁵ Currently, transcatheter arterial chemoembolization (TACE) is the first-line therapy for patients in BCLC B stage.⁶ Nevertheless, the objective response rate for TACE was 52.5% and the 3-year overall survival (OS) was 40.4%.⁷ Accordingly, understanding the prognosis of TACE patients and predicting the survival period of TACE patients is crucial in the overall treatment, which can stratify the risk of diseases and help clinicians to make more personalized decisions, such as determining the follow-up time and planning the treatment programme.

Previous studies have suggested several clinical factors are associated with the prognosis of TACE, including tumor size, tumor numbers, and alpha-fetoprotein (AFP) levels.^8–10 Moreover, relevant researchers have developed scoring systems that integrate various clinical information, such as the BCLC B subclassification, albumin-bilirubin (ALBI) grade, and neutrophil-to-lymphocyte ratio.^11–13 The above methods can assist clinicians in predicting survival to some extent, whereas, they implicitly assume that the relationship between predictor variables and output results is linear, which may not be a valid assumption, and few studies incorporate radiomic information. CT examination contains not only tumor size and imaging features, but also more information about the prognosis of the primary tumor.^14,15 Studies have shown that illustrates that the imaging phenotype at the macro level is closely related to the tissue phenotype at the micro level, and even closely related to molecular pathways and genetic changes.^16,17 Obviously, high-dimensional information mined from CT can be used to predict the survival of TACE patients.

In recent years, image-based deep learning methods have shown promising potential for predicting survival, it constructs a model with a large number of hidden layers, automatically extract features, and find various complex nonlinear relationships between input and output after repeated correction.^18,19 Therefore, deep learning can be used to predict the survival problem of TACE based on CT images.²⁰ However, in these studies, the prediction model did not account for important confounders (such as preoperative clinical variables and postoperative treatment response after TACE), particularly, postoperative treatment response is an important predictor of TACE survival.^21,22 In other words, the deep learning model based on CT images and clinical factors (preoperative and postoperative) to predict the survival of TACE is worthy of further study.

In this study, we aim to develop a deep learning survival prediction model (DLOP_Combin) based on preoperative CT images, clinical factors, and postoperative treatment responses to predict the overall survival (OS) prognosis of TACE in HCC patients in the BCLC B stage, and compare it with other models, to finally provide a reference for individual decision making after TACE and overall treatment.

Materials and Methods

The study was approved by the institutional review board of Harbin Medical University (KY2020-267) and performed by the ethical standards outlined in the Declaration of Helsinki. Written informed consent was waived as this study was collected retrospectively, but patient data is kept strictly confidential.

Patients

A total of 301 patients from the Second Hospital of Harbin Medical University between January 2012 to December 2015 were enrolled. Eligible patients were those primary patients diagnosed with HCC and treated with TACE for the first time. Exclusion criteria were radiotherapy, ablation, or other systemic treatment before TACE, cancers other than HCC, lymph node or distant metastases, lack of necessary clinical information, poor CT image affecting analysis, and no available survival data. All patients were randomly divided into a training cohort and a validation cohort in a ratio of 8:2 (Figure S1).

Patient demographic and clinical characteristics (including age, gender, diabetes, cirrhosis, hepatitis), laboratory variables (including alpha-fetoprotein (AFP), alanine aminotransferase (ALT), aspartate aminotransferase (AST), albumin (ALB), total bilirubin (TBIL), microvascular infiltration (MVI), BCLC B subclassification, albumin-bilirubin grade (ALBI), child-pugh score were obtained from the electronic medical records. Tumor characteristics (number, size, location) were recorded from radiology reports.

CT Image Acquisition

All enrolled patients underwent enhanced abdominal CT scans before treatment. Detailed CT scans are given in Appendix E1. CT images of all patients were reviewed by radiologists with 8 and 10 years of diagnostic imaging experience, respectively, without knowledge of clinical information and follow-up data.

Follow-Up and Treatment Response

The endpoint was overall survival, defined as the time between the date of initial TACE and death from causes relevant to this study. To ensure that OS was evaluated in the context of clinical practice, patients received subsequent radiotherapy, surgery, or sorafenib after TACE treatment according to clinical guidelines. Data from patients lost to follow-up or surviving at the last follow-up (January 2022) were considered as censored data.

Appendix E2 for details of the TACE process. CT enhancement /MRI follow-up was performed every 4–6 weeks after TACE until disease stabilization. The treatment response of target lesions was assessed according to the modified Response Evaluation Criteria in Solid Tumors (mRECIST1.1).²³ Treatment response was assessed independently by 2 radiologists with ≥ 8 years of experience, and results were categorized as complete response (CR), partial response (PR), stable disease (SD), and progressive disease (PD).

Model Construction

The workflow is shown in Figure 1.To integrate CT images and clinical data, we propose an improved resnet101 deep learning model (DLOP_Combin) Appendix E3. The model consists of three main parts: CT feature extraction, clinical feature integration and multimodal features fusion.

(1) CT feature extraction - Input CT image size is set to 224*224 pixels. High-dimensional semantic information was extracted from CT images by stacking residual convolution blocks, and features were reduced to one dimension by globally averaging pooling layers with the following structure: two convolutional layers, two pooling layers two, two BatchNormalization layers and one residual connection structure. To visualize the figure, a thermal peak activation map was generated using Grad CAM.

Figure 1 Workflow of TACE patient survival prediction model based on deep learning (DLOPcombin).

(2) Clinical feature integration - We trained a multilayer perceptron classifier (MLP) consisting of three fully connected 16, 32, and 64 hidden units to splice clinical factors after univariate and multivariate COX regression analysis.

(3) Multimodal features fusion – The CT feature is 2048 dimensions and the clinical feature is 64 dimensions. CT features and clinical features were placed into the DLOP_Combin fully connected layer, which consisted of five perceptrons. Represents the survival probability of the following time intervals: 0–365 days, 365–730 days, 730–1095 days, and 1095–1460 days, respectively. The final output layer is defined as [cumulative survival probability (product of the first three perceptrons)] *100. The final output of the model represents an approximate probability of a 3-year survival (1095 days).

Model Comparison

To verify the validity of the deep learning model proposed in this study, DLOP_Combin was compared with the following four survival prediction models (Appendix E4).

1) DLOP_CTR model: The above obtained image target areas were applied to Resnet101 to build a TACE survival prediction model for liver cancer based only on CT images.

2) DLOP_Cox model: The output of the DLOP_CTR model was used as a separate variable and combined with clinical factors to build a multi-factor COX regression prediction model.

3) Radiomic model: The image information was transformed into high-dimensional data, and the survival prediction model was established through image normalization, delineation of interest region, feature extraction, correlation coefficient calculation and LASSO COX dimension reduction.

4) Clinical model: Univariate and multivariate cox regression analyses were used to predict independent risk factors for OS, and clinical models were constructed for clinical variables with statistical differences.

Statistical Analysis

Patient characteristics were compared using the x² test or Fisher test. Univariate and multifactorial COX regression analyses were performed to screen for the factors most associated with survival. To assess the discriminatory performance of DLOP, we calculated the Harrell C index. The performance of the model was evaluated using time-dependent area under the receiver operating characteristic curves (TD AUCs) and positive and negative predictive values were calculated. The calibration curves were plotted to assess the degree of overfitting of the model. The Youden index was used to calculate the optimal cut-off point and to classify patients into high-risk and low-risk classes. The clinical efficacy of the model was analyzed using DCA curves. The discriminatory performance of DLOP_Combin was benchmarked and the student’s t-test was used to compare the C indices of other models. DeLong test is used to compare TD AUC between models. The false discovery rate (FDR) method is used for calibration. In addition, the output of DLOP_Combin was used as a separate variable to reconcile multivariate risk ratios (HRs) using Cox regression analysis.

All statistical analyses were performed using R software. P < 0.05 were considered statistically different Appendix E5.

Results

Patient Characteristics

A total of 301 patients with HCC were included in our retrospective study. The detailed results of patients in the training cohort (n=241), validation cohort (n=60) are summarized in Table 1. There was no significant difference between the training cohort and the validation cohort in all patient characteristics.

Table 1 Baseline Demographic and Clinical Characteristics

Development and Validation of DLOPCombin

After adjusting for variables in the multivariate Cox regression model in the training cohort, treatment response, MVI, ALBI score and BCLC B subclassification were significant prognostic factors in predicting OS (Table 2). The TD AUC of DLOP_Combin in predicting 3-year survival in the training and validation cohorts were 0.98 (95% CI: 0.94, 1.00) and 0.96 (95% CI: 0.88, 1.00), respectively (Table 3 and Figure 2A and B). The C-index of the DLOP_Combin for OS prognosis was 0.90 (95% CI: 0.85, 0.95) in the training cohort and 0.88 (95% CI: 0.80, 0.96) in the validation cohort (Table 4). Calibration curves show good agreement between observations and predictions (Figure 2C and D).The Youden index is 0.902, Figure 3A and B show that patients with a score ≤ 0.902 showed significantly better OS than those with scores higher than 0.902.

Table 2 Univariate and Multivariate Cox Regression Analysis of Clinical Factors in the Training Cohort

Table 3 Performance of the Model in Predicting the Prognosis of HCC Patients in the Training Cohort and the Validation Cohort

Table 4 Discrimination Performance and Calibration of the Survival Prediction Model

Figure 2 Performance of different survival prediction models. Time-dependent receiver operating curves under the area for 3-year survival of patients (A) training cohort (B) validation cohort. Calibration curves of the model (C) training cohort (D) validation cohort.

Figure 3 Kaplan-Meier estimates of overall survival according to the Youden index of DLSPcombin categorized high- and low-risk groups (A) training cohort (B) validation cohort.

Comparison of DLOP_Combin with Other Models

In the validation cohort, the TD AUC of DLOP_CTR was 0.89 (95% CI: 0.80, 0.98) and the C-index was 0.75 (95% CI: 0.75, 0.89) (Table 3 and Table 4). In addition, multifactorial COX regression analysis of the training set showed that the output variables of DLOP_CTR were significantly associated with OS (HR: 0.14; 95% CI: 0.10, 0.21; P < 0.01) (Table S1). For the clinically adjusted model (DLOP_Cox), the C-index was 0.79 (95% CI: 0.69, 0.89) for the DLOPcox training set and 0.77 (95% CI: 0.70, 0.84) for the validation set (Table 4).

Five radiomic features were selected by LASSO COX regression analysis to construct the radiomic model (Table S2 and Figures S2, S3). The TD AUC of the Radiomic model was 0.70 (95% CI: 0.61, 0.79) and 0.63 (95% CI: 0.49, 0.76) for the training and validation sets, respectively. In addition, the clinical model showed the lowest TD AUC: 0.66 (95% CI: 0.60, 0.72) (Table 3). The details as shown in Appendix E6.

The TD AUC of DLOP_Combin was 7.9%, 7.9%, 52.4%, and 54.8% higher than those of DLOP_CTR, DLOPcox, Radiomic, and clinical model, respectively. DeLong test results show that DLOP_Combin model is significantly different from other models (Figure 4A–D). The NPV of DLOP_Combin was higher than those of the above four models (Table 3). Furthermore, DLOP_Combin has a significant difference with other models in C index (P < 0.05) (Table 4). In addition, the DCA in the validation cohort showed that the DLOP_Combin model had higher overall net benefit than other models (Figure 5A and B). Furthermore, the Kaplan-Meier analysis of OS of BCLC substage is shown in Figure S3.

Figure 4 Results of the DeLong test between the different models (A) training cohort (B) validation cohort. Results of the FDR after calibration of the p-value (C) training cohort (D) validation cohort.

Figure 5 Decision curve analysis of the (A) training cohort (B) validation cohort.

Activation Mapping of the DLOPCombin

Activation profiles of HCC tumors showed that the deep learning survival prediction model had different phenotypic patterns among patients. As shown in Figure 6, the characteristic maps of patients with longer survival periods showed richer activity in the tumor internal heat map (Figure 6A1-3). In contrast, in patients with poor TACE response, the peritumor heat map activity was relatively complex (Figure 6B1-3). An example of a prediction error is shown (Figure 6C1-3), which may indicate that different activation patterns predict TACE postharcoma.

Figure 6 Examples of CT images (A), Heatmap (B), and their corresponding gradient-class activation maps (C) on the validation cohort. On both the heat map and the gradient class activation map, red areas indicate that the area may be associated with a high probability of survival for the patient, while blue lesions indicate a low probability of survival. (A) A 64-year-old woman with 4 years of survival after TACE and Ct-based deep learning survival prediction model activation focused on the tumor region. (B) A 52-year-old male with no internal tumor activation and a disease-free survival of 240 days. (C) The patient with the incorrect prediction, a 48-year-old male, survived 397 days after surgery. The model predicted low risk; however, death occurred after 1254 days.

Discussion

We developed a deep learning survival prediction model (DLOP_Combin) based on CT images and postoperative treatment responses for the first time to successfully predict 3-year survival in patients with HCC undergoing TACE. The model performed well in the training set and validation set. The results were better than DLOP_Cox and DLOP_CTR models (C index: 0.98 vs 0.92, P < 0.001; 0.96 vs 0.89, P < 0.001). In addition, the TD AUC of DLOP_Combin was higher than the Radiomic model (0.96 vs 0.63; P < 0.001) and clinical model (0.96 vs 0.62; P < 0.001).

CT images contain some important information reflecting HCC.¹⁵ It shows the status of the tumor and contains information about the tumor perimeter, which is 5 cm around the tumor and is an important cause of death in patients with TACE. Previous studies have shown that CT-driven parameters have been shown to correlate with disease prognosis or severity. In this regard, deep learning can not only extract important prognostic information directly from CT images through end-to-end learning, but also learn more complex decision boundaries by training complex multivariate functions to fit the data.^18,19 In this paper, our constructed CT-only deep learning model successfully extracts important features from CT images, quantifies them into values that are correlated with patient outcomes and showed good performance. However, further in-depth study showed that the DLOP_Combin model after combining CT images and clinical variables yielded a model with the best performance.

In the past, the prediction of survival in TACE patients by clinical factors has been extensively explored. For example, Campani et al used different clinical variables to study survival in TACE with some success.^24–26 Jung found that treatment response was an important variable in predicting prognosis in TACE,²¹ and similarly this paper found that TACE treatment response was an independent prognostic factor associated with survival through multivariate COX regression studies. In fact, treatment response contains important information after TACE, including the status of the target lesion, lymph node condition, and the presence of new lesions, and may provide more prognostic information than pre-TACE treatment variables.²²

More interestingly, two different approaches, deep learning and Cox regression, were tried in integrating CT images with postoperative treatment response. The results reveal that the performance of deep learning is higher than that of Cox regression, which may be mainly due to the following points: (1) the deep model represents CT images and clinical features in different network layers, which is easy to classify and identify the information; (2) the deep learning model is a nonlinear framework, which can fit the data better by optimizing and adjusting the parameters. To further validate the performance of the proposed DLOP_Combin model in terms of survival prognosis, we also built a radiomics model, and the results showed that the DLOP_Combin model outperformed the radiomics model in predicting the survival prognosis of TACE.Jiang also obtained the same results, and the analysis may be due to the fact that the multilayer structure in the deep learning neural network can automatically learn hierarchical rich image features to improve robustness.²⁷ At the same time, the deep learning model avoids redundant proposal generation, caused by the lower learning efficiency.

In our study, we controlled for several important confounders, including the number and size of HCC, by univariate and multifactorial cox regression analyses that were not shown in prognostic models. Previous studies have shown that the number and size of HCC are important prognostic factors for survival.²⁸ The reason for not including these factors in this study is that treatment response contains both variables and there is linear overlap in multivariate COX analysis. Our study also showed that BCLC B subclassification, albumin-bilirubin (ALBI) grade and microvascular infiltration are clinical factors associated with TACE prognosis, which is consistent with previous studies.^29–31 Finally, clinical model as also has some predictive power for the prognosis of TACE. However, in our study, the TD AUC of the clinical model validation set was 0.62. The analysis may be that the clinical model ignores the information of image heterogeneity, which may be the reason for the decreased performance.

Our study has some limitations. Firstly, as a retrospective study, treatment route was an important confounding factor that was difficult to control for, including potential differences in immunotherapy and adjuvant therapy. Although treatment was performed according to treatment guidelines, some variability may still exist in actual clinical practice. Secondly, although we proposed a CT image activation map, the mechanism of DLOP_Combin model to predict TACE prognosis is unclear and not better explained at present, and the mechanism of deep learning characterization needs to be further investigated. Thirdly, regarding differences in cancer biology, our model was developed and validated in the context of the majority of HBV-associated HCC patients, but hepatitis C and alcoholism are the leading causes of HCC in Western countries. This may lead to differences in the choice of TACE treatment options in clinical practice, therefore, the model should be validated in a Western setting. Finally, this study is a single-center study and lacks an external validation set. In the future, we will expand the sample size and introduce an external validation set to further improve the reliability and generalization of the study.

In conclusion, deep learning models based on preoperative CT and postoperative treatment response for predicting disease-free survival in HCC patients are important tools for preoperative risk stratification and can facilitate clinical decision making in the era of precision medicine.