A Transformer-Based Deep Learning Model for predicting Early Recurrence in Hepatocellular Carcinoma After Hepatectomy Using Intravoxel Incoherent Motion Images

Introduction

Hepatocellular carcinoma (HCC) is the most common primary liver cancer and a major cause of cancer-related death worldwide.¹ Resection is considered the first-line treatment for HCC, particularly for patients with early-stage HCC and well-preserved liver function.² However, the prognosis of HCC remains poor, and its recurrence rates after surgery reach 80%.³ HCC recurrence is generally classified as early (within two years) or late (>2 years) recurrence.⁴ Early recurrence of HCC is related to intrahepatic metastases and is usually associated with a worse prognosis.^5,6 Therefore, identifying patients at a high risk of recurrence after surgery might allow clinicians to provide appropriate surveillance to detect HCC recurrence at its earliest stage. It could also help identify potential candidates for clinical trials of adjuvant therapy, which could improve individualized management and gain prognostic benefits.^4,5,7

Some specific factors were associated with early recurrence of HCC, including large tumor size, multiple tumors, lower true diffusion coefficient (D) values, lower apparent diffusion coefficient (ADC) values, microvascular invasion (MVI), and poor histologic differentiation.^3,4,8–10 Deep learning (DL) can automatically extract and learn deep features from original images to comprehensively quantify tumor heterogeneity. Numerous studies reporting the application of DL have demonstrated remarkable performance in enhancing the diagnosis and predicting the prognosis for HCC, offering promising clinical advantages.^11–13 Recently, DL, which uses a vision transformer (ViT) framework, has emerged as a promising technique for many medical image processing tasks, replacing convolutional neural networks (CNNs) due to its superior performance and robustness.^14–16 By incorporating the self-attention and multi-attention mechanisms, the ViT framework gains a deep understanding of the weight of importance of each input data point and effectively captures intricate spatial relationships, patterns across different levels, and locations of multiple sequences within the input data that are usually neglected in CNNs.^17–19 Moreover, some previous studies about 2D or 2.5D tumor analysis resulted in less effective capture of the morphological and spatial information of the entire tumor.^20,21 Given the high heterogeneity of HCC, we used 3D analysis based on whole tumor delineation that can comprehensively evaluate the information about the entire tumor.

However, most DL studies for HCC prognosis were based on routine magnetic resonance imaging (MRI) sequences,^6,8,22 while advanced techniques such as intravoxel incoherent motion (IVIM) have seldom been employed. Notably, a previous study extracted the deep features of multiple b-value and IVIM diffusion-weighted imaging (DWI) images using traditional deep CNNs to predict MVI in HCC.²³ The IVIM technology is derived from DWI with multiple b-value images without using contrast agents, reflecting tissue diffusion and microcapillary perfusion separately. Traditional quantitative IVIM parameters have showed limited diagnostic efficacy in evaluating HCC prognosis,⁹ often neglected high-dimension information hidden behind the IVIM images, and this high-dimension information can be further effectively captured through ViT-DL network. Therefore, this study aimed to develop and validate a ViT-DL network based on IVIM-DWI for accurately predicting early recurrence of HCC.

Materials and Methods

Study Population

This retrospective study was approved by the Ethics Committee of Nanfang Hospital of Southern Medical University (approval number: NFEC-202305-Y10). The study was conducted in compliance with the Declaration of Helsinki. Informed consent was waived by the Ethics Committee due to the retrospective design of anonymized patient data. Between December 2015 and July 2022, 350 consecutive patients with suspected or confirmed malignant hepatic lesions underwent preoperative gadoxetate disodium-enhanced liver MRI examination with a nine b-values DWI sequence at our institution. The inclusion criteria were as follows: (1) patients with postoperative pathological confirmation of HCC; (2) preoperative gadoxetate disodium-enhanced abdominal MRI, including a nine b-values DWI sequence, performed within one month before the operation; (3) tumor without macrovascular invasion on preoperative MRI; (4) no history of preoperative treatment for HCC, such as radiofrequency ablation, transarterial chemoembolization, or other targeted therapies. The exclusion criteria were as follows: (1) signs of extrahepatic metastases on imaging (n=6); (2) history of other malignant tumors (n=1); (3) poor image quality on MRI scans (n=13); (4) incomplete clinical, radiological, pathological, or follow-up data (n=95); (5) small HCCs less than 1 cm in diameter or HCC showing iso- or hypointensity on DWI (n=3). Finally, 122 patients were included in this study. These patients were randomly divided into training and test sets at a 7:3 ratio. The patient flowchart is shown in Figure 1.

Figure 1 The patient flowchart.

Clinical Characteristics and Follow-Up

The clinical information included age, sex, alpha-fetoprotein (AFP) level, alanine transaminase (ALT), aspartate aminotransferase (AST), total bilirubin, direct bilirubin, indirect bilirubin, albumin, platelet count, Child-Pugh grade, Barcelona Clinic Liver Cancer Stage, tumor size and origin of liver disease.

All patients were followed up for at least two years after surgical resection. Patients were screened for tumor recurrence through serum AFP level, ultrasonography, and/or contrast-enhanced CT or MRI examination every three months in the first year and every 3–6 months after that. The censored follow-up date was August 1, 2024.

Early recurrence was defined as one or more of the following events occurring within two years after hepatectomy: (a) presence of new hepatic lesions with typical imaging findings of HCC; (b) atypical imaging findings with biopsy or re-postoperative histopathology confirming HCC diagnosis, or if postoperative transarterial chemoembolization indicated tumor staining; (c) extrahepatic metastases confirmed by typical imaging features or histopathology. No early recurrence was considered if recurrence occurred after more than two years or if no recurrence occurred during follow-up.

MRI

The MRI scans were obtained using two 3.0 T MRI systems (Achieva and Ingenia, Philips Healthcare, the Netherlands). The IVIM-DWI sequences with nine b-values (b = 0, 10, 20, 40, 80, 200, 400, 600, and 1000 s/mm²) were performed using a respiratory trigger with a single-shot echo-planar sequence. MRI sequences and scan parameters are listed in the MRI protocols in the Supplementary Materials 1 and Supplement Table S1.

Post-Processing IVIM Maps and Parameter Analysis

The nine b-value DWI images were transferred to a workstation (version AW4.6, GE Healthcare) with Functool software to generate post-processing IVIM parameter maps (including ADC, D, D* and f maps).

All nine b-value images were used as input data. IVIM data were analyzed using in-house software based on 64-bit MATLAB 2014b (Math Works, Natick, MA), automatically generating the quantitative IVIM parameters, including ADC, D, D* and f values. Method for quantitative IVIM parameters analysis were provided in Supplementary Materials 2.

Image Pre-Processing and Segmentation

The patient’s nine b-value images and IVIM parametric maps were utilized for further analysis. Preprocessing method of original images about nine b-value images and IVIM parametric maps were recorded in Supplementary Materials 3.

Subsequently, above mentioned two radiologists with six and ten years of experience manually delineated the VOIs in the nine b-value DWI images to encompass the entire tumor using ITK-SNAP software. Meanwhile, the delineation of VOIs on DWI (b=1000s/mm²) automatically transferred to four IVIM parametric maps to gain tumor VOIs on the corresponding four parametric maps. The VOIs were confirmed or corrected by the above mentioned senior radiologist with more than 15 years of work experience. Then, the tumor areas were extracted from the original images using Python. And the detailed information about tumor areas processing were provided in Supplementary Materials 4. The processes of image preprocessing and tumor segmentation were shown in Figure 2.

Figure 2 Images preprocessing (A) and tumor segmentation (B).

Histopathological Analysis

The pathological reports of all included patients were retrospectively reviewed. Pathology data included the background liver parenchymal tissue, Edmondson-Steiner grade, and the presence of MVI. MVI was defined as tumor tissue within a vascular space lined by endothelial cells and visible only by microscopy. All pathology examinations were performed by two pathologists with more than ten years of experience in liver pathology who were blinded to the radiological and clinical findings.

Vision Transformer DL Network Construction

We used the ViT-DL model to predict early recurrence of HCC based on the IVIM-DWI images. The input dimension was the 64 × 64 × 64 × 9 IVIM-DWI images generated by channel stacking, which consisted of nine b-value images from the axial view VOIs of the tumor.

We also evaluated the ViT-DL model based on IVIM parametric maps. The input images were normalized tumor areas extracted from four IVIM parametric maps. The deep features used in our combined model were obtained from the last transformer block. The model had 100 training iterations, a batch size of 25, and an initial learning rate of 6e-6. We chose AdamW optimizer with L2 regularization, set to 3e-4 in server graphics card [NVIDIA GeForce RTX 3090 (24 GB) GPU]. Details related to the ViT-DL model development are described in Figure 3 and Supplementary Materials 5.

Figure 3 Model development. Abbreviations: ViT, Vision transformer; DL, deep learning; ER, early recurrence.

Feature Extraction and DL Model Development

We used the ViT-DL model to extract two feature sets, ViT-bDL and VIT-mDL, from the tumors in nine b-value images and IVIM parametric maps, respectively. A fusion feature set (ViT-fDL) was generated by performing concat and layer normalization on the ViT-bDL and VIT-mDL feature sets. We developed three DL models: (1) using only the ViT-bDL features (ViT-bDL model); (2) using only the ViT-mDL features (ViT-mDL model); (3) using the ViT-fDL features (ViT-fDL model). To prevent model overfitting, we first performed feature dimension reduction using Pearson’s coefficient analysis with a threshold of 0.8 and then analyzed the screened features by ANOVA to obtain the most relevant deep features. We also used the five-fold cross-validation method in our DL models in the training set to enhance model robustness. The ViT-fDL model was based on the ten most relevant deep features, and its output probability was used to predict early recurrence, which represented the likelihood of patients with HCC exhibiting early recurrence after surgery.

Construction of the Clinical and Combined Models

Statistically significant clinical predictors were identified by multivariate logistic regression analysis and the calculated odds ratios (ORs) using the training set. Subsequently, a clinical model was developed as a baseline for comparison. A combined model integrating the deep features from the ViT-fDL model and independent clinical predictors was then constructed to further enhance the model’s performance in predicting early recurrence of HCC. Details of the clinical, ViT-fDL, and combined model development and training processes are described in Figure 3.

Model Performance Assessment and Interpretability

The diagnostic performance of the models in predicting early recurrence of HCC was assessed by the area under the curve (AUC) of the receiver operating characteristic curve (ROC) with 95% confidence interval (CI). The corresponding sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were also calculated. Model fitting was assessed by using calibration curves using 1000 resampling bootstraps and the Hosmer–Lemeshow test. Decision curve analysis was implemented to estimate the clinical utility of the models by quantifying their net benefits under various threshold probabilities. Gradient-weighted Class Activation Mapping (Grad-CAM) were used to visualize the location and distribution of decision feature information captured by the DL model in predicting early recurrence of HCC.

Statistical Analysis

Statistical analysis was performed using IBM SPSS Statistics for Windows, Version 20.0 (IBM Corporation, Armonk, NY, USA), MedCalc, Version 18.2 (MedCalc Software Ltd, Mariakerke, Belgium), and R (version 4.4.1, http://www.r-project.org). Continuous parameters were evaluated using an independent sample t-test (normally distributed) or Mann–Whitney U-test (non-normally distributed). Categorical variables were compared by the χ² or Fisher’s exact test. Inter-observer and inter-scanner variability in the IVIM parameters were analyzed by calculating the interclass correlation coefficient (ICC), ICC value of >0.75 was considered to represent good agreement. Univariate predictors with P < 0.05 were used in the multivariate logistic regression analysis to identify independent predictors for early recurrence of HCC to be used in the clinical model. The least absolute shrinkage and selection operator-logistic regression algorithm was applied to identify the most valuable features for predicting early recurrence of HCC in our combined model using the 5-fold cross-validation penalty procedure. Two-sided P values < 0.05 were considered statistically significant.

Results

Baseline Patient Characteristics

The study included 122 patients, 85 in the training set and 37 in the test set. The cohort included 49 (40.1%) patients with early recurrence, with similar rates in the training (40.0%, 34/85) and test (40.5%, 15/37) sets (P = 0.971). By the end of follow-up, the remaining 73 of the 122 patients censored (without observed early recurrence), and the proportion of censored cases in total population was 59.84%.

Baseline patient clinical characteristics in the training and test sets were similar (Table 1), and baseline clinical characteristics were not significantly different between the training and test sets (all P > 0.05). In the training set, tumor size, AST, MVI, and D were significantly different between patients with and without early recurrence (all P < 0.05; Table 2), so these were incorporated into the multivariate logistic regression model. Multivariable logistic regression showed that tumor size (OR=1.870; P = 0.003) and D value (OR=0.037; P = 0.011) were independent predictors for early recurrence of HCC. The baseline clinical characteristics of patients with and without early recurrence of HCC in the test set are shown in Table 3.

Table 1 Baseline Clinical Characteristic of Training and Test Sets

Table 2 Comparison Baseline Clinical Characteristics Between Early Recurrence and without Early Recurrence HCCs in Training Set

Table 3 Comparison Baseline Clinical Characteristics Between Early Recurrence and without Early Recurrence HCCs in Test Set

Interobserver Agreement in the Training Set

The ICC assessment indicated good-to-excellent agreement between radiologists in evaluating the IVIM parameters (ICC=0.800–0.884). Agreement between the two MRI scanners in evaluating the IVIM parameters was good-to-excellent (ICC=0.996, Supplementary Table S2).

Performance of the ViT-DL Models

The respective AUCs in the training and test sets were 0.957 (95% CI, 0.889 −0.989) and 0.747 (95% CI, 0.577–0.875) in the ViT-bDL model, 0.751 (95% CI, 0.646 −0.839) and 0.594 (95% CI, 0.420–0.752) in the ViT-mDL model. The ViT-fDL model which combine deep learning features of 9 b-value images and IVIM parametric maps improved performance in both training and test sets, with AUC of 0.968 (95% CI 0.905–0.994) in the training set and 0.815 (95% CI 0.653–0.923) in the test set.

Performance of the Clinical and Combined Model

The AUCs in the training and test sets in the clinical model were 0.755 (95% CI, 0.650–0.842) and 0.764 (95% CI, 0.596–0.887), respectively. We selected the best-performing ViT-fDL model from three deep learning models to integrate its ten most relevant deep features with the independent clinical predictors to generate the combined model, which achieved AUCs of 0.991 (95% CI, 0.940–1.000) and 0.821 (95% CI, 0.660–0.927) in the training and test sets, respectively (Figure 4 and Table 4).

Table 4 Diagnostic Performance of the Clinical Model, ViT-fDL Model, and Combined Model

Figure 4 Model evaluation. The ROCs for the clinical model, ViT-fDL model, and combined model of the training set (A) and test set (B). The DCAs for the clinical model, ViT-fDL model, and combined model of the training set (C) and test set (D).

Decision curve analysis revealed that the combined model performed good clinical utility in both the training and test sets in predicting early recurrence of HCC (Figure 4). Calibration curves for the combined model in both the training and test sets are presented in Supplementary Figure S1, which showed predicted and actual probabilities of HCC with early recurrence. And both P values of the Hosmer-Lemeshow test in the training (P=0.932) and test sets (P=0.125) were greater than 0.05, which also showed that fitting of combined model was acceptable, it means no significant difference between the predicted probability and the actual probability.

Model Interpretability

Grad-CAM provided valuable information for predicting early recurrence of HCC. It facilitated visualizing the pixel weight distribution using various colors. The visualization results of the ViT-DL model are shown in Figure 5 to better understand the differences in the IVIM images between patients with and without early recurrence of HCC. Notably, Grad-CAM indicated that the DL network fixated most of its attention on the multiple and large tumor marginal regions in early recurrence of HCC, while Grad-CAM showed that few regions were activated within the HCC tumor of without early recurrence.

Figure 5 Visualization of gradient-weighted class activation mapping (Grad-CAM). Representative images and illustration of visualization results for early recurrence case (A) and without early recurrence case (B). From left to right in each row: origin DWI b=1000s/mm2 image; the cropped tumor region for input; Grad-CAM; Grad-CAM overlaid on the cropped tumor region, where red indicates highest contribution to the classification, followed by yellow, while green and blue regions indicate lower contribution. (A) Grad-CAM showed that large and multiple marginal regions activated within the tumor of early recurrence case, indicating that tumor heterogeneity and margin invasion contribute to prediction of early recurrence in HCC; (B) Grad-CAM emphasized few regions activated within the tumor of without early recurrence case.

Radiomics quality score (RQS) provided in Supplementary material 6, and the codes for ViT-DL model analysis are available in a GitHub repository (https://github.com/alan-qiu-gzucm/CL-VIT).

Discussion

This study established a ViT-fDL model for predicting early recurrence of HCC by extracting deep features from nine b-value images and IVIM parametric maps. The ViT-fDL model demonstrated better predictive performance for early recurrence of HCC than the ViT-bDL, ViT-mDL, and clinical models. This ViT-fDL model could become a noninvasive and effective approach for predicting early recurrence of HCC.

By integrating deep features from the ViT-fDL model with clinical features, we developed and validated a combined model that accurately identified HCC patients at high risk of early recurrence, achieving the best predictive performance with AUCs of 0.991 and 0.821 in the training and test sets, respectively.

Notably, considering IVIM sequences did not require contrast agents, our combined model demonstrated efficacy in predicting early recurrence of HCC, similar to previous studies based on contrast-enhanced imaging.^8,22 Therefore, the combined model is of great clinical significance for patients with HCC who cannot undergo examinations with contrast agents.

This study evaluated the performance of quantitative IVIM parameters for predicting early recurrence of HCC. Previous studies showed that lower ADC and D values were useful for predicting early recurrence of HCC. However, we found that only the D value was an independent predictor for early recurrence in the clinical model. This difference could be because previous studies derived the ADC from a single ROI on a maximum tumor cross-section.^3,9 In contrast, we used VOIs, which included necrotic portions, yielding different results. We believe that whole tumor analysis, as done in our study, captures the high heterogeneity of HCC. Relying solely on maximum tumor cross-section might not offer comprehensive information about the entire tumor. Our results showed the tumor size was another independent predictor for early recurrence of HCC, similar to previous studies.^5,21 We built a clinical model based on the D value and tumor size, resulting in AUCs of 0.755 and 0.764 in the training and test sets, respectively. However, high-dimension features of these data could not be extracted by traditional statistical methods for further analysis, limiting the value of clinical model.

Previous research predominantly relied on 2D analysis of the maximum HCC tumor cross-section to predict MVI and survival outcomes.^8,21,23 Considering the high heterogeneity of HCC tumors, relying solely on maximum tumor cross-section might not offer comprehensive information about the entire tumor.¹⁷ Moreover, some studies have confirmed that the whole-tumor volume was associated with HCC prognosis.^24–26 Our ViT-bDL model processed and integrated diverse data through a multi-attention mechanism, efficiently mining high-dimensional image features to comprehensively quantify tumor information. Furthermore, our ViT-bDL model superimposed the nine b-value images in the channel dimension, facilitating a deep understanding of the content of each b-value image and capturing potential patterns and complex spatial relationships between multiple b-values.^17,18,27 Consequently, the ViT-bDL model provided accurate and comprehensive insights for predicting early recurrence of HCC.

We also evaluated the performance of IVIM parametric maps, using the ViT-mDL model, in predicting early recurrence of HCC. Our results showed that the ViT-bDL model performed better than the ViT-mDL model. This difference adequately reflects the advantages of multiple b-value images over IVIM parametric maps when using ViT deep features for predicting early recurrence of HCC. This difference is due to the fact that the IVIM parametric map calculation is usually prone to errors and is calculation intensive due to the pixel-by-pixel fitting of the bi-exponential model.^23,28 These drawbacks could be responsible for the relatively poor predictive efficiency of the IVIM parametric maps for early recurrence of HCC. To overcome this limitation and improve the predictive efficiency, we constructed a fusion ViT-fDL model that combined the ViT-bDL and ViT-mDL models.

Grad-CAM is a visual interpretation tool that helps clinicians explain how the ViT-DL model extracted deep features from regions inside the tumor that are most influential for the model’s predictions. Grad-CAM showed that most attention was paid to extracting deep features from the tumor’s margins for predicting early recurrence of HCC. Tumors in patients with early recurrence of HCC showed more heterogeneity than those in patients without early recurrence of HCC. Consistent with Zhao et al,²² our results suggest that intra-tumoral heterogeneity and invasive margins contribute to predicting early recurrence of HCC.

The limitations of this study must be acknowledged. First, single center with a relatively small population especially test set and potential selection bias may affect the generalization of our model, The IVIM sequence might not be available at some institutions. Furthermore, the setting of the multiple b values in the IVIM sequence is uncertain. However, our combined model yielded a predictive accuracy for HCC prognosis similar to that of DL research using contrast-enhanced MRI. Therefore, expanding the database and performing prospective multi-center studies using IVIM sequences should be performed in future in order to ascertain the generalizability of the model. Second, the calibration curves were not very ideal due to the relatively small sample size. However, the mean absolute error values of the calibration curves in both the training and test sets were less than 0.1, which indicated that the overall fitting effect of our combined model were acceptable, In general, the overall fitting effect of our combined model through MAE analysis and Hosmer–Lemeshow test (all P>0.05) is reasonable. Thus, there is room for improvement in the calibration curve through large-scale and multi-center studies in future. Third, we used early stopping and cross-validation techniques to mitigate the risk of overfitting arising from our small sample. Finally, our study aimed to explore whether a transformer framework-based DL network based on IVIM images could predict early recurrence of HCC preoperatively. Future research should integrate other conventional scanning sequences, such as T2WI. The fusion of multi-sequence information could help to explore effective biomarkers for predicting early recurrence of HCC.

Conclusion

A combined model that incorporated deep features from the ViT-fDL model and independent clinical predictors achieved excellent predictive performance for predicting early recurrence of HCC. Thus, the combined model is proposed as a useful tool for predicting early recurrence of HCC and holds promise for guiding individualized therapies. Our research currently represents preliminary findings, and future work will require expanding the sample size and external validation for further investigation.