A Machine Learning Guided System for Therapeutic Decision-Making in Intra-Arterial Therapy of Hepatocellular Carcinoma

Introduction

Hepatocellular carcinoma (HCC) constitutes roughly 90% of all primary liver malignancies, ranking as the sixth most prevalent cancer diagnosis and the third leading cause of cancer-associated deaths globally.^1,2 For individuals diagnosed with unresectable HCC (uHCC), intra-arterial therapies (IATs), encompassing hepatic arterial infusion chemotherapy (HAIC) and transarterial chemoembolization (TACE), are established as standard therapeutic approaches in clinical practice. International clinical guidelines recommend TACE as the first-line intervention for intermediate-stage HCC,^3,4 whereas HAIC has become an essential treatment option for intermediate-advanced HCC in Asian populations5. In recent years, growing evidence has shown that combining IATs with molecular-targeted agents (MATs) and immune checkpoint inhibitors (ICIs) produces favorable therapeutic effects characterized by profound and long-lasting efficacy.^5,6 Furthermore, sequential local-regional therapies—such as surgical resection (SR), liver transplantation, and thermal ablation (TA)—administered following conversion therapy based on IAT-inclusive comprehensive treatment have substantially enhanced the long-term survival of uHCC patients.^7–10

Numerous previous studies have identified a variety of clinical factors closely linked to HCC prognosis, and multiple predictive models have been constructed based on these variables to assist clinicians in precisely assessing patient outcomes and conducting risk stratification.^11–14 However, The aforementioned model is limited by factors such as data deficiency and insufficient computing power, resulting in slightly inadequate predictive ability for the prognosis of IAT for uHCC. On the contrast, machine learning (ML) emerges is a promising technology that utilizes statistical, probabilistic, and optimization algorithms for model training, offering unique advantages such as standardized handling of large-scale multicenter data, imputation of missing values, and high computational efficiency.^15–19 By automating analytical models, ML algorithms can detect patterns and support decision-making with minimal human involvement, and they have been increasingly utilized in the development of recurrence prediction models within the field of oncology.

HCC displays substantial tumor heterogeneity at the histological, genomic,²⁰ and molecular levels. Additionally, therapeutic decisions vary depending on tumor response to IAT and disease progression, resulting in marked interindividual differences in the prognostic outcomes of uHCC patients. Consequently, accurate assessment of dynamic tumor progression during IAT is essential for improving long-term survival results. However, most existing predictive models are constrained by a limited range of clinical factors, small sample sizes for model development, inconsistent pathological evaluation criteria, and insufficient clinical applicability. To date, there remains an absence of a noninvasive, convenient, and accurate model to guide clinicians in IAT decision-making.

We hypothesize that ML models incorporating postoperative variables will outperform preoperative only models and existing staging systems in predicting 5-years mortality. The present study aimed to develop and validate an ML model based on clinical data from uHCC patients who received IAT, with the goal of providing guidance for clinicians in therapeutic decision-making and surveillance strategy selection.

Methods

This retrospective, multi-center study was carried out in line with the principles outlined in the 1975 Declaration of Helsinki. The study protocol was approved by the Institutional Review Board of XXX (No. XX). Owing to the retrospective nature of the study, written informed consent was exempted. To preserve patient privacy and data confidentiality, patient data have been deidentified before analysis. Clinical data were extracted from an internally developed database, and the study was reported in adherence to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) guideline.²¹

Study Design

HCC diagnosis was confirmed in accordance with the guidelines established by the European Association for the Study of Liver (EASL) and the American Association for the Study of Liver Disease (AASLD).^22,23 From April 2008 to October 2022, clinical data from a total of 7829 uHCC patients who underwent initial IAT were retrospectively collected from 15 tertiary hospitals in China (eTable 1). We performed TACE, HAIC, and combined targeted immunotherapy at each participating hospital, with the specific steps. The detailed descriptions of IAT procedures, criteria for discontinuing the protocol, and combination with targeted-immunotherapy are available in the Supplemental information E 1.1–1.3. The inclusion criteria were as follows: (a) age between 18 and 75 years; (b) Eastern Cooperative Oncology Group (ECOG) performance status < 2; (c) Child-Turcotte-Pugh (CTP) class A or B liver function. The exclusion criteria included: (a) prior treatment before IAT; (b) concurrent other malignancies; (c) loss to follow-up for more than 6 months. The patient enrollment pathway is illustrated in Figure 1A.

Figure 1 Enrollment pathway of unresectable HCC patients who underwent various IAT schemes. (A) HCC patients were enrolled to build an MLGS model; (B) both the preoperative and postoperative variables were input in the MLGS-pos model according to tumor progression and therapeutic procotols and the death risk score was calculated; (C) the clinical decision-making for IATIs based on death risk change from MLGS-pre model to MLGS-pos model. Abbreviations: HCC, hepatocellular carcinoma; IAT, intra-arterial interventional therapy; ML, machine learning.

Follow-Up Protocol and Endpoint Definition

All patients underwent regular follow-up after IAT, with serum alpha-fetoprotein (AFP) testing and imaging examinations conducted 1 month after IAT and every 3 to 6 months thereafter until death or the final follow-up date (October 31, 2025). The objective response (OR) to the first IAT was assessed by two radiologists in accordance with the modified Response Evaluation Criteria in Solid Tumours (mRECIST).²⁴ Early recurrence (ER) was defined as the presence of suspected lesions exhibiting arterial phase enhancement and portal venous phase washout on contrast-enhanced imaging within 1 year^25. Overall survival (OS) was defined as the time from the date of initial IAT to the date of death or the final follow-up. The observation window for clinical indicators was set at the time of initial IAT, and the prediction window was defined as 1, 3, and 5 years after initial IAT. For patients with censored data within 5 years, their outcomes were classified as “nonevent” in all models.

Machine Learning-Guided System Construction

Data Collection

A total of 30 preoperative and 6 postoperative variables were collected for the development of ML models. The abbreviations and detailed descriptions of these 36 clinical variables are presented in the eTable 2. Five tumor staging systems—Barcelona Clinic Liver Cancer (BCLC), American Joint Committee on Cancer (AJCC) tumor-node-metastasis (TNM), China Clinic Liver Cancer Staging (CNLC), Japan Society of Hepatology (JSH), and EASL—and several clinical evaluation scores, including albumin-bilirubin (ALBI) grade, platelet-albumin-bilirubin (PALBI), neutrophil-to-lymphocyte ratio (NLR), platelet-to-lymphocyte ratio (PLR), and model for end-stage liver disease (MELD), were also collected to compare the predictive performance of the ML models.

Selection of Clinical Variables

Variables with a missing data rate exceeding 20% (including pathological variations) were excluded from the model. Five representative supervised ML algorithms—XGBoost, CatBoost, GBDT, LGBM, and random forest (RF)—were applied and compared in this study (Supplemental Information E1.4). The ML model with the optimal discriminative capability was utilized to develop a simplified restricted system for guiding IAT scheme selection (hereafter referred to as the ML-guided system [MLGS]), which integrated preoperative and postoperative clinical information. The code parameters for ML model development are provided in the eTables 3 and 4. To optimize model performance and evaluate its generalization capability, a random hyperparameter search combined with 5-fold cross-validation was employed to identify the optimal hyperparameter set. Subsequently, the final model was trained utilizing the entire training dataset based on the best-performing hyperparameters identified. Feature importance was then analyzed from this final model. In the cohort of the Sun Yat-sen University Cancer Center, HCC patients were divided into training and internal testing datasets at a ratio of 7:3, and the training datasets was optimized using 5-fold cross-validation. During the same period, data from another 14 tertiary hospitals were used as an external testing datasets.

Dynamic Prognostic Risk Based on the ML Model

In the current study, the MLGS was employed to assess the dynamic prognostic risk of HCC patients receiving IAT and to guide clinicians in the timely adjustment of treatment and follow-up monitoring schemes by stratifying the 5-year mortality risk as a long-term outcome after IAT. The guidance process of the MLGS primarily comprises two components: first, the preoperative ML model assists clinicians and patients in evaluating preoperatively the effectiveness of IAT for uHCC based on significant mortality risk stratification; second, the postoperative ML model helps clinicians determine whether to adjust sequential treatment schemes during follow-up based on IAT response and tumor recurrence.

The preoperative ML model assigns a clear death risk score to uHCC patients scheduled to receive IAT. However, when postoperative variables—such as IAT response, conversion therapy modalities, and tumor recurrence—are sequentially input into the preoperative ML model, the risk score of uHCC patients changes accordingly (Figure 1B). Concurrently, the original death risk stratification may shift (eg, from low risk to high risk), enabling clinicians to adjust therapeutic and monitoring strategies during this dynamic process (Figure 1C).

Statistical Analysis

All statistical analyses were performed using R software version 3.6.3 (http://www.r-project.org/). The performance of the ML models was evaluated using the area under the receiver operating characteristic curve (AUC), sensitivity (SEN), specificity (SPE), positive predictive value (PPV), negative predictive value (NPV), and F1 score. To improve the applicability of the predictive model by simplifying input variables, a stepwise Cox proportional hazards model was built based on the top important variables derived from the optimal ML model.

The discriminative ability of the models and different tumor staging systems was measured using the concordance index (C-index) and compared via the DeLong test. Time-dependent receiver operating characteristic curves and AUC values at each time point from 12 to 60 months were used to assess OS prediction accuracy at different time points. The prediction error of the models was evaluated using the integrated Brier score (IBS). To interpret the causal relationship of the MLGS, the Shapley Additive exPlanations (SHAP) method was used to explain the prediction results of each ML model.^26,27

Survival curves were generated using the Kaplan-Meier method and compared using the Log rank test. All statistical tests were two-sided, and a P value < 0.05 was considered statistically significant.

Results

Baseline Characteristics

Finally, a total of 5646 eligible patients (4930 male, 716 female; median age, 57 years, IQR: 37–63 years) were included and divided into a derivation cohort consisted of training datasets (TD) (51%, N = 3367) and internal testing datasets (ITD) (13%, N = 1130) at the Sun Yat-sen University Cancer Center and an external testing datasets (ETD) (36%, N = 1129) at the another 14 hospitals. Overall, the mortality rate among patients who received IAT prior to the final follow-up endpoint was 61.5% (3360 out of 5646 patients). Among all deaths, 61.6% (2018/3367) occurred death in the TD, 62.0% (683/1130) in the ITD, and 60.7% (659/1129) in the ETD, with no statistically significant differences between these rates (P = 0.913). Table 1 presents a comparison of baseline characteristics across the three datasets. All preoperative variables were well-balanced and homogeneous among the datasets (all P > 0.05). A feature heatmap was generated through correlation analysis of 35 risk factors, resulting in a correlation coefficient matrix (eFigure 1). In the ITD and ETD cohorts, OR to the first IAT, sequential local treatment, targeted-immunotherapy, lymphocyte count, and creatinine level were strongly negatively correlated with patient outcomes. Additionally, we compared the performance of the five ML models in predicting 1- and 3-year mortality across the three datasets (eTables 5 and 6), and the predictive capability of the models for 1- and 3-year mortality was consistent with that for 5-year mortality.

Table 1 The Baseline Characteristics of uHCC Patients Underwent IATs in Three Datasets

Comparison of the Predictive Performance of the ML Models

Table 2 summarizes the number of predictors and the AUC, SEN, SPE, PPV, NPV, and F1 score of each ML model based on clinical variables for predicting 5-year mortality. For preoperative ML models, the CatBoost model achieved the optimal predictive performance, with an AUC of 0.777 (95% CI, 0.712–0.815) in the TD, 0.743 (95% CI, 0.705–0.811) in the ITD, and 0.735 (95% CI, 0.68–0.782) in the ETD when trained using the 23 most important variables identified by the CatBoost importance score. Among postoperative ML models, the XGBoost model demonstrated the best predictive performance. When trained using the 20 most important variables identified by the XGBoost importance score, the XGBoost model achieved an AUC of 0.904 (95% CI, 0.876–0.963) in the TD, 0.886 (95% CI, 0.845–0.927) in the ITD, and 0.862 (95% CI, 0.818–0.932) in the ETD (Figure 2A).

Table 2 The Performance Comparison of Five ML Models Based on Preoperative and Postoperative Variables

Figure 2 The death risk possibility of prediction based on ML models were explained using SHAP algorithm. (A) the XGBoost model trained using the 20 most important variables measured by the AUC values; (B) SHAP summary plots show the contributions of the feature to the ML model; (C) SHAP bar chart represented this clinical variables towards the 5-years death rate with a probability of specific percentage; (D) a heatmap of interpretation for outcomes of uHCC patients underwent IAT; (E) the mean SHAP value were compared between three death risk subgroups.

Interpretation Methods for the ML Models

As depicted in the SHAP summary plots (Figure 2B), the contribution of each feature to the ML model was evaluated using average SHAP values and presented in descending order. Additionally, local explanation was employed to analyze how a specific prediction was made for an individual by incorporating personalized input data. Figure 2C illustrates the impact of clinical variables on the 5-year mortality rate with a specific probability. Furthermore, a heatmap interpreting the outcomes of uHCC patients who received IAT in the ETD is shown in Figure 2D, and the mean SHAP values were compared among the three mortality risk subgroups (Figure 2E). Our results demonstrated that ER was most strongly associated with poor prognosis (eFigure 2). Non-OR to the first IAT was the second most significant risk factor for mortality and was linked to a higher risk of poor prognosis than OR (eFigure 3). Patients who received IAT alone had a significantly higher risk of adverse outcomes than those who received IAT combined with sequential local treatment (eFigure 4) or targeted-immunotherapy (eFigure 5). Patients with an AFP concentration > 400 ng/mL had a higher risk of poor prognosis than those with an AFP concentration ≤ 400 ng/mL (eFigure 6). Patients with a tumor diameter > 10 cm or more than 3 lesions had a poorer prognosis than those with a tumor diameter ≤ 10 cm or 1–3 lesions, respectively (eFigures 7 and 8). The relationship between laboratory findings (including creatinine) and SHAP values exhibited an S-shaped curve with distinct turning points (eFigure 9). The HAIC protocol without metastasis was associated with better OS than the TACE protocol or HAIC with metastasis (eFigures 10 and 11).

Prognostic Risk Stratification of the MLGS

Stratifying prognoses for specific patient populations is vital for establishing criteria for their inclusion in clinical trials and therapeutic interventions (Figure 3A). Based on the cutoff values (22.34, 76.25) from both the preoperative MLGS (MLGS-pre) and postoperative MLGS (MLGS-post) models, HCC patients were divided into three risk groups: low-risk, middle-risk, and high-risk. The person-months in each risk group are shown in Figure 3B, indicating that patients in the low-risk group had better OS than those in the high-risk group. According to the MLGS-pre model, the low-risk group in the TD had significantly higher 1-, 3-, and 5-year cumulative OS rates (87.8%, 62.1%, and 49.1%, respectively) compared to the middle-risk group (63.2%, 33.4%, and 25.4%, respectively) and the high-risk group (35.9%, 13.0%, and 8.9%, respectively) (P < 0.001). Similar results were observed in the ITD and ETD cohorts using the preoperative model (Figure 3C–E). According to the MLGS-post model, the 1-, 3-, and 5-year cumulative OS rates in the low-risk group in the TD were 98.3%, 81.7%, and 65.8%, respectively, while those in the middle-risk group were 68.0%, 22.1%, and 13.5%, respectively—both significantly higher than those in the high-risk group (20.5%, 1.7%, and 0%) (P < 0.001). Similar results were obtained in the ITD and ETD cohorts using the postoperative model (Figure 3F–H).

Figure 3 Risk strata for predicting the prognosis of HCC patients who underwent IATs. (A) Segmenting prognoses for particular patient populations were shown using line chart; (B). The uHCC patients were categorized into three risk group and the person-months were showed in different risk group; the survival outcomes of HCC patients received the IAT scheme between three risk subgroups based on MLGS-pre model (C) OS, (D) PFS, and (E) EPFS the survival outcomes of HCC patients received the IAT scheme between three risk subgroups based on MLGS-post model (F) OS, (G) PFS, and (H) EPFS. Model discrimination measured with time-dependent areas under the receiver operating characteristic curves at various time points for the training (I), internal testing (J) and external testing (K) cohorts. The integrated Brier score of models for the training (L), internal testing (M) and external testing (N) cohorts.

Comparison of the Predictive Accuracy Between MLGS, Tumor Stages, and Biomarkers

To evaluate the discriminative capability of the ML models, we compared their prognostic performance with that of ALBI grade, PALBI grade, PLR, NLR, MELD grade, and other staging systems (BCLC, JSH, AJCC TNM, EASL, and CNLC). Time-dependent AUC analysis revealed that the MLGS had superior OS prediction performance compared to the staging systems at various time points in both study cohorts (all Log rank test, P < 0.001) (Figures 3I–K). The prediction error curves of all models and staging systems are shown, and the IBS values for MLGS-post were 0.17 in the TD, 0.14 in the ITD, and 0.22 in the ETD (Figures 3L–N). The bootstrapped calibration curves plotted with 5-years mortality was well matched with the idealized 45°line for the MLGS in the three datasets (eFigure 12). Decision curve analysis (DCA) graphically indicated that the MLGS provided a larger benefit across the range of reasonable threshold probabilities in the three datasets (eFigure 12).

Clinical Value of the MLGS

The MLGS developed in this study comprises two ML models: the preoperative and postoperative models. The preoperative model provides a prognostic risk score for individual uHCC patients before IAT and classifies them into low-risk, middle-risk, and high-risk groups. Subsequently, the six postoperative variables are input into the preoperative model, and the risk coefficient changes with the input of positively and negatively correlated variables. During follow-up after IAT, inputting the time and status of tumor recurrence into the preoperative ML model leads to a significant increase in the mortality hazard. Furthermore, the IAT scheme plays a crucial role in improving survival benefits—for instance, selecting between HAIC and TACE, or combining IAT with targeted-immunotherapy and sequential local treatment. Based on the dynamic changes in mortality risk following the input of these variables, the MLGS can guide clinicians in continuously adjusting treatment schemes during the IAT process.

Discussion

IAT possesses considerable potential as an effective treatment for uHCC, with growing evidence highlighting its significant role in improving survival and quality of life for HCC patients.^4,28,29 Accurate and rational IAT decision-making is critical for achieving long-term survival benefits. Therefore, we developed the MLGS to assist clinicians in formulating comprehensive IAT plans throughout the treatment process. The MLGS can effectively classify HCC patients who received IAT into three distinct risk groups with significantly different long-term OS durations. Additionally, it can visually display the dynamic changes in mortality risk over time, taking into account variations in tumor status and treatment strategies. Thus, the MLGS may facilitate personalized decision-making and reduce future mortality rates, treatment-related side effects, and economic burden for a larger number of uHCC patients receiving IAT.

This study has several strengths and innovations. First, the MLGS was developed and externally validated using clinical data from 5646 patients (from 15 hospitals in China) and 35 available clinical variables, with a relatively long median follow-up time (4.2 years). This ensures the comprehensiveness, robustness, and accuracy of the MLGS from multiple perspectives. Second, the MLGS was developed using a wide range of variables, including demographic characteristics, tumor features, laboratory test results, and treatment schemes. Preoperative and postoperative variables were sequentially input based on the timeline before and after IAT, enabling visualization of the dynamic trajectory of mortality risk. Third, five representative ML algorithms were compared, and the XGBoost model—with the most reasonable variable input—was selected to develop the MLGS for prognostic risk stratification. Fourth, the predictive performance of the MLGS was significantly superior to that of other internationally recognized clinical and biological markers, which are considered important in clinical practice.

Given the lack of interpretability of ML processes, the SHAP method was utilized to explain the relationship between clinical variables and mortality risk. These variables mainly fell into two categories: those associated with increased mortality risk (eg, high AFP levels, non-OR to the first IAT, ER) and those associated with reduced mortality risk or improved survival benefits (eg, local treatment after IAT, combination of IAT with targeted-immunotherapy). These findings are consistent with the recent global Phase III randomized clinical trial (EMERALD-1), which demonstrated better progression-free survival with the combination of bevacizumab, durvalumab, and TACE compared to TACE monotherapy in patients with intermediate HCC.²⁹ Previous studies have also clarified that the tumor microenvironment, including proteoglycans, immune cells, and hypoxia, plays a critical role in tumor initiation and progression during the invasion-metastasis cascade.^30–32

Previous research has focused solely on decision-making for a single treatment option. For example, Liu et al constructed two models using semantic information and ultrasound image data to predict ER in HCC patients undergoing TA or SR and identify suitable candidates.³³ Ding et al developed a hybrid model to accurately predict the probability of ER after TA or SR, providing reliable evidence for optimal treatment decision-making in patients with single 3- to 5-cm HCCs.³⁴ These decision support models were individual-based and cross-stratified twice according to treatment type. In contrast, the MLGS developed in this study is based on a dynamic risk trajectory formed by the input of preoperative clinical data, postoperative tumor status, and IAT schemes. Our results showed that tumor burden, tumor immunity-related biomarkers, and liver function affect the prognosis of uHCC patients before IAT.However, with the selection of IAT modality, achievement of initial OR, and application of subsequent treatment schemes, the ability to predict mortality risk was significantly improved compared to the preoperative model. By leveraging the dynamic changes in risk scores generated by the MLGS, clinicians can more accurately formulate personalized IAT regimens.

The MLGS incorporates simple, easily accessible clinical and radiological variables that can be used to identify high-risk HCC patients. High-risk patients require prolonged and intensive monitoring due to their elevated mortality risk, along with additional systemic medications. The MLGS effectively classifies HCC patients into different subgroups based on risk levels and helps identify the most suitable candidates for IAT. In the TD, 33% of uHCC patients were classified as high-risk, with a 5-year OS rate of 0%, while the low-risk group had a 5-year OS rate of 49.1%. The MLGS is clinically significant as it enables the identification of a subset of IAT-treated patients at high risk of mortality who may benefit from targeted interventions. For example, if patients receiving HAIC are classified as high-risk by the MLGS, clinicians may modify the treatment plan to TACE or implement a proactive targeted-immunotherapy and monitoring strategy with shorter intervals.

This study possesses several inherent limitations. First, participants were enrolled across 15 Chinese medical institutions, each with distinct protocols, surgical techniques, and medication regimens for TACE and HAIC. These institutional variations may have introduced heterogeneity, potentially impacting the generalizability of the findings. Second, the pharmacological diversity in TACE and HAIC regimens across centers may have introduced confounding variables affecting the final outcomes. Furthermore, the clinical characteristics of the study cohort differ substantially from Western populations, where patients typically present with smaller tumor burdens and alcoholic cirrhosis as the primary etiology of HCC, limiting the broad applicability of our results to Western settings. Finally, given the established role of radiomics in predicting prognosis for patients undergoing IAT, future investigations should integrate radiomic features into our ML models to enhance predictive precision.

In conclusion, the MLGS accurately stratifies prognostic risk and may assist in identifying high-risk patients who could benefit from more intensive surveillance or adjunctive therapies. Prospective studies are needed to evaluate its clinical utility.