A Machine Learning Model Based on Health Records for Predicting Recurrence After Microwave Ablation of Hepatocellular Carcinoma

Lay Summary

Microwave ablation (MWA) is an effective treatment for patients with early-stage hepatocellular carcinoma (HCC). However, in fact, the early hepatic recurrence rate including local recurrence and hepatic distant recurrence can reach to approximately 30%. Moreover, 5- or 10-mm ablative margin is always related to local recurrence, which attract the attention of scholars. Because of the lack of highly reliable model that predicts post-ablation recurrence, we developed an artificial intelligence system based on machine learning (ML) models using large clinical datasets extracted directly from electronic health records to predict recurrence and provide guidance for MWA. In particular, we used the SHapley Additive exPlanations and local interpretable model-agnostic explanations algorithms to realize the interpretability of the ML model. This ML system can improve the likely success of the MWA treatment and guide clinical management after thermal ablation of HCC.

Introduction

Hepatocellular carcinoma (HCC), the fifth most common malignancy, results from hepatitis viral infections and causes considerable mortality and morbidity in China.^1–3 Locoregional ablation (LA) is recommended as a first-line treatment in patients with early-stage HCC, as well for surgical resection (SR) and liver transplantation (LT),^4–6 according to multiple international guidelines. In addition, LA as a primary and alternative treatment is also widely applied in patients with HCC who are ineligible for surgery, and is used as bridge to LT,^7,8 with the advantage of less trauma, better repeatability, fewer complications, and better cost-effectiveness.

Microwave ablation (MWA) is an effective LA technology owing to the higher intratumoral temperature and a more massive region of cell necrosis, with a comparable 5-year overall survival (OS) for early-stage HCC, similar to SR,^9,10 MWA is a curative option for small HCC lesions, but the post-ablation recurrence occurred commonly by the progression of residual de novo carcinoma and dissemination of microscopic lesions. Unfortunately, the approximately 50% of 5-years intrahepatic recurrence (local recurrence and distant recurrence) rate after MWA remains a great challenge in clinical practice. Especially, early recurrence (ER) (considered generally less than 2 year) may cause poor prognosis. Among them, while local recurrence rates are low in experienced hands the rate is variable, being able to predict who is likely to have local recurrence could allow for improved treatment options. For example, post-ablation recurrence risk stratification can help interventional radiologists design the ablation strategy and post-ablation medication (eg, preventive transcatheter arterial chemoembolization [TACE] or multi-targeted tyrosine kinase inhibitors [TKIs]).¹¹

A large amount of clinical data can be obtained easily from an electronic medical record system. However, these data interfere with each other commonly and weaken the predictive power of traditional statistical methods (eg, Cox regression and logistic regression [LR]) because of their poor calculation ability to analyze multidimensional clinical features. It is difficult to accurately identify the optimal biological indicators for recurrence. Machine learning (ML) is a branch of artificial intelligence that employs statistical, probabilistic, and optimization techniques to train a machine how to learn.^12,13 ML algorithms can learn from clinical data, identify patterns, and make decisions with minimal human intervention by automating analytical model building, which has been used to calculate the probability of the future outcome of a specific individual so as to obtain risk prediction models.

In this study, we developed and validated multiple ML models using clinical data extracted directly from electronic health records to predict ER risk after MWA for early-stage HCC, with the aim of building a favorable method to provide guidance for MWA indication selection and HCC preoperative planning.

Materials and Methods

Study Design and Patient Data

This retrospective, multi-center study protocol was approved by the Ethics Committee of all participating institutions (Chinese PLA General Hospital, The Third Affiliated Hospital of Harbin Medical University, The Second Hospital of Nanjing, and The Tianyou Hospital of Wuhan University of Science and Technology).Because this is a retrospective study, the requirement for written informed consent was waived after obtained the approval of each ethics committee. All participants in this study promised to maintain the confidentiality of all patient data and strictly abide by the Declaration of Helsinki.

Between August 2005 and December 2019, a total of 2423 patients with HCC who subsequently underwent ultrasound-guided percutaneous microwave ablation (US-PMWA) were taken from a medical database at four high-volume medical centers in China and reviewed. The follow-up duration was terminated in July 2020. HCC was diagnosed according to the guidelines of the European Association for the Study of the Liver (EASL) and the American Association for the Study of Liver Diseases (AASLD).^14,15 Details of patient recruitment are outlined according to the following inclusion criteria: a) patients with HCC who achieved complete ablation; b) patients with an Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1; c) patients with Child–Turcotte–Pugh (CTP) grades A or B; d) patients with a single tumor ≤ 5 cm or a maximum diameter of tumors ≤ 3 cm when the number of tumors ≤ 3; e) patients without major vascular infiltration or extrahepatic metastasis. Exclusion criteria: a) patients who had HCC combined with other malignancies; b) patients with incomplete ablation; c) patients with previous treatment before MWA, excluding SR and LT; d) patients lost to follow-up > 6 months. Finally, the data of 1574 patients with HCC (307 females and 1267 males; mean age, 58.4 ± 11.2 years) with 2100 HCC lesions (mean diameter, 2.4 ± 0.9 cm) in medical records were screened for eligibility. Figure 1 shows the patient enrollment pathway and the flowchart of building the ML model based on clinical text data. The US-PMWA procedure, post-ablation assessment, and follow-up protocol are described in detail in Supplementary Methods 1.1 and 1.2.

Figure 1 The patient enrollment pathway and the flowchart of building the ML model.

We developed and validated the predictive models for recurrence to confirm the stability and repeatability of the ML models applied in the study. Then, 1069 patients were assigned to the training set, and 268 were assigned to the internal validation set based on a 4:1 ratio at hospital 1 (Chinese PLA General Hospital). Moreover, 237 patients with HCC from the other three hospitals (The Third Affiliated Hospital of Harbin Medical University, The Second Hospital of Nanjing, and The Tianyou Hospital of Wuhan University of Science and Technology) were assigned to the external validation set during the same period.

Variable Collection and Feature Selection

The variable data with a missing ratio greater than 20% were excluded in our study. A total of 36 variables were collected to develop the predictive models. First, 36 variables (Table S1) were analyzed by a univariate analysis using LR. Subsequently, the variables with significant differences were used to automatically adjust the number of variables through recursive feature elimination with cross-validation (RFECV) to obtain the best feature combination.¹⁶ RFECV includes two parts: elimination of recursive features and cross-validation. Given an external estimator, recursive feature elimination (RFE) selects features by recursively considering smaller and smaller sets of features. First, the estimator is trained on the initial set of features, and the initial importance of each feature is obtained either. Then, the least important features are pruned from the current set of features. The procedure is recursively repeated on the pruned set until the desired number of features to select is eventually reached. Cross-validation guarantees the reliability of each step of pruning.

Development and Validation of ML Predictive Models

In this study, one conventional statistical method, LR, and three conventional ML classification algorithms—random forest (RF), support vector machine (SVM), and eXtreme Gradient Boosting (XGBoost)—were used to develop and validate the predictive models.^17,18 These models underwent continuous parameter optimization to compare the fitting effects and predictive ability of each model. For each base learner, the optimized XGBoost algorithm randomly samples 50% of the data and 80% of the features for training. Each iteration selects the feature with the largest current information gain as the node. When the predefined threshold is reached, this base learner is accomplished. Given a learning rate of 0.01, the predictive model completes the construction of 901 base learners, each of which is fitted based on the residual of the previous one. That is, we calculate the loss function of the current prediction and the actual value and minimize the Taylor expansion of the second order of the loss function. To raise the ability of model’s generalization, XGBoost limits the maximum tree depth to 2 and optimizes the model’s weight parameters in L1 and L2 regularization terms.

(1)

The other three predictive models are introduced in detail in Supplementary Methods 1.3 and.Figures S1–S3.

The validation queue data were used to evaluate the prediction performance of the model and calculate the area under receiver operator characteristic (AUC). A calibration curve was drawn, the calibration degree of the model was measured, and the degree of consistency between the predicted risk and the actual risk of the model was evaluated.

Interpretation Methods of the Predictive Models

Advanced ML models are usually black boxes. Although these models retain good accuracy, such metrics can be misleading. In this study, we used the SHAP and LIME algorithms as interpretation algorithms of the ML black box model.^19–21

The SHAP algorithm is a game theoretical approach that explains the output of any ML model. It connects optimal credit allocation with local explanations using the classic Shapley values from game theory. The algorithm calculates the marginal contribution of a feature when it is added to the model and then considers whether the factor is different in all factor sequences. The marginal contribution fully explains the influence of all factors included in the model for prediction and distinguishes the attributes of the factors (risk factors/protective factors).

The LIME algorithm is used to locally approximate the black box model by giving weights to the disturbance input such that the observation model gives the basis for interpretation of the sample prediction results. In the present study, we randomly sampled the validation set and used the LIME algorithm to fit the model’s predictive behavior in order to verify the rationality of the model’s basis for predicting results. This can greatly improve the credibility of human-computer interaction.

Statistical Analysis

Continuous variables were given as a median with interquartile range (IQR) and compared by the Kruskal–Wallis test. Categorical variables were presented as frequencies with percentages and compared by the chi-squared test. Survival curves were calculated using the Kaplan–Meier method and compared with the Log rank test. Univariate Cox regression analyses were applied to calculate the hazard ratios (HRs) and corresponding 95% confidence interval (CI) of variables and identify independent prognostic factors. The discrimination, predictive accuracy, and calibration of the model were assessed by AUC and calibration curve. Given that a quick evaluation of recurrence risk levels with cutoff values might be useful in routine clinical practice, based on the optimal cutoff value thresholds for the scores determined by the maximally selected log-rank statistics, we stratified patients into low-risk, middle-risk, and high-risk subgroups. All statistical analyses were conducted using R version 4.0.4. A two-tailed p-value of less than 0.05 was considered as statistically significant.

Results

Patient Characteristics

Over a 10-year follow-up period, the median follow-up durations were 25.7 months (IQR, 6.3, 160.7 months), 26.9 months (IQR, 6.5, 159.2 months), and 23.7 months (IQR, 7.2, 124.8 months) in the training set, internal validation set, and external validation set, respectively, with no significant differences (P = 0.139). The baseline characteristics of patients in the three datasets are outlined in Table 1. The 512/1574 (32.5%) recurrence occurred 2 years after MWA. All HCC patients underwent complete ablation, and 35.1% (375/1069), 30.2% (81/268) and 28.7% (68/237) patients experienced recurrence at 2 years after MWA in the training set, internal validation set, and external validation set, respectively. Among them, local recurrence rate was 9.2% (98/1069), 7.1% (19/268) and 8.0% (19/237), respectively.

Table 1 The Baseline Characteristics of Patients in Three Datasets

Variables Selected

By 36 risk factors for correlation analysis, the correlation coefficient matrix heat map of the features (Figure 2) showed that the top five features correlated negatively with the outcomes were platelet (PLT), ablation time, prothrombin activity (PTA), cholinesterase (CHE), and α-fetoprotein (AFP). The top five characteristics correlated positively with outcomes were number of tumors, Child grade, gender, etiology, and total bilirubin (Tbil). Given 36 variables might interfere with each other, thereby reducing the final predictive ability of the ML models, we used the univariate Cox regression to screen out 17 risk factors with statistical differences. The results from univariate Cox analysis for recurrence outcome are summarized in Table 2. We input these 17 factors into the ML algorithms to develop and validate the predictive model for recurrence after MWA.

Table 2 The Results from Univariate Analysis for Recurrence Outcome

Figure 2 The correlation coefficient matrix heat map of all 36 variables. (A) The Correlation analysis of 18 variables including age; (B) the correlation analysis of the other 18 variables including RBC.

Building and Evaluation of Four Predictive Models

The four ML models and the codes and construction processes are detailed in Table S2. Moreover, we list the 1-, and 2-year AUC values of the three datasets of the different applied predictive models in Table 3. Censoring until 2 years after MWA, the AUC values of the XGBoost model were 0.74 (95% confidence interval [CI], 0.71–0.77), 0.67 (95% CI, 0.59–0.71), and 0.73 (95% CI, 0.66–0.79) for the training set, internal validation set, and external validation set, respectively, which had the best predictive ability among three ML models and significantly outperformed the LR model (P < 0.001) (Figure 3). The robustness of the ML predictive model was further explored in a subgroup analysis. The screening results of RFECV showed (Figure S4) that when the 9 variables were retained, XGBoost exhibited the best predictive accuracy and discrimination. XGBoost sorted the nine variables according to feature values in advance and stored them as a block structure, which is convenient for quick query every time a node is split. The calibration plots for the recurrence outcome had good predictive value and were validated well in the internal validation set (Figure S5A) and the external validation set (Figure S5B). The XGBoost model continued to show good predictive accuracy and discrimination in different subgroups of patients stratified according to the nine variables (Table S3).

Table 3 Summary of Predictive Performance of Multiple Models

Figure 3 The ROC curves of four models in three datasets, (A) the training set; (B) the internal validation set; (C) the external validation set.

Risk Stratification

We identified two optimal cutoff values (34.12 and 73.35) of the XGBoost model to stratify patients into low-risk, middle-risk, and high-risk groups in the training cohorts. The cumulative 1-, 2-, 3-, 4-, and 5-year recurrence-free survival (RFS) rates were 55%, 42%, 13%, 7%, and 3% in the low-risk group, 74%, 62%, 34%, 25%, and 19% in the middle-risk group, and 97%, 85%, 59%, 48%, and 30% in the high-risk group, respectively, which had a significant statistical difference among the three groups (P < 0.001) (Figure 4A). Both in the internal validation (cutoff value, 62.31 and 112.21) and external validation cohort (cutoff value 60.98 and 117.30), the cumulative 1-, 2-, 3-, 4-, and 5-year RFS rates among the high-risk, middle-risk, and low-risk groups also had a significant statistical difference (both, P < 0.001) (Figure 4B and C). An AI system for recurrence prediction after MWA accessed online is available at http://114.251.235.51:8001/ (Figure S6).

Figure 4 The risk stratification of the XGBoost Model. (A) The training set; (B) the internal validation set; (C) the external validation set.

Interpretation of the XGBoost Model

We used the SHAP and LIME algorithms to visualize the influence of each variable in the XGBoost model on the outcome variable in Figure 5. Based on the SHAP algorithm, the influence of nine variables on the outcome in the XGBoost model is shown in Figure S7. The feature-importance ranking was as follows (Figure S8): number of tumors, PLT, AFP, comorbidity score (CS), WBC, CHE, PT, NEU, and etiology. The number of tumors, PLT, and CS had approximately linear effects on the outcome; the number of tumors and CS were positively correlated with the outcome. It is worth noting that the CS stabilized after five points; PLT was negatively correlated with the outcome; when the value was greater than 150, the impact became stable The effects of AFP, WBC, and CHE on the outcome all had peaks and troughs, and beyond the SHAP value, gradually stabilized. The influence of PT and NEU on the outcome was slightly more complicated. The SHAP value of etiology was near 0, which had little effect on the outcome. The LIME algorithm explained the predictions of the XGBoost model on each sample and summarized the predictions of the model in the training set, internal validation set, and external test set, showing the distribution of four types of results: true positive, true negative, false positive, and false negative. Overall, the number of samples predicted to be true positive and negative both accounted for 70%, indicating that the performance of the XGBoost prediction model is relatively stable In addition, we randomly selected four types of samples (true positive, true negative, false positive, false negative) and visualized them separately. When the prediction probability of the model is far from 0.5, the physician only needs to judge whether the prediction result is reasonable based on the model prediction basis given by LIME. If the model prediction basis is unreasonable, the result will not be adopted; when the predicted probability of the model is close to 0.5, such samples are usually clinically difficult to judge.

Figure 5 Interpretation of the XGBoost model. (A–C) SHAP algorithms to visualize the influence of each variable in the XGBoost model; (D and E) LIME algorithms to visualize the influence of each variable in the XGBoost model.

Discussion

In this study, we developed and validated ML models using clinical data extracted directly from electronic health records to predict individual ER outcomes after MWA based on a long-term follow-up cohort comprising 1574 patients with early-stage HCC who were recommended as MWA candidates. These analysis results suggest that the XGBoost model has more favorable performance and discrimination than the other three predictive ML models, which identify three ER risk strata and help further clinicians analyze HCC patient survival benefit.

A real world study based on a consecutive multi-center cohort was performed, with a large sample size from four high-volume medical centers by setting three cohorts, which ensured the stability of the ML algorithm.The clinical data available on biomarkers for prediction of ER were obtained. Previously identified risk factors for hepatic ER after LA mainly included tumor size, number, and AFP level.^21–23 However, these clinical variables that predict post-ablation recurrence remain controversial because of individual differences in patients and tumor heterogeneity. Moreover, traditional staging systems, such as tumor-node-metastasis (TNM), Barcelona Clinic Liver Cancer (BCLC) grade, and Okuda score, were used to try to predict HCC outcomes,²⁴ but they had insufficient predictive ability on postoperative recurrence. Aggressive pathological factors, including microvascular invasion (MVI), microsatellite lesions, and pathological grade, could provide valuable information on tumor invasion ability, which is associated with early recurrence after SR.^25,26 However, preoperative diagnosis mainly relies on biopsy during MWA procedure, and complete pathological tissue cannot be obtained, so it was difficult to collect effective pathological information for MWA.

The imaging feature commonly reflects the aggressive characteristics of a tumor; therefore, several radiomics models were built in prediction of HCC recurrence in the last 5 years. Among them, two radiomic predictive models of post-ablation recurrence were developed. Yuan et al reported that a radiomics nomogram based on a contrast-enhanced CT was developed to predict the early recurrence of HCC cases eligible for curative ablation, and the C-index of the radiomics model and radiomics, combined with a clinical model, were 0.72 and 0.79, respectively.²⁷ Shen et al reported that a radiomics model based on a post-treatment CT image was built to predict the early recurrence after ablation, and the AUC value was 0.89.²⁸ As the abovementioned radiomics model takes 200–300 cases, it is not enough to provide big-data information and verification, causing inadequate stability and robustness. In addition, feature extraction is complicated and dependent on the model, which is difficult to apply and lacks image visualization. As a cost-effective and non-invasive manner, clinical text data are a better choice. Therefore, we decided to establish a ML model for ER risk prediction based on clinical text data.

The predictive models based on the XGBoost algorithm, with advantages including efficient computing power and the ability to handle missing values, were built and tested via two validation sets, which further ensured the stability. In our study, the XGBoost model could reduce eigenvalues from a great number of electronic health records compared with the other models. In terms of missing value processing, the XGBoost algorithm could handle missing data automatically by adding a default direction based on learning during the training procedure in each tree node. When a value was missing in the validation data, the instance was classified into the default direction. In this study, 36 variables were input into the XGBoost model, and 9 variables were output to develop the optimal predictive performance in this study, with AUC values of 0.74 and 0.73 in the training and validation datasets, respectively. ML has remarkable computing power and will not be affected by large samples or multi-dimensional data. Development of the predictive model using ML to help physicians formulate follow-up treatment plans in clinical practice has been increasing.^29,30 For example, a precise prognostic assessment of COVID-19 in an ICU was built by ML methods that can accurately screen out high-risk groups and guide physicians in effectively prevention and treatment.³¹ In addition, an ML model to predict kidney outcomes in IgA nephropathy was developed in multiple centers, which can help physicians in rational use of drug according to these predictive results.³² Similarly, our XGBoost algorithm model showed the ability to generalize across populations and screening protocols. This suggests that in future clinical deployments, the XGBoost model offers strong baseline performance but may benefit from fine-tuning with local data. Interventional radiologists can use preventive TACE or neoadjuvant chemotherapy treatment (NCT) for high-risk recurrence HCC cases based on ML model prediction and can reduce the recurrence risk after MWA further, thereby “nipping it in the bud”, as the saying goes.

In this study, we used the SHAP and ITME algorithms to explain the XGBoost model because the black boxes used to understand the principles behind ML model could be accessed online conveniently. After full explanation by the SHAP and LIME algorithms, the XGBoost model showed accurate and stable prediction ability in recurrence. However, parts of clinical variable data that should not be used may have accidentally leaked into the training data. The SHAP algorithm calculates the marginal contribution of a feature when it is added to the model and then considers whether the variables are different in all variable sequences. The marginal contribution fully explains the influence of all variables included in the model prediction and distinguishes the attributes of the factors (risk/protective factors). Moreover, we randomly sampled the test set and used the LIME algorithm to fit the model’s predictive behavior to verify the rationality of the model’s basis for predicting results. This approach can greatly improve the trust value of human–computer interaction.

There are some limitations to our study. First, the risk of selection bias is unavoidable in observational studies. However, this risk was minimized by the inclusion of all consecutive patients and a large cohort of early HCC cases for MWA to date. Second, the cohort was not from a prospective therapeutic trial because of the nature of retrospective research; therefore, part of the data was lacking. Fortunately, the XGBoost model had the ability to deal with missing values in this study. Third, the prediction model based on clinical text prediction is effective and convenient, but if it is combined with imaging information, more accurate results may be obtained; a prospective randomized trial is underway to confirm this hypothesis.

In conclusion, with an easy-to-use XGBoost model consisting of nine clinical variables, the model exhibited adequate performance and individualized predictive ability that could stratify patients into three ER risk groups with a referable cumulative recurrence rate. Therefore, the visual XGBoost model may help physicians with decision-making before MWA for HCC in clinical practice and trials.