Machine Learning Models for Predicting Liver Metastasis at Diagnosis and Overall Survival in Ovarian Cancer: A SEER-Based Study

Introduction

Ovarian cancer (OC) is the third most common malignant tumor of the female reproductive system. It is reported that in 2022, a total of 324,398 new cases of OC were diagnosed globally, and 206,839 deaths were attributed to OC.¹ With the advancement of medical science and technology, the survival rate of OC patients has significantly increased, the number of OC patients with metastasis is also on the rise.² The common metastatic routes of OC include hematogenous metastasis, lymphatic metastasis, seeding metastasis and direct invasion.³ More than two-thirds of OC patients have distant metastases at the time of diagnosis.⁴ It is well known that the liver is the most common site of distant metastasis of OC, and nearly 57% of distant metastases are liver metastases.⁵ Patients with liver metastatic OC at diagnosis have a poorer prognosis, with a median survival time of only 30 months in this population.⁶ In addition, liver-related events caused by liver metastasis, such as liver failure, have a significant negative impact on the prognosis of OC patients.⁷ Therefore, identifying the presence of liver metastasis at the time of OC diagnosis, and predicting the subsequent survival of those patients, is of great clinical importance. The Tumor, Node, Metastasis (TNM) staging system and the pathological classification have always been regarded as the prognostic assessment systems for liver metastases of OC.^8–12 However, these systems lack sufficient basic demographic information and have limited variable inclusion. Some studies have found that age, race, and treatment are all predictive factors for liver metastases of ovarian cancer.¹⁰ This study aims to develop a more accurate clinical model that includes as many variables as possible.

In terms of model development, the nomogram is currently the most commonly used predictive model for liver metastases of OC, but its overall predictive performance is not high [area under the curve (AUC)=0.764].¹³ With its powerful advantages of handling massive amounts of data and achieving high-precision predictions, Machine Learning (ML) is being increasingly widely applied in the medical field.^14,15 In this retrospective study, we extracted demographic, pathological, and survival information of OC patients from the Surveillance, Epidemiology, and End Results (SEER) database. By comparing the performance of multiple ML models, we aimed to build a reliable model for (1) diagnosing the presence of liver metastasis at the time of initial diagnosis, and (2) predicting overall survival among OC patients with liver metastasis. The results are intended to provide targeted references for the treatment of OC patients with liver metastasis.

Materials and Methods

Study Population

This retrospective study used data from the SEER database. The SEER database, established by the National Cancer Institute (NCI) in 1973, is one of the most representative large-scale tumor registry databases in North America, collecting a large amount of data related to evidence-based medicine and providing systematic evidence support and invaluable first-hand information for clinicians’ evidence-based practice and collects a large amount of data related to evidence-based medicine, which provides systematic evidence support and valuable first-hand information for clinicians’ evidence-based practice and clinical research. In this study, SEER*Stat software was used to identify patients diagnosed with OC from the SEER database 2010–2020.

The study population needed to meet the following inclusion criteria: (1) diagnosed with OC (C56.9) according to the International Classification of Diseases for Oncology, 3rd Edition (ICD-O-3); (2) the age at diagnosis was between 18 and 75 years old; (3) having complete clinical, pathological and follow-up data (such as liver metastasis status, survival time). The exclusion criteria included: (1) patients with missing information on the degree of tumor differentiation; (2) insufficient follow-up time (<1 month), (3) unknown AJCC T and N stages, and AJCC T0 stage. Finally, a total of 27,065 participants were included in this study. The process of selecting participants was shown in Figure 1. Ethical approval and the requirement for informed consent forms were waived in this study, as the data were accessed from SEER, a publicly available database.

Figure 1 Flow chart of patient screening.

Data Collection

The primary endpoints of this study were liver metastases in OC patients and the 12-, 36- and 60-month overall survival (OS) of OC patients with liver metastases. OS refers to the time from the diagnosis to death due to any cause. The following variables were extracted from the SEER database: age, race (black, white, other), marital status (married, single), laterality (only one side, bilateral), stage T (T1, T2, T3), stage N (N0, N1), grade (I, II, III, IV), histologic type (serous, endometrioid, clear cell, mucinous, other), cancer antigen-125 (CA-125) (negative, positive), surgery (no, yes), chemotherapy (no/unknown, yes), and radiotherapy (no/unknown, yes), bone metastasis (no, yes), brain metastasis (no, yes), lung metastasis (no, yes), liver metastasis (no, yes). Specifically, the variables included in the prediction of liver metastasis at diagnosis were limited to age, race, marital status, laterality, stage T, stage N, grade, histologic type, and CA-125; all variables were included in the prediction of OS.

Feature Selection and Validation Strategy

In order to minimize the negative impacts of overfitting, we carried out feature selection to remove irrelevant or redundant invalid features. In this study, the Boruta algorithm was adopted to systematically evaluate the importance of features. Variables with P<0.01 were ultimately included in the prediction and diagnostic models.

The overall dataset collected from the SEER database was randomly divided into two cohorts, namely the training cohort and the validation cohort, at a ratio of 6:4. In the diagnostic model, indicators such as accuracy, kappa, sensitivity, specificity, positive predictive value, negative predictive value and AUC were used to evaluate the reliability of the nine ML models. In the prognostic model, the AUC and Brier score was applied to assess the reliability of the five ML models. After a comprehensive comparison of different ML models, we selected the model with the best predictive ability as the final predictive model. To further confirm the applicability of the selected model, we evaluated it in the validation cohort.

The importance of each feature was ranked through the Shapley Additive Explanation (SHAP) in diagnostic models. The feature importance was ranked through the Brier score in prognostic models.

ML Algorithms

R language (version 4.4.3) software is used to build machine learning prediction models.

Ridge regression is a biased estimation regression method specifically used for the analysis of collinear data. In essence, it is a modified least squares estimation method. By giving up the unbiasedness of the least squares method and at the cost of losing some information and reducing the accuracy, it obtains a regression method in which the regression coefficients are more in line with reality and more reliable.

Logistic regression is a generalized linear regression analysis model. In fact, it is mainly used to solve binary classification problems or multi-class classification problems. The model is trained with the given n sets of data (training cohort), and after the training is completed, it classifies the given one or more sets of data (validation cohort).

The Neural Network (NN) is a mathematical or computational model that imitates the structure and function of a biological neural network (the central nervous system of animals, especially the brain). It is used to estimate or approximate functions and is mainly composed of an input layer, hidden layers, and an output layer.

Support Vector Machine (SVM) is a classic supervised learning algorithm used to solve binary and multi-class classification problems. Its core idea is to find an optimal hyperplane in the feature space for classification, with the maximum margin. SVM can perform linear or non-linear classification, regression, and even outlier detection tasks.

Light Gradient Boosting Machine (LightGBM) is a tree-based ensemble learning method that employs gradient boosting technology. It combines multiple weak learners into a powerful model. It is a gradient boosting machine algorithm used to solve classification and regression problems.

Decision Tree (DT) is a model that presents decision rules and classification results in a tree-shaped data structure. As an inductive learning algorithm, its focus is to transform seemingly disordered and chaotic known data into a tree-shaped model that can predict unknown data through certain technical means. Each path from the root node (the attribute that contributes the most to the final classification result) to the leaf node (the final classification result) represents a decision rule.

Random Forest (RF) is an ensemble learning algorithm based on multiple decision trees. It can be used not only for classification problems but also for regression problems. Randomness is introduced during the construction process of the RF, which helps to reduce overfitting.

Extreme Gradient Boosting (XGBoost) is a kind of composite algorithm formed by combining basis functions and weights, which has a good fitting effect on data. Different from traditional gradient boosting decision trees, XGBoost adds a regularization term to the loss function. And since some loss functions are difficult to calculate the derivative, XGBoost uses the second-order Taylor expansion of the loss function as the fitting of the loss function.

The K-Nearest Neighbor (KNN) algorithm can be used for both classification and regression. KNN performs classification by measuring the distances between different feature values. For any n-dimensional input vector in the KNN algorithm, which corresponds to a point in the feature space, the output is the corresponding category label or predicted value of the feature vector.

The Cox Proportional Hazards (Coxph) is used to analyze the impact of covariates on survival time in survival data. Based on a semi-parametric method, this model does not require specific assumptions about the hazard function. Instead, it uses the hazard ratio (HR) to compare the risks among different covariate groups.

Lasso regression is mainly designed to address the problems that traditional linear regression encounters when dealing with high-dimensional data. In a high-dimensional space, traditional Ordinary Least Squares (OLS) regression may face issues such as difficulties in variable selection and model overfitting. By introducing a tuning parameter (λ), Lasso penalizes the absolute values of the coefficients, compelling some unimportant coefficient values to become zero.

Random Survival Forest (RSF) is a survival analysis method based on the random forest. It constructs a large number of survival trees and, in the form of voting, weighted elects the final prediction result from individual trees.

Statistical Analysis

R 4.4.3 software was used for data description and statistical analysis. Normally distributed quantitative data were shown as mean [standard deviation, (SD)] and analyzed by Student’s t- test. Non-normal quantitative data were presented as median (interquartile range, [IQR]) and analyzed by Mann–Whitney U-test. Categorical variables were described as frequency (%) and analyzed by chi-square test. The Boruta algorithm was used to screen important features. Variables with P<0.01 were included in the prediction and diagnostic models. To explore the impact of surgery, chemotherapy, and radiotherapy on OS in OC patients, age, histological type, grade, surgery (matched when analyzing chemotherapy and radiotherapy), radiotherapy (matched when analyzing chemotherapy and surgery), and chemotherapy (matched when analyzing radiotherapy and surgery) were subjected to propensity score matching (PSM), followed by Kaplan-Meier survival analysis. Difference was considered statistically significant when the P<0.05.

Results

Population Features

A total of 27,065 OC patients were included in this study. Among them, 1,053 patients developed liver metastasis at diagnosis, and 26,012 patients did not liver metastasis at diagnosis. Table 1 showed the demographic and clinical characteristics of OC patients with and without liver metastasis at diagnosis. Compared with OC patients without liver metastasis at diagnosis, OC patients with liver metastasis at diagnosis were older (P<0.001), were more likely to be black (P=0.046), were more likely to be single marital status (P=0.031), had bilateral tumors (P<0.001), presented with a more advanced T stage (P<0.001), presented with a more advanced N stage (P<0.001), had a higher pathological grade (P<0.001), had more serous histology (P<0.001), had more CA125 positive (P<0.001).

Table 1 Demographic and Clinicopathological Characteristics of All Included Patients

Risk and Prognostic Factors for OC Liver Metastasis at Diagnosis

Feature selection was carried out using the Boruta algorithm, and the results were shown in Figure 2. It was observed that histological type, T stage, grade, age, N stage, CA125, laterality, and race were associated with liver metastasis at diagnosis in OC patients. In the meantime, histological type, chemotherapy, surgery, radiotherapy, lung metastasis, bone metastasis, age, grade and marital status with OS in OC liver metastasis patients.

Figure 2 Feature Importance by Boruta. (A) Diagnostic mode. (B) Prognostic model. Green boxplots represent features with high importance that passed statistical significance testing. Red boxplots indicate low-importance features that failed to meet statistical significance. Blue boxplots represent shadow features.

Predictive Performance of the ML Models for Diagnosis and Prognosis

Diagnosis Model

Figure 3 showed the receiver operating characteristic (ROC) curves of nine ML models used for predicting the training cohort and the validation cohort. In the training cohort, the AUC value of KNN was the highest, which was 0.863 (Table 2). In the validation cohort, the AUC value of Ridge was the highest, reaching 0.758 (Table 3). However, given the uneven distribution of positive and negative events in the dataset, the AUC alone is insufficient to explain the performance of the model. Therefore, accuracy, kappa, sensitivity, specificity, positive predictive value, and negative predictive value were adopted to make up for the deficiencies of the ROC curve, so as to further evaluate the advantages and disadvantages of the model. Among the 9 ML models, Ridge regression showed comparatively favorable comprehensive performance. In the validation cohort set, the AUC (0.758), sensitivity (0.929), and negative predictive value (0.994) were the highest (Table 3).

Table 2 Performance Metrics of Machine Learning Algorithms in Diagnosis Model on the Training Cohort

Table 3 Performance Metrics of Machine Learning Algorithms in Diagnosis Model on the Validation Cohort

Figure 3 The ROC curves of diagnostic models based on machine learning. (A) Training cohort. (B) Validation cohort. Abbreviation: ROC, receiver operating characteristic.

Prognostic Model

Five ML algorithms, namely Coxph, LightGBM, Lasso, RSF, and XGBoost, were used for survival analysis. Those models evaluated through AUC and Brier score. Among five models, RSF showed the best comprehensive performance. The ROC curves of five ML models in the training cohort and the validation cohort at 12-months OS, 36 months OS, and 60 months OS, were shown in Table 4. At the 12-month OS, the AUC of the RSF model in the training cohort was the highest, reaching 0.876, and similar results were found in the validation cohort. At the 36-month OS, the AUC value of the XGBoost model in the training cohort was the highest, reaching 0.814, and the highest AUC value of the RSF model in the validation cohort was 0.720. At the 60-month OS, the AUC value of the XGBoost model in the training cohort was the highest, reaching 0.827, and the highest AUC value of the Coxph model in the validation cohort was 0.689 (Figure 4 and Table 4). Overall, the AUC values of most models generally showed a decrease over time, both in the training cohort and the validation cohort (Figure 5) and Table 5 showed the results of the Brier score. Regardless of whether it was at 12-month OS, 36-month OS or 60-month OS, the Brier score of the XGBoost in the training cohort was the lowest, which were 0.095, 0.173 and 0.154 respectively. In the validation cohort, the Brier score of the RSF was the lowest, which were 0.124, 0.215 and 0.187 respectively. In terms of the trend, the Brier scores of each model first increase and then decrease over time (Figure 6).

Table 4 Time-Dependent Area Under Curve of Machine Learning-Based Survival Models

Table 5 Time-Dependent Brier Score of Machine Learning-Based Survival Model

Figure 4 The time- dependent ROC curves of prognostic models based on machine learning in the validation cohort. (A) 12-month OS, (B) 36-month OS, (C) 60-month OS. Abbreviations: ROC, receiver operating characteristic, OS, overall survival.

Figure 5 The time-dependent AUC of prognostic models based on machine learning. (A) Training cohort. (B) Validation cohort. Abbreviation: AUC, area under the curve.

Figure 6 The time-dependent Brier score of prognostic models based on machine learning. (A) Training cohort. (B) Validation cohort.

Feature Importance in the Machine Learning Models

The Ridge regression with favorable performance was selected for the subsequent SHAP importance analysis. As can be seen from Figure 7A, among those factors, the stage T was the most discriminative features for diagnosing liver metastasis at diagnosis, followed by histological type, stage N, and grade. Considering the performance of AUC and Brier score in both the training cohort and the validation cohort, the RSF model showed the best comprehensive performance. As shown in Figure 7B, for the OS at 12 months, chemotherapy was the most discriminative feature for prognosis, followed by histological type, surgery and age. For the OS at 36 months, histological type was the most discriminative feature for prognosis, followed by chemotherapy, surgery and age. For the OS at 60 months, age was the most discriminative feature for prognosis, followed by histological type, grade and surgery.

Figure 7 Feature importance ranking. (A) Ridge regression. (B) Random Survival Forest.

Survival Analysis

The results of PSM were shown in Table 6–8. As depicted in Figure 8, compared with patients who did not receive surgery, those who underwent surgery had a significantly higher survival probability (P<0.0001). Similarly, chemotherapy recipients had a higher survival probability than those who did not receive chemotherapy (P<0.0001). However, patients who did not receive radiotherapy had a higher survival probability.

Table 6 Comparison of Patient Characteristics According to Chemotherapy Before and After Propensity Score Matching

Table 7 Comparison of Patient Characteristics According to Surgery Treatment Before and After Propensity Score Matching

Table 8 Comparison of Patient Characteristics According to Radiotherapy Before and After Propensity Score Matching

Figure 8 Kaplan-Meier (K-M) survival analysis after propensity score matching. (A) OS of patients with different surgical treatment; (B) OS of patients with/without chemotherapy; (C) OS of patients with/without radiotherapy. Abbreviation: OS, overall survival.

Discussion

In this study, multiple ML models were used based on the SEER database to establish diagnostic models and prognostic models for liver metastasis at diagnosis of OC patients and the importance of features in the models was explained. This study found that Ridge regression showed favorable performance in the diagnostic model, while the RSF model demonstrated relatively better overall performance in the prognostic model.

With the rapid development of artificial intelligence technology, ML is being applied more and more in the medical field. Zhou et al used six ML models to predict the recurrence of OC and it was found that ML algorithms outperformed traditional logistic regression analysis, and XGBoost demonstrated the best performance in predicting the recurrence of OC, with an accuracy rate of 0.95.¹⁶ Feng et al established a diagnostic model for OC and found that the AUC curve of the diagnostic model reached 0.948, at the same time, its diagnostic efficiency is significantly superior to the traditional method of detecting CA125 alone.¹⁷ A SEER-based study has found that the RSF performs better in predicting the 1-year, 3-year, and 5-year survival rates of epithelial ovarian cancer, with the AUC of the 1-year, 3-year, and 5-year survival rates being 0.926, 0.748, and 0.836 respectively.¹⁸ The RSF algorithm is a popular integrated ML tool, which is widely applied to clinical decision support and prognosis prediction tasks.^19,20 In this study, RSF also demonstrated the relatively favorable predictive performance among the five ML models. The model has a high predictive value, providing clinicians with more accurate predictions to inform clinical decisions.

In this study, the incidence of liver metastasis in patients with OC was 3.89%. Yuan et al analyzed the SEER data of OC patients from 2010 to 2014 and found that 3.61% of OC patients had liver metastasis.¹¹ Another study based on the SEER database identified 1,744 patients with liver metastasis of ovarian cancer from 2010 to 2016, accounting for 6.7% of all OC patients.¹⁰ In another study, it was found that 5.67% of OC patients had liver metastasis.²¹ The occurrence of the above situation may be related to the inclusion and exclusion criteria.

In this study, the Boruta algorithm was used to screen out eight factors related to liver metastasis at diagnosis, including histological type, stage T, grade, age, stage N, CA125, laterality, and race. According to the importance order of the SHAP diagram, the features with prominent contributions were stage T, stage N, histological type, and grade. The study showed that the increase in the stage T and stage N of malignant tumors indicated an increase in tumor volume, as well as an expansion in the extent of involvement of adjacent tissues and lymph nodes, which was indicative of the further progression of malignant tumors.²² The stage T and stage N were found to be associated with liver metastasis in patients with various cancers. In patients with primary epithelial ovarian cancer, the stage T and stage N were also found to be associated with the occurrence of liver metastasis.²³ It was found that the stage T, and stage N were risk factors for liver metastasis in gallbladder cancer patients.²⁴ The main histological types of ovarian cancer include serous carcinoma, endometrioid carcinoma, clear cell carcinoma, mucinous carcinoma, etc.²⁵ High-grade serous carcinoma is the most common type of ovarian cancer, usually with a high degree of invasiveness and metastatic potential, and it is more likely to develop liver metastasis.^26,27 The grade of tumors is usually determined based on the degree of differentiation of tumor cells. High-grade ovarian cancer often has a higher frequency of gene mutations and more complex genomic alterations.^28,29 These alterations may enable cancer cells to acquire stronger abilities of migration, invasion, and angiogenesis, thus promoting the occurrence of liver metastasis.³⁰ Besides, it was found that chemotherapy was the most crucial factor for the 12-month OS of OC patients with liver metastasis. For OC patients with liver metastasis, chemotherapy is one of the important treatment methods.³¹ By using chemotherapy drugs, such as cisplatin, it can effectively kill cancer cells, control tumor growth, and prolong the survival period of OC patients with liver metastasis.³² In this study, surgery was associated with improved OS. Multiple studies have found that surgery is beneficial for the survival of OC patients.^33,34 However, in this study, radiotherapy was associated with OS, which may be related to the small sample size of patients who received radiotherapy.

Our study has several limitations. First, it is important to clarify that our diagnostic model addresses the presence of liver metastasis at the time of initial diagnosis, not the future development of liver metastasis. Second, the SEER database only includes cancer information from the U.S. population, which inevitably introduces selection bias. Data from other countries are needed to validate the generalizability of our findings. Third, the SEER database lacks detailed information on certain clinical variables, including routine blood parameters (eg., white blood cell count, red blood cell distribution width) and comorbidities. Moreover, the granularity of available variables is limited., which may affect the feature importance and predictive performance of our model, Future studies with more rigorous temporal ordering of predictors. Fourth, due to the absence of key biological data (such as biomarkers, genetic or molecular profiles), we could not fully account for underlying biological mechanisms that might influence the outcomes. Fifth, as a retrospective study, our analysis is subject to inherent biases, including information bias and unmeasured confounding. Prospective studies with more comprehensive data collection and randomized designs are needed to confirm our results. Finally, we acknowledge that the clinical utility of our models is currently limited by the lack of external validation and the modest predictive performance observed in the validation cohort.

Conclusion

Histological type, stage T, grade, age, stage N, CA125, laterality, and race were associated with liver metastasis at diagnosis in OC patients. Meanwhile, marital status, grade, age, bone metastasis, lung metastasis, radiotherapy, surgery, chemotherapy, and histological type were associated with OS in OC patients with liver metastasis at diagnosis. The Ridge regression and RSF had favorable predictive performance in the diagnostic and prognostic models, respectively, compared with the other tested models. These findings may provide supportive evidence to assist clinicians in identifying OC patients at risk of liver metastasis at diagnosis and facilitate individualized clinical decision-making.