Development and Validation of a New Multiparametric Random Survival Forest Predictive Model for Breast Cancer Recurrence with a Potential Benefit to Individual Outcomes

Introduction

Breast cancer (BC) is a malignant tumor originating from the epithelial tissue of the breast. The incidence of BC has been increasing annually, and BC has become a significant threat to women’s health. According to the data of GLOBOCAN 2020, the number of new cases of female BC is estimated to be 2.3 million (11.7%), surpassing lung cancer as the most common cancer type.¹ In China, morbidity and mortality rates of BC have increased in recent years because of lifestyle changes, dietary regimens, and the natural environment.^2,3

Conventional prognostic factors for BC include tumor-node-metastasis (TNM) stage (tumor size, number of metastatic lymph nodes, distant metastatic state), tumor grade, and expression of molecular biomarkers such as estrogen receptor (ER), progesterone receptor (PR), Ki-67, and human epidermal growth factor receptor 2 (HER-2).⁴ However, patient characteristics, including age, body fat, and nutritional and inflammatory status, affect tumor prognosis.^5–8 With the recent development of gene technology, several studies have focused on the genetic diagnosis of BC, and researchers have proposed the integration of multiple genes and molecular markers to construct new prognostic models for predicting BC prognosis. One of them was the 21-gene recurrence score assay, which considers the expression of 21 genes to accurately predict the risk of BC recurrence in patients.⁹ However, gene detection requires proficient technological devices and is costly; thus, it is still not widely used.⁹

Overall, prognostic factors of BC include not only the tumor itself but also patient characteristics. Using tumor or patient characteristics separately to predict BC recurrence may be inaccurate.^10,11 The genetic prediction of BC recurrence has been shown to be accurate, but obtaining genetic information is laborious, time-consuming, and relatively expensive, which has greatly limited use in clinical practice.⁹

Breast cancer is a multi-factorial disease, so individualized patient profiling is instrumental to improving BC management and individual outcomes. Thus, the paradigm shift from a reactive to a predictive, preventive, and personalized medicine approach is essential to improve BC management.^12–14 Thus, a comprehensive, accurate, and low-cost predictive model using multi-level diagnostics and disease modeling that can be used in clinical practice is needed.

The rapid development of machine learning technologies has facilitated the construction of predictive models that can efficiently evaluate numerous parameters. However, conventional models often have a low level of predictive accuracy due to overfitting.¹⁵ Random survival forest (RSF), derived from random forest, is a machine learning method based on both random forest and survival analysis.¹⁶ It has the following advantages: it has no special requirements for the data set and can be used to analyze data for which a number of variables are significantly larger than the sample size. Moreover, RSF effectively avoids problems associated with overfitting and collinearity.^17–19 Further, no restrictions on the type of data or the association between predictive variables and outcomes are needed, nor is RSF constrained by proportional risk or logarithmic linear assumptions. As a result, higher levels of accuracy are achieved.^16,20 To capitalize on the advantages of the machine learning method, we developed a new predictive model for BC recurrence using RSF, which was based on baseline, cross-sectional, common clinical variables, including general patient information, blood tests, pathological examinations, and adverse events due to treatment.

Here, we hypothesized that multi-level diagnostics and disease modeling may lead to the identification of risk of recurrence for BC patients. Furthermore, a multi-omic predictive model using machine learning was considered a potent tool for stratifying patients with high versus low risk for BC recurrence.

Methods

A retrospective survey of BC patients’ medical records was performed at the Third Affiliated Hospital of Sun Yat-sen University and Liuzhou Women and Children’s Medical Center between January 2013 and December 2020. All patients enrolled in this cohort study were diagnosed with stage 0 to III primary BC and had received primary BC therapy. Patients lost to follow-up and those with tumor stage IV, a history of cancer, other synchronous malignancies, or incomplete important information (lacking more than 50% variables) were excluded from the study. Patients diagnosed and treated at the Third Affiliated Hospital of Sun Yat-sen University were included in the training set for model development, and patients diagnosed and treated at Liuzhou Women and Children’s Medical Center were included in the validation set for model validation. The flowchart of the study design and patient selection is shown in Figure 1.

Figure 1 Flowchart of the study design and patient selection.

This study was approved by the ethics committee of the Third Affiliated Hospital of Sun Yat-sen University and Liuzhou Women and Children’s Medical Center. The study was in compliance with the Declaration of Helsinki and its later amendments. All study participants provided informed consent to review their medical records. Identifiable data involving the individuals in this study were encrypted.

Potential Predictors

The patients’ data were obtained from their medical records. The results of the routine peripheral blood parameters and biochemical parameters before initiating any treatment were reviewed. The complete blood cell counts and biochemical parameters were measured by standard clinical laboratory methods. White blood cell (WBC), neutrophil (NEUT), and platelet (PLT) counts were collected, and the neutrophil count to lymphocyte count ratio (NLR) and platelet count to lymphocyte count ratio (PLR) were calculated. We retrospectively investigated patients’ characteristics, including age, type of chemotherapy, chemotherapy toxicities, and prognoses based on a review of patients’ medical records or telephonic follow-up.

ER, PR, and Ki67 status was assessed by immunohistochemistry (IHC). HER-2 status was evaluated by IHC and/or fluorescence in situ hybridization (FISH). Tumors exhibiting greater than or equal to 10% positivity for ER or PR at any staining intensity among the total tumor cells were considered positive. The HER-2 staining intensity score was evaluated from 0 to 3+. HER-2 membranous staining was evaluated as 0 if no cells showed staining; as 1+ if incomplete, faint staining was present in >10% of cells; as 2+ if complete, moderate staining was present in >10% of cells; and as 3+ if complete, strong staining was present in >10% of cells. An HER-2 score of 0–1+ was considered negative, and when HER-2 score was 2+ and 3+, further examination of FISH was performed. Specimens scored as 3+ or confirmed to display amplification based on FISH were considered positive. The patients were categorized into four subtypes based on ER, PR, HER-2, and Ki67 via IHC of their tumor in the following manner: Luminal A (ER+ and/or PR+, HER-2‐, Ki67 <14%), Luminal B (ER+ and/or PR+, HER-2+) or (ER+ and/or PR+, HER-2‐, Ki67 ≥ 14%), HER-2+ (ER‐, PR‐, HER-2+), and TNBC (ER‐, PR‐, HER-2‐). The TNM staging was performed according to histopathology results using the recommendation of the American Joint Committee on Cancer.

Assessment of Adverse Events

Data on adverse events were collected and assessed by the Common Terminology Criteria for Adverse Events version 5.0. The severity of adverse events was measured using grades 1 to 5 as follows:

Grade 1: Mild, asymptomatic or mild symptoms, clinical or diagnostic observations only, and intervention not indicated.

Grade 2: Moderate; minimal, local, or noninvasive intervention indicated; limiting age-appropriate instrumental activities of daily living (ADLs).

Grade 3: Severe or medically significant but not immediately life-threatening; hospitalization or prolongation of hospitalization indicated; disabling; and limiting self-care ADLs.

Grade 4: Life-threatening consequences and urgent intervention indicated.

Grade 5: Death related to adverse events.

Severe adverse events included grades 3 to 5.

Statistical Analyses

The patients’ baseline demographic and clinical characteristics are listed as percentages or means with standard deviation. Student’s t-test or Mann–Whitney U-test was employed to estimate the continuous data, and the chi-squared test was employed to estimate the categorical data. R software was employed when performing all statistical analyses. All analyses were two-tailed, and differences were statistically significant when P was < 0.05.

We used the R mice package (PMM) to interpolate the missing data in the training and validation sets. Based on the characteristics of BC, demographic characteristics, routine blood tests, treatment adverse events, and interruption of treatment, the random forest-recursive feature elimination (RF-RFE) (run by R caret package) was used to determine the best variable set, and the RSF method was used to develop a predictive model for the recurrence risk of BC patients. All the pairs of ntree and mtry were formed by a grid search using 10-fold cross-validation, and those with the best concordance index (C-index) values were identified as the optimized parameters. Moreover, the C-index was used to evaluate the discrimination of the predictive model, and the receiver operating characteristic (ROC) curve and Kaplan-Meier (KM) survival analysis were used to evaluate the precision of the model in predicting BC recurrence.

Results

In total, 774 patients were included in the study. Patients diagnosed with BC and treated at the Third Affiliated Hospital of Sun Yat-sen University were included in the training set (n=623) for model development, and patients diagnosed with BC and treated at Liuzhou Women and Children’s Medical Center were included in the validation set (n=151) for model validation.

We used R-Pack mice (PMM) to interpolate the missing data in the training and validation sets. Basic information regarding the training and validation sets is shown in Table 1.

Table 1 Basic Information of the Training and Validation Sets

The RF-RFE program of the R caret package was used to filter the most highly predictive variables of the set, and we selected the optimal number of variables according to the root mean square error (RMSE). To evaluate the accuracy of a model, the RMSE of test and predicted values is determined. The lower the RMSE value, the higher the predictive accuracy of the model. Figure 2 shows that when the model included 14 variables, the RMSE value was lowest.

Figure 2 Evaluating the number of variables contained in the optimal set using the root mean square error.

The best variable set (14 variables) filtered by RF-RFE included the pathological (TNM) stage, gamma-glutamyl transpeptidase (GGT), total cholesterol (CHOL), Ki-67, lymphocyte count, low-density lipoprotein (LDL), age, apolipoprotein B (ApoB), high-density lipoprotein (HDL), serum globulin (GLB), neutrophil count to lymphocyte count ratio (NLR), alanine aminotransferase (ALT), triglyceride (TRIG), and serum albumin to serum GLB (A/G) ratio data. Variable importance (VIMP) indicated by RF-RFE is shown in Figure 3. The positive VIMP value indicates that one variable improves predictive accuracy, while the negative value indicates an adverse effect in the prediction.²¹ VIMP indicated the contribution of each variable to model prediction separately, but it did not consider the contribution of combinations of variables.^22,23 VIMPs of A/G were negative, but when they were included in the variable set, the RMSE value was reduced to the lowest level measured, indicating that the variable set that included A/G had the best predictive performance. Moreover, the previous studies reported that the increased A/G ratio often predicts a good prognosis,^6,24–26 which was reversed to recurrence. Therefore, A/G ratios were included in the variable set.

Figure 3 Variable importance values derived from the random forest-recursive feature elimination analysis.

RSF using the R software RandomForestSRC package was used to construct the model. As shown in Figure 4, the error rate of the model gradually stabilized as the numbers of fixed trees increased. Between 4000 and 6000, the out-of-bag error rate steadily decreased and tended to be approximately 0.3. The error rate was stable when the number of fixed trees was 10,000. Therefore, the selection of 10,000 trees (ntree = 10,000) was appropriate, and the best performing parameters (ntree = 10,000; mtry = 4) were selected to develop the RSF prognostic model. Subsequently, RSF-based scores for individual samples were calculated. The C-index was 0.997 (95% confidence interval [CI], 0.995–0.998) (strong discriminatory power).²⁷ ROC curve analysis was used to evaluate the performance of the developed RSF prognostic model in the training set. Based on RSF scores, the area under the ROC curve (AUROC) was 0.994 (95% CI, 0.9848–1.0), with a sensitivity of 97.9%, specificity of 98.4%, and an optimal cut-off value of 2.81 in the training set (Figure 5).

Figure 4 Change in the prediction error rate of the recurrence risk model of breast cancer patients with tree number.

Figure 5 Receiver operating characteristic curve of the developed random survival forest model.

We divided samples into high-and low-risk groups, which included above- and below RSF-based scores of 2.81, respectively. KM analyses revealed highly significant differences between overall survival times of the high-risk and low-risk groups (P < 0.0001) (Figure 6). Results showed that the developed RSF prognostic model was able to predict BC recurrence accurately.

Figure 6 Kaplan-Meier survival curves of recurrence-free survival for the training set.

The developed RSF prognostic model was applied in the assessment of an independent validation set, and the predictive performance of the model was evaluated using data of the Liuzhou Women and Children’s Medical Center cohort. The C-index was determined to be 0.936 (95% CI, 0.891–0.981) (strong discriminatory power).²⁷

ROC curve analysis was used to evaluate the performance of the developed RSF prognostic model in the validation set. Based on the RSF-based scores, the AUROC was 0.961 (95% CI, 0.926–0.996), with a sensitivity of 100% and specificity of 87.9% (Figure 7).

Figure 7 Receiver operating characteristic curve of the developed random survival forest model assessment by the validation set.

The optimal cut-off value of 2.81 in the RSF-based score in the training set was used to stratify patients into high-risk and low-risk groups in the validation set. KM analyses revealed highly significant differences in recurrence-free survival between high-risk and low-risk groups (P < 0.0001) (Figure 8).

Figure 8 Kaplan-Meier survival curves of recurrence-free survival for the validation set.

Discussion

This study employed an RF-RFE algorithm²⁸ to automatically select the most relevant features among 40 variables identified for further RSF model development. Variable selection is the process of selecting a data set that includes relevant features for further analysis to minimize possible generalization error.

The 14 selected variables for the model developed included pathological (TNM) stage, GGT, CHOL, Ki-67, lymphocyte count, LDL, age, ApoB, HDL, GLB, NLR, ALT, TRIG, and A/G. These were reported to be closely associated with BC recurrence risk in previous studies, and they were used in this study to develop a reliable model.

The Characteristic of the Selected Variables

The most important variable selected was pathological stage (TNM stage). Pathological stage has been widely used in clinical practice to predict prognosis and survival as well as guide clinical treatment.²⁹ Proliferating cell nuclear antigen (Ki-67) is a protein closely related to cell proliferation. Current studies have shown that Ki-67 is highly correlated with the differentiation, invasion, and metastasis of BC,³⁰ which can be used to predict the prognosis of BC patients. Some studies have confirmed that Ki-67-positive BC patients have a poor treatment response and prognosis.^29,31

Lymphocyte count, NLR, and platelet count to lymphocyte count ratio (PLR) are variables assessed in routine blood tests. In previous studies, it was found that a change in the white blood cell (WBC) count in peripheral blood is related to the systemic inflammatory response.⁵ Moreover, some studies have found that tumor-related systemic inflammatory response is an independent predictor of tumor prognosis in patients.^32,33 NLR and PLR can reliably reflect the body’s inflammatory status. In fact, studies confirmed the classification of WBC count in peripheral blood, and NLR and PLR can be used to predict the prognosis of BC.^5,34–36 Increases in NLR and PLR often indicate poor response to treatment and poor prognosis, whereas low NLR and PLR are often indicative of good treatment response and good prognosis.^37–39

Age is an independent risk factor for BC. The incidence and mortality rate of BC increases with age. Many factors related to age play an important role in the occurrence and development of BC, such as changes in hormone levels before and after menopause, cumulative DNA damage with aging, the occurrence of various types of chronic infections, and changes in the immune system.^40–42 At the same time, some studies have shown that the prognosis of young BC patients is often poor, which may be due to the higher pathological grade of young patients’ tumors, as these patients are also often cancer cell hormone receptor-negative and HER-2-positive and may have other adverse indications.^8,43

The liver function test is routinely performed to assess ALT, AST, total bilirubin, ALP, GGT, ALB, GLB, and A/G. Some researchers have used ALT and GGT to predict all-cause mortality in the general population and found that predictions using the factors were moderately accurate.⁴⁴ Previous studies have shown that high levels of GLB are associated with a poor prognosis in patients with breast and rectal cancers.^6,45,46 On the other hand, in various tumors including BC, non-small cell lung cancer, renal cell cancer, and laryngeal cancer, increased ALB levels and ALB/GLB ratios (A/Gs) are often predictive of a good prognosis.^6,24–26 Therefore, these liver function parameters can be used as predictors of BC prognosis.

BC patients with baseline hypertriglyceridemia may have poor prognosis.⁴⁷ Moreover, CHOL is associated with BC recurrence,⁴⁸ whereas a high HDL/CHOL ratio is associated with a good prognosis.⁴⁷

Novelty of the Proposed Model

However, breast cancer is a multi-factorial disease, and individualized patient profiling is instrumental to improving BC management and individual outcomes.^12–14 It is widely accepted that the prognosis of cancer patients depends on not only tumor characteristics but also patient characteristics.⁴⁹ Thus, we determined that it would be best to develop a predictive model using both tumor characteristics and patient characteristics. In terms of validity and reliability, we found that our predictive model performed well even upon external validation using an independent data set. C-indexes were 0.997 and 0.936 for the training and external validation sets, respectively, and discriminatory power was good. The KM analyses (P < 0.0001 for both training and validation sets) were used to evaluate the performance of the model, and we found that our model could accurately predict BC recurrence. Further, AUROCs were 0.994 and 0.961 for training and validation sets, respectively, indicating that our model was able to reliably predict BC recurrence. Moreover, the AUROC of the previously reported predictive model ranged from 0.69 to 0.92, suggesting that our model was more accurate than the models previously described.^50–53

Compared with published predictive models created based on new molecular biomarkers derived from gene or protein expression analysis, our existing models rely on simple and easy-to-obtain demographic data and clinical routine examination indicators obtainable in clinical practice. This means that a high-accuracy prediction may be made without increasing cost to patients. From the perspective of economic cost, using conventional laboratory indicators has lower cost than using new molecular biomarkers, particularly since new molecular biomarkers measurements are not routinely conducted in clinical practice. Further, this additional expense is not covered by insurance. The model constructed in this study included comprehensive selected features for both tumor and patient-related features and can be performed without additional cost to patients and is easy to operate. The model has the potential to help clinicians identify and provide interventions for high-risk patients in the early stage of disease and allow physicians to perform more accurate, targeted, efficient, and individualized treatment plans to improve the prognosis of BC patients.

Our study has some limitations. First, all study participants were of Han descent. Thus, the model has limited applicability to other races until external validation using data of patients from other regions and ethnicities is performed. Second, this is a cross-sectional study, which created a prediction model based on the baseline level, suggesting that timing and causality could not be determined. Thus, studies conducted at multiple centers that include larger cohorts are required.

Conclusions

We developed and validated a model to predict BC recurrence in patients in China. Predictive variables were selected based on data commonly obtained in clinical practice, which would not incur an additional cost to patients. The RSF model exhibited high discriminatory accuracy and good calibration, which may facilitate recurrence prediction. Moreover, by using the model, clinicians can deliver precise and efficient individualized treatment for BC patients to improve their prognosis. This new multiparametric RSF model is instrumental for breast cancer recurrence prediction and potentially improves individual outcomes.