A Proposed Heterogeneous Ensemble Algorithm Model for Predicting Central Lymph Node Metastasis in Papillary Thyroid Cancer

Introduction

Incidence of thyroid cancer is rapidly increasing worldwide. Papillary thyroid cancer (PTC) is the most common pathological type, accounting for 80–85% of thyroid cancers.¹ In the United States, the overall incidence of thyroid cancer is increasing by 3% each year, and the incidence and mortality of advanced PTC have increased.^2,3 The incidence-based mortality is increasing, mainly due to the changing biology of thyroid cancer, which in turn could be some unknown attributable reason, such as environmental, etiologic factors.⁴ Despite the growing molecular analysis in thyroid cancer prognostics, the actual prognostic value of molecular markers still needs to be validated.^5,6 PTC has over 98% survival rate in patients aged <45 years. Nevertheless, PTC is prone to lymph node metastasis, which worsens the prognosis of patients.⁷ Central lymph node metastasis (CLNM) is the most common type of lymph node metastasis.⁸ Previous studies have shown that patients with CLNM are more likely to have distant metastases and a poor prognosis.^9,10 Therefore, it is beneficial and necessary to perform lymph node dissection in patients with PTC who are identified with CLNM. However, it is controversial whether to perform routine central lymph node dissection in clinically negative CLNM. Several researchers believed that performance of routine central neck dissection could achieve a less recurrence, better outcomes, decreased postoperative thyroglobulin levels and a lower morbidity rate related to initial operation.^11–14Nevertheless, others thought that a more aggressive procedure could increase the risk of surgical complication such as permanent hypocalcemia, hypoparathyroidism and recurrent laryngeal nerve paralysis.^15,16 Cervical ultrasound (US) and computed tomography (CT) have poor performance in detecting CLNM in PTC patients.^17–19

Machine learning (ML) is a new and advanced processing method involving the study and development of computational models of learning processes and big data analysis. Additionally, this technique has been proven effective for analyzing medical problems.^20,21 However, previous studies focused on utilizing the single-base ML model for medical research.^22,23

The ensemble learning algorithm, also known as multiple classifier systems, is a more effective ML model than the single-base model.²⁴ There are two types of ensemble algorithms. One ensemble algorithm consists of several identical basic learners, called homogeneous ensemble algorithms; the other algorithm consists of different basic learners, called heterogeneous ensemble algorithms. A previous study pointed out that the performance of the basic model is better than that of ordinary US in detecting CLNM in patients with PTC.²⁵ The current study proposes a heterogeneous ensemble algorithm based on clinical data from a single medical institution. To better present and interpret the heterogeneous ensemble algorithm for clinicians, we have used a partial dependence plot for visualization and to analyze how the significant variable affects CLNM in PTC patients.²⁶

The aim of the current study was to develop a new machine learning model to assist clinicians in decision-making. If a PTC patient with positive or high probability of CLNM according to the prediction of new machine learning model, the CLNM should be paid more attention and the patient should have more tendency to undergo central lymph node dissection.

Methods

Data Collection

This study was a retrospective clinical investigation approved by the Ethics Committee of Shanghai Jiao Tong University Affiliated Sixth People’s Hospital. All patients provided informed consent to participate in the study. A total of consecutive 1046 patients who underwent initial thyroidectomy between January 2014 and December 2018 at the Shanghai Sixth People’s Hospital were included. Patients with PTC were confirmed by postoperative pathology. We excluded PTC patients for the following reasons: Other subtypes of thyroid cancer; previous treatments including drug medication; or radiotherapy before initial surgery; relevant information was not detailed. The screening process of data collection is illustrated in Figure 1.

Figure 1 Detailed screening process of data collection.

Surgical Method

Patients with unilateral PTC underwent unilateral thyroidectomy plus ipsilateral central lymph node dissection, and patients with bilateral PTC or isthmic tumors underwent total thyroidectomy with bilateral CLND.²⁷ The definition of the central neck (N1a) included both level VI (peritracheal, paratracheal, or Delphina) and level VII (upper mediastinal) lymph node compartments.²⁸

Clinical Features

The current study collected information on the patient’s clinical characteristics, thyroid function tests, and US features. Clinical characteristics include basic information such as sex and age. We also collected data on the following items from thyroid function tests one week before surgery: free T3 (FT3), free T4, thyroid-stimulating hormone (TSH), thyroglobulin (TG), anti-thyroglobulin antibody, anti-thyroid peroxidase antibody (TPO-Ab), thyroid-stimulating hormone receptor antibody, and calcitonin. All patients underwent cervical US within one month before surgery at our hospital with GE Voluson E8 Color Doppler Ultrasound Diagnosis System. US characteristics of thyroid nodules included tumor size, unclear margin, hypo-echogenicity, the total number of lesions, microcalcification, irregular shape, US-suspected, and blood supply.²⁹ The US feature of hypo-echogenicity of a thyroid nodule is strongly related to an increased risk of malignancy.³⁰ Therefore, the mass was divided into hypo-echogenicity and none hypo-echogenicity. Tumor size was defined as the sum of the maximum diameter of a single-lesion mass and the total tumor diameter of a multi-lesion mass from US examination.³¹ An unclear boundary indicated that the boundary of the mass is difficult to determine using US detection. The total number of lesions was defined as the number of thyroid gland tumors. Blood supply was defined as obvious blood flow signals in the tumor using the US with color Doppler flow imaging. Microcalcification was defined as a spot-like strong echogenicity with an inner diameter <1 mm. Irregular shape was defined as a ratio of >1 in the anteroposterior diameter to the horizontal diameter when measured in the transverse plane. US-suspected was defined as the high possibility of tumor metastasis in level VI lymph nodes.

Feature Selection

Univariate and multivariate analyses were utilized for feature selection to avoid the overfitting state of the constructed machine learning model due to many variables. Variables with P < 0.05 in univariate analysis were included in the multivariate analysis. Variables with P < 0.05 in the multivariate analysis were important variables for ML model construction.

Development of Basic Machine Learning Model

Using the ratio of 8:2, the overall data set was divided into a training set (n=837) and a test set (n=209) using a random stratified sampling method.³² One-hot encoding is adopted to achieve data transformation of categorical features. For example, sex (male, female) with two values can be described as [(1,0), (0,1)]. Based on the training set, five basic ML models were developed: extreme gradient boosting, k-nearest neighbors (KNN), random forest, gradient boosting, and AdaBoost. The test set was used to assess the performances of the above models. Several previous studies confirmed that the above models perform well in category classification processing.^23,33 Extreme gradient boosting is an effective algorithm, which has been applied for regression and classification. KNN is a typical classification supervised algorithm, which can solve complex classification problems. Random forest is a homogeneous ensemble algorithm of decision trees, which is used to deal with the classification issues. Gradient boosting is machine model that integrated several same weak classifiers. AdaBoost is the combination of some multiple classifiers. In our research, the above five machine models were described as the basic model elements that were used to create the new proposed machine learning model.

Development and Explanation of the Heterogeneous Ensemble Algorithm Model

In this study, the heterogenous ensemble algorithm model (HEM) was defined as the advanced machine learning model, which is composed of five basic learning algorithms. We adopted an ensemble learning to develop HEM. Ensemble learning is a method to determine the final prediction target by combining the prediction of several different basic ML models.²⁴ The final prediction of the HEM depended on the prediction of the five basic ML models. When predicting, we obtain the five basic algorithms’ prediction results and use the prediction result with the most votes as the result. The algorithm formula was as following:

Where X is n dimensional feature vector inputs. Cj is the j class with more than half of the votes.

T means the number of basic ML models. Bj i(x) is the outputs of basic ML on the j class. M(x) is the prediction of the HEM model. For example, the prediction of three ML algorithms is a PTC patient with CLNM, and predictions of two ML algorithms is a PTC patient with NCLNM. The final prediction of HEM is a PTC patient with CLNM. The specific process of the heterogeneous integration algorithm is shown in Figure 2. Furthermore, the partial dependent plot, which is a tool for graphic visualization, is utilized to show how the response of the predicted CLNM is affected by dependent variables for explanation of HEM.²⁶ Moreover, optimal cut-off of tumor size and age by the ROC curve analysis.

Figure 2 Detailed procedure of development of heterogeneous ensemble algorithm model.

The programming software version R version 4.0.5 (https://www.r-project.org) and Python version 3.8.1 (https://www.python.org) were used for the statistical analysis and development of models.

Results

Clinical Characteristics

A total of 1046 consecutive PTC patients consisted of 821 PTC patients with NCLNM (78.5%) and 225 PTC patients with CLNM (21.5%), including 207 patients with ipsilateral CLNM (19.8%) and 18 PTC patients with bilateral CLNM (1.7%). Our sample included 256 males (24.5%) and 790 females (75.5%) and the mean age was 43.9±12.92 years. The mean tumor size was 1.02±0.76 cm and the mean number of tumor lesions was 2.33. The mean number of removed and positive lymph nodes was 5.02 and 1.56, respectively. A total of 307 patients with PTC (29.3%) underwent total thyroidectomy plus bilateral central lymph node dissection, and 739 (70.7%) underwent unilateral lobe thyroidectomy plus unilateral central lymph node dissection. Additionally, there were 69 PTC patients (6.6%) with US-suspected CLNM. More detailed clinical information regarding PTC patients is shown in Table 1. Additionally, younger PTC patients were more likely to have CLNM compared to those with older age, as illustrated in Table 2. Moreover, FT3, TG, TPO-Ab, size, sex, irregular shape, microcalcification, US-suspected CLNM and surgical resection showed significant difference between groups.

Table 1 Detailed Clinical Information of the PTC Patients

Table 2 Comparison of Information Between PTC Patients with NCLNM and PTC with CLNM

Feature Selection

Univariate logistic analysis showed that CLNM was significantly associated with clinical characteristics such as sex, age, FT3, TSH, TG, TPO-Ab, and the US features such as microcalcifications, irregular shape, US-suspected CLNM, and tumor size (p<0.05) (Table 3). Variables with a P value <0.05 in the univariate logistic analysis were selected for the multivariate logistic analysis. The multivariate logistic analysis showed that age (OR=0.974, 95% CI=0.961–0.986, P<0.001), TG (OR=0.997, 95% CI=0.994–0.999, P=0.011), TPO-Ab (OR=0.996, 95% CI=0.993–0.998, P=0.001), US-suspected CLNM (OR=2.202, 95% CI=1.251–3.387, P=0.006), size (OR=1.261, 95% CI=1.021–1.558, P=0.032), and microcalcification (OR=1.559, 95% CI=1.13–2.15, P=0.007) were significantly related with the CLNM in PTC patients and selected for development of ML model. More information is provided in Table 4.

Table 3 Univariate Analysis of Clinical Features Related to Central Lymph Node Metastasis in the PTC Patients

Table 4 Multivariate Analysis of Clinical Features Related to Central Lymph Node Metastasis in the PTC Patients

Performance and Visualization of HEM Model

Six variables significantly associated with CLNM were selected to develop the five basic predictive ML models, which were used to create HEM. Compared with US (area under the receiver-operating characteristic curve [AUC]=0.525), basic predictive ML models, including extreme gradient boosting (AUC=0.558), KNN (AUC=0.629), random forest (AUC=0.663), gradient boosting (0.609), and AdaBoost (AUC=0.614) performed better in detecting CLNM in PTC patients, as shown in Figure 3 and Table 5. The HEM model performed better (AUC=0.670) in predicting CLNM compared to the basic ML model. Additionally, brier score of the HEM model was 0.169 as shown in calibration curve (Figure 4). To better understand the HEM model, the response of predicted CLNM affected by age and TPO-Ab is shown in the partial dependence plot. As illustrated in Figure 5A, the predicted probability of CLNM shows an overall decreasing tendency as the age increases. Thus, for PTC patients, the earlier age diagnosed, the prediction of HEM is more likely to be CLNM. Additionally, the predicted probability of CLNM decreases sharply when the value of age increases from 20 to 40 years. The phenomenon indicates the PTC patients with an earlier age of diagnosis are more likely to have CLNM, and clinicians should pay more attention to younger patients and plan appropriate surgery for them. As illustrated in Figure 5B, the predicted probability of CLNM shows an overall decreasing tendency as the TPO-Ab increases. In other words, for PTC patients, the larger the value of the TPO-Ab, the prediction of HEM is more likely to be NCLNM. Several changes in the probability of CLNM should be noticed in the range of the TPO-Ab. Additionally, the predicted probability of CLNM increases sharply when the value of TPO-Ab increases from 25 KIU/L to 75 KIU/L. However, the predicted probability of CLNM decreases sharply when the value of age increases from 110 KIU/L to 240 KIU/L. The phenomenon indicates a non-linear relationship between the TPO-Ab and the CLNM. The cut-off age and size were 33 years and 0.8 cm, respectively, using ROC analysis.

Table 5 The Performance of the Machine Learning Models and Ultrasonography

Figure 3 Receiver operating characteristic curves of all machine learning models.

Figure 4 Calibration curve of the heterogeneous ensemble algorithm model.

Figure 5 Partial dependent plot of predicted probability of CLNM against age (panel (A)) and partial dependent plot of predicted probability of CLNM against TPO-Ab (panel (B)).

Discussion

PTC has a better overall prognosis than other pathological types of thyroid cancer, with a 5-year survival rate of >98%.³⁴ However, PTC is prone to CLNM, which is associated with a higher risk of tumor recurrence, lateral cervical lymph node metastasis, and locally advanced thyroid cancer.³⁵ Clinically, it is necessary to perform central lymph node dissection in PTC patients with CLNM. However, it is controversial whether performing central lymph node dissection is recommended for PTC patients with unidentified CLNM due to surgical complications, such as permanent hypocalcemia and laryngeal nerve paralysis.^36–38 Several researches about detecting cervical lymph node metastasis in PTC patients have been reported.^39,40 High success results of artificial intelligence assisted ultrasound in detecting central lymph node metastasis have been discussed previously.²⁵ Therefore, to further improve the efficiency of diagnosing CLNM, we have proposed using HEM, consisting of a combination of basic ML models.

In this study, we found that the US sensitivity and specificity were only 10.6% and 40.7%, respectively, and the AUC was only 0.525, which was consistent with the idea of low efficacy of US in previous studies.^17–19 Additionally, all basic ML models perform better than US, demonstrating that the ML model may be used to predict CLNM. Wu et al confirmed that ML algorithms might predict lymph node metastasis in the central area.²⁵ A possible reason is that the ML algorithm analyzes other important variables about CLNM, ignored by US detection. The innovation in this research is to develop a HEM to predict CLNM by integrating basic learning algorithms. We found that HEM performed better than basic models, demonstrating that HEM is an optimal and novel model for predicting CLNM. The HEM offers an effective technique for obtaining increased levels of predictive accuracy by combining the predictions of different ML models.⁴¹ The probable reason is that HEM uses different base algorithms for training, which can identify different types of errors during training procedure and correct false prediction in time, which is beneficial for improving the accuracy of integration during testing. Herein, five different algorithms were used; the principles of each model differed increasing the number of errors that could be eliminated and the accuracy rate.

In addition to developing a novel ML model, we found that several risk factors were significantly associated with CLNM. In the multivariate analysis, we found that age was strongly associated with CLNM. Younger patients with PTC have a higher risk of CLNM. Receiver operating characteristic (ROC) analysis suggested the optimal cut-off age was 33 years (AUC=0.520; sensitivity = 0.639; specificity=0.533). Moreover, the predicted probability of CLNM with the change of age indicates that younger PTC patients were more prone to CLNM as shown in Figure 3A. Liu et al have previously demonstrated that age ≤45 years is an independent risk factor for CLNM.⁴² It is reasonable that younger patients with PTC should be paid more attention. However, more clinical research is needed to find the optimal cut-off age.

Additionally, TPO-Ab may be a protective factor against CLNM according to the multivariate logistic analysis. Aydoğan et al identified that TPO-Ab was negatively associated with CLNM in differentiated thyroid cancer patients with chronic lymphocytic thyroiditis (CLT), which is consistent with our research result.^43,44

However, Babli et al demonstrated that the coexistence of CLT and PTC appeared to be associated with unfavorable pathological features in patients aged <45 years.⁴⁵ Currently, it is still controversial whether PTC patients with CLT have favorable clinicopathological characteristics compared with PTCs without CLT. As shown in Figure 3B, we found an unclear relationship between CLNM and TPO-Ab levels. To clarify the relationship between outcomes of PTC and CLT, more research is required to determine the association between the serum TG and CLNM.

We also found a positive correlation between tumor size and CLNM in patients with PTC. Liu et al confirmed that tumor size >1.0 cm was an independent risk factor for CLNM.⁴² Moreover, we also identified that the optimal cut-off of tumor size was 0.8 cm (AUC=0.539; sensitivity=0.435; specificity=0.528). Zhou et al believe that 0.7 cm is a better cut-off point for tumor size for CLNM in his research.⁴⁶ Therefore, tumor size is closely connected to CLNM, and more clinical research is required to investigate the optimal cut-off of the tumor size.

Additionally, we found that the serum TG level was negatively associated with the risk for CLNM. Nevertheless, a previous study has demonstrated that patients with TG >5000 ng/mL have a significant disease burden and preoperative TG levels are associated with the metastatic disease.⁴⁵ Moreover, Scappaticcio et al thought that Sustained de novo TG-Ab appearance is rare and may predict structural recurrences and similar disease-free survival was noticed in patients with continued de novo TG-Ab and negative TG-Ab DTC patients.⁴⁷ Therefore, more investigation is required to understand the associations between TG and CLNM. Currently, the US has relatively low accuracy for the diagnosis of CLNM, but we found that US-suspected CLNM is a significant predictive factor for CLNM. Therefore, it is necessary to reconsider the surgical method for a patient with suspected US-CLNM. Moreover, we found that microcalcification was significantly associated with the CLNM. Microcalcification implies deposition of calcium salt in the thyroid nodules owing to the proliferation of blood vessels and fibrous, reflecting the rapid growth of cancer cells. Therefore, if microcalcification is present in the nodules, the lymph node status in the central region should be evaluated more cautiously.⁴⁸

The innovation in this research is a combination of traditional and new statistical methods. First, univariate and multivariate analyses were used to select relatively important factors associated with CLNM. We then developed a HEM by combining basic ML models to predict CLNM in PTC patients with better performance. Second, to help clinicians better understand the novel model, partial dependency graphs were used to explain the model and visualize the change in tendency of risk of CLNM.

Limitations

The main purpose of this research was to develop a HEM model to predict the CLNM of PTC patients. However, there were some limitations to this research. Firstly, the number of cases studied was relatively small, and the model should be optimized based on the dataset with more PTC cases. Secondly, some important variables may have been ignored during data collection in the development of models after feature selection to avoid overfitting of models, such as TSH, thyroid imaging reporting and data system (TIRADS) classification.⁴⁹ Thirdly, potential selection bias, survivorship bias and study scope limitations cannot be avoided because of the retrospective study. Finally, prospective research is needed to verify the efficacy of the novel HEM model. As for other subtype thyroid cancer, the novel model should be designed based on the collection of data about patients with other subtypes of thyroid cancer to achieve implementation and validation of this algorithm.

Conclusion

Age ≤33 years, tumor size ≥0.8 cm, US-suspected CLNM, and microcalcification were risk factors for CLNM. Anti-thyroid peroxidase antibody and serum thyroglobulin levels were favorable factors for CLNM. The HEM has better performance than the US and other basic ML models, indicating that ML models could improve the function for predicting the CLNM. We believe that there will be more refined algorithms for clinical disease prediction based on this research.