Back to Journals » Cancer Management and Research » Volume 12

Prediction of Lymph Node Metastasis in Superficial Esophageal Cancer Using a Pattern Recognition Neural Network

Authors Chen H, Zhou X, Tang X, Li S, Zhang G

Received 1 July 2020

Accepted for publication 29 October 2020

Published 27 November 2020 Volume 2020:12 Pages 12249—12258

DOI https://doi.org/10.2147/CMAR.S270316

Checked for plagiarism Yes

Review by Single anonymous peer review

Peer reviewer comments 2

Editor who approved publication: Dr Beicheng Sun



Han Chen,1,2,* Xiaoying Zhou,1,2,* Xinyu Tang,2,3 Shuo Li,1,2 Guoxin Zhang1,2

1Department of Gastroenterology, The First Affiliated Hospital of Nanjing Medical University, Nanjing, People’s Republic of China; 2The First Clinical Medical College, Nanjing Medical University, Nanjing, People’s Republic of China; 3Department of Radiation Oncology, The First Affiliated Hospital of Nanjing Medical University, Nanjing, People’s Republic of China

*These authors contributed equally to this work

Correspondence: Guoxin Zhang
Department of Gastroenterology, The First Affiliated Hospital of Nanjing Medical University, 300# Guangzhou Road, Nanjing 210029, People’s Republic of China
Email [email protected]

Background or Purpose: It is important to predict nodal metastases in patients with early esophageal cancer to stratify patients for endoscopic resection or esophagectomy. This study was to establish a novel artificial neural network (ANN) and assess its ability by comparing it with a traditional logistic regression (LR) model for predicting lymph node (LN) metastasis in patients with superficial esophageal squamous cell carcinoma (SESCC).
Methods: A primary cohort was established, composed of 733 patients who underwent esophagectomy for SESCC from December 2012 to December 2019. The following steps were applied: (i) predictor selection; (ii) development of an ANN and a LR model, respectively; (iii) cross-validation; and (iv) evaluation of performance between the two models. The diagnostic assessment was performed with sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), accuracy, C-index, net reclassification improvement (NRI), and integrated discrimination improvement (IDI).
Results: The established ANN model had 6 significant predictors: a past habit of alcohol taking, tumor size, submucosal invasion, histologic grade, lymph-vessel invasion, and preoperative CT result. The ANN model performed better than the LR model in specificity (91.20% vs 72.59%, p=0.006), PPV (56.49% vs 39.78%, p=0.020), accuracy (90.72% vs 74.49%, p< 0.0001), C-index (91.5% vs 86.8%, p< 0.001), and IDI (improved by 23.3%, p< 0.001). There were no differences between these two models in sensitivity (87.06% vs 83.21%, p=0.764), NPV (98.17% vs 95.21%, p=0.627), and NRI (improved by − 1.1%, p=0.824).
Conclusion: This ANN model is superior to the LR model and may become a valuable tool for the prediction of LN metastasis in patients with SESCC.

Keywords: superficial esophageal squamous cell carcinoma, lymph node metastasis, neural network, machine learning

Introduction

Esophageal cancer is the third most common cancer and the fourth leading cause of cancer-related mortality according to Cancer Statistics in China.1 Although there has been a global reversal in the ratio of squamous cell cancers to adenocarcinomas,2 the esophageal squamous cell cancers in China still dominate all cases (~90%).3 Due to the prevalence of upper gastrointestinal endoscopy screening, a large part of esophageal cancers tends to be detected in an early stage.4 Accordingly, this shift in cancer stage distribution leads to a change in treatment strategy in China.

Traditionally, esophagectomy is considered a curative treatment for Superficial Esophageal Squamous Cell Carcinoma (SESCCs) but is usually associated with considerable postoperative morbidity and mortality.5 Endoscopic submucosal dissection (ESD) has recently become an alternative to esophagectomy in China, especially for T1a stage SESCCs without lymph nodes (LN) metastasis. Recent studies show that the 3- and 5-year overall survival rate still exceeds 90% among patients who even underwent endoscopic resection with T1b (invading into SM2 layer) cancer.6,7 Thus, the application of ESD for esophageal cancer has been gradually expanding within the past years.8 However, the main limitation of ESD is that it is only curative in those without LN metastasis because endoscopic treatment could not achieve lymphadenectomy. Therefore, clinicians must establish a model for predicting the risks of LN metastasis, especially in Asia.

Neural networks have been widely used in a broad range of areas such as business, data mining, drug discovery, and biology. In medical fields, they also have been applied successfully in the detection of disease, evaluation of new drugs, and estimation of the treatment cost. In several previous papers, the discriminatory power of logistic regression (LR) and ANN models was compared.9–11 Both models performed equally well in most cases, whereas the more flexible ANN model generally outperforming LR model in the remaining cases. Among the published prediction models for LN metastasis, there are already three LR-based nomograms.12–14 However, there are still no established ANN models. The aim of this study was to establish an ANN model and assess its ability by comparing it with a traditional Logistic Regression (LR) model for predicting LN metastasis in patients with SESCC.

Method

Study Design

We identified 3926 patients consecutively from a clinical electronic database. All patients were pathologically diagnosed with esophageal cancer by endoscopic biopsy between December 2012 to December 2019 in the First Affiliated Hospital of Nanjing Medical University. A retrospective cohort was established, composed of 733 consecutive patients who underwent primary surgical resection and lymphadenectomy for SESCC. The study was approved by the Institutional Ethics Committee of the First Affiliated Hospital of Nanjing Medical University. All participants gave informed consent for reviewing their medical records in the study.

Data Collecting Procedure

Inclusion criteria included: (1) histopathological diagnosis of esophageal squamous carcinoma on surgical specimens; (2) pT1 stage carcinoma (no tumor invasion beyond the submucosa); (3) patients who underwent primary surgical resection and at least two-field lymphadenectomy; (4) no history of previous malignancies and anticancer therapies. Exclusion criteria included: (1) esophageal adenocarcinoma or other types of esophageal cancer; (2) mixed types of esophageal cancer; (3) synchronous multiple lesion in esophagus; (4) tumor with undefined pathological origin or metastatic esophageal cancer; (5) esophagectomy after Endoscopical resection; (6) patients under 18; (7) perioperative mortality.

Before surgery, all participants were histopathologically assessed to define esophageal cancer after endoscopic biopsies. Positive LN metastasis in CT was defined as having at least one enlarged lymph node with a short-axis dimension of ≥1m. The 8th edition AJCC/UICC staging system of esophageal cancer was applied.15 Tumor sizes were determined as the maximum diameter in two dimensions, measured by Vernier’s calipers. Location (L) is defined as the position of the epicenter of the tumor. If no statement of epicenter was provided, the following measurements were applied: (1) upper: 15–24cm from incisors; (2) middle: 25–29cm from incisors; (3) lower: 30–40/45cm from incisors. Histologic grade (G) was categorized as well-differentiated (G1), moderately differentiated (G2) and poorly differentiated (G3). Macroscopic tumor type was classified using the 2016 Japanese Classification of Esophageal Cancer, 11th Edition.16

General clinical features were documented. A past habit of alcohol taking means taking in at least 60g of ethanol per day for men and at least 40g for women within the past five years of cancer diagnosis, as defined by WHO and the European Medicines Agency.17 The pathologic diagnosis of esophageal squamous cell carcinoma was performed by two experienced pathologists who assessed the surgically resected specimens independently. Pathologic features were recorded, including tumor size, invasion depth, microscopic type, histologic grade, lymph-vessel invasion (LVI), LN status (LN metastasis and the location of the metastasis). Invasion depth was divided into four categories: epithelium (EP)/lamina propria mucosa (LPM), muscularis mucosa (MM), submucosal (SM)1, SM2 or deeper. Invasion depth and LVI was further confirmed by immunohistochemical staining.

Statistical Analysis

Significant predictors for establishing ANN were identified using IBM SPSS Statistics for Windows, Version 23.0 (SPSS, Chicago, IL). Pearson chi-square test or Fisher’s exact test was applied in dichotomous variables to identify independent risk factors of LN metastasis in SESCC. Variables significantly associated with LN metastasis (p ≤.05) were identified as candidate for multivariate logistic regression. The optimal cutoff value was assessed by Youden index in the receiver operating characteristic (ROC) curve. The area under receiver operating characteristic curves (AUC) was calculated to evaluate the diagnostic accuracy and was compared by the DeLong’s test. The net reclassification improvement (NRI) and the integrated discrimination improvement (IDI) were calculated to quantify the refinement in predictive accuracy. The lasso regression, NRI, IDI and ROC were conducted with R software (version 3.5.1, http://www.R-project.org/). The packages used in R are listed in Supplementary text S1.

Establishment of the Predictive ANN Model

A pattern recognition ANN was established by using the “nprtool” for pattern recognition neural network in Matlab 2019a (MathWorks Institute, USA). This neural network contains three kinds of layers: the input layer, hidden layer, and output layer. A backpropagation Levenberg-Marquardt algorithm (trainlm) from the MATLAB Neural Network Toolbox was applied to estimate the error for the output layer, as well as for each of the trained network neuron, and for neurons’ correction weights following current values. The input layer primarily contained the nonlinear neurons and organized them into a feed-forward multi-layer structure. The activation function was set to be hyperbolic tangent sigmoid transfer function. The performance indicator used was Means Squared Error (MSE). As the training process was stopped at maximum validation error, the toolbox returned the best-validated epoch and its performance.18 We examined the Receiver Operating Characteristic (ROC) plots and Confusion plots to check the best-validated epoch.

Results

Baseline Characteristics

A total of 9043 lymph nodes were resected from 733 patients by surgery (513 male and 220 female; median age: 62.8; range: 34–83). The overall incidence of lymph node metastases was 18.1% (133/733). The average tumor size was 1.71±1.01cm (range: 0.3–7.0cm). In the T1a stage (242 patients, 33.1% of the cases), the number of patients with EP/LPM and MM invasion was 128 (17.4%) and 114 (15.6%), respectively. In the T1b stage (491 patients, 67.0%), the number of patients with SM1 and SM2 or deeper invasion was 382 (52.1%) and 109 (14.9%), respectively. The LN metastasis rates of T1a and T1b tumors were 2.48% (6/242), 25.9% (127/491), respectively. The accuracy was calculated to be 66.4% for CT with a sensitivity of 51.9% and specificity of 69.7%.

Candidate Predictors

Table 1 summarizes the characteristics of patients according to lymph node status. Patients’ age, sex, history of smoking, family history of GI tumors, tumor location, and macroscopic type were not significantly associated with LN metastases. The optimal applicable cut-off value of tumor size was defined as 1.85cm by calculating Youden’s index. The area under the ROC curve was 0.62 (95% confidence interval [CI], 0.565–0.679).

Table 1 Characteristics of Patients According to Lymph Node Status

The multivariate logistic regression and lasso regression (Figure 1) both identified the same risk factors as follows: (1) past habit of alcohol taking, (2) size≥1.85cm, (3) submucosal invasion, (4) poorly differentiated histologic, (5) positive lymph-vessel invasion, and (6) positive image results (Table 1).

Figure 1 Selection of candidate predictors by LASSO regression. (A) Identification of the tuning parameter (λ) by 10-fold cross-validation on the basis of minimum criteria. Binomial deviance was plotted as a function of log(λ) from cross-validation procedure. The y-axis represents the binomial deviance, and the lower x-axis represents the log(λ). The numbers listed in the upper x-axis indicates the number of selected candidate predictors corresponding to a different λ value. The red dots stands for average deviance values of each model when given a certain λ value, and vertical bars through the red dots show the upper and lower values of the deviances. The black dotted lines determine the optimal λ values via the minimum criteria and the 1 standard error of the minimum criteria (the 1-SE criteria). The optimal λ value of 0.023 with log (λ) = - 3.63 was finally determined. (B) LASSO coefficient profiles of the twelve candidate predictors. The black dotted vertical line was drawn at the value selected using 10-fold cross-validation in Figure 1A. The optimal λ value yielded six candidate features with nonzero coefficients.

The LR Model

The predictor variables of the ultimate LR model were tumor size≥1.85cm, submucosal invasion, poorly differentiated histologic grade, a past habit of alcohol taking, positive lymph-vessel invasion, as well as positive CT results. The ultimate LR equation is as follows: Probability of metastasis=1/(1+e–z), where z= 2.061*Invasion depth (EP/LPM labels and MM labels 0, SM1/SM2 or deeper labels 1) +3.216*LVI (negative labels 0, positive labels 1) + 0.956*Histologic grade (G1 and G2 labels 0, G3 labels 1) +1.107*CT-Results (negative labels 0, positive labels 1) + 0.594*Alcohol taking (no labels 0, yes labels 1) + 1.327*Tumor Size (<1.85cm labels 0, ≥1.85cm labels 1) −5.213, where “e” represents the natural base. According to the Youden index, the cut-off point of the model was a value of 3.735.

The ANN Model

Figure 2 presents the structure of the established neural network. The input layer consists of six parameters: tumor size, invasion depth, a past habit of alcohol use, histologic grade, lymph-vessel invasion, and image results. The hidden layer consists of 20 neurons. The two output layers represented positive LN metastasis and negative LN metastasis, respectively. The performance plot shows the best validation performance at epoch 6 with a mean squared error of 0.0432, which indicates an excellent performance of the ANN model (Supplementary Figure S1). Cross-validation was performed in the primary data-set. The default ratios were 0.7/0.15/0.15 in the training, validation, and testing groups. These groups were randomly allocated by the nprtool in the Matlab. The validation yielded good discrimination with all the areas under curve exceeding 0.85 (training set 0.904, validation set 0.938, testing set 0.960, and overall set 0.915; Figure 3).

Figure 2 Establishment of an Artificial Neural Network Model. A pattern recognition ANN model was generated. This non-parametric model consisted of 3 layers. The input layer consists of six parameters: tumor size, invasion depth, a past habit of alcohol use, histologic grade, lymph-vessel invasion, and preoperative CT results. The hidden layer consists of 20 neurons. The two output layers represented positive LN metastasis and negative LN metastasis, respectively.

Figure 3 ROC curves of the established models. The blue curve and the green dotted curve represent the development data of the ANN model and the LR model, respectively. AD represents the C-index of the training group, the validation group, the testing group, and the whole group, respectively.

Comparisons ANN Model with LR Model

The gold standard was the pathologically confirmed LN metastasis by surgical resection. The sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy were further compared between ANN and LR models. Table 2 shows the classification distribution of the two models.

Table 2 The Classification of the Established Models

According to DeLong’s test, the ANN model was superior to the LR model in specificity (91.20% vs 72.59%, p=0.006), PPV (56.49% vs 39.78%, p=0.020), accuracy (90.72% vs 74.49%, p<0.0001), and C-index (0.915 vs 0.868, p<0.001). There was no difference between the two models in Sensitivity (87.06% vs 83.21%, p=0.764) and NPV (98.17% vs 95.21%, p=0.627).

To further quantify refinement in predictive accuracy, NRI and IDI were applied. Although NRI was not significantly different between the two models (improved by −1.1%, z=−0.222, p=0.824), IDI indicated that the ANN was significantly improved by 23.3% in prediction performance when compared with the LR model (z=4.338, p<0.001). The comparisons of the diagnostic assessment are listed in Table 3. Table 4 summarizes the model performance measures.

Table 3 Comparison of ANN Model and LR Model for Predicting LN Metastasis

Table 4 Summary of Model Performance Measures

Discussion

This study presented the first established pattern recognition neural network for predicting LN metastasis in SESCC. The model was developed with the largest sampling pool that we know of: the number of patients was twice greater than the previously published model data,19,20 and a larger sample size generally leads to increased precision of the model.

The significance of our established model is to provide indications of additional treatment after endoscopic resection procedures. At present, the necessity of further treatment after ESD is still controversial. Some studies showed that ESD alone could not be considered curative for patients with T1b tumors due to a higher risk for lymph node metastasis.21,22 However, Takeuchi et al have recently reported that the diagnostic ESD for cM3-SM2 esophageal cancer was feasible and safe. Approximately 20% of these patients can potentially avoid further esophagectomy after endoscopic treatment.23 Although we collected clinic-pathological variables (tumor size, submucosal invasion, histologic grade, and lymph-vessel invasion) from pathological results after surgical resection, they are also available after routine ESD procedures by histologic analysis of the endoscopically resected specimens. Thus, using the established model, endoscopists can determine whether patients need additional treatment after ESD procedures. Another potential application of the model is to predict LN metastasis even before ESD. During routine pre-ESD examinations, we can obtain data of tumor size, submucosal invasion, and histologic grade by endoscopic ultrasound (EUS), magnifying endoscopy with narrow-band imaging (ME-NBI), and endoscopic biopsies, respectively.24,25 Nevertheless, the accuracy of these preoperative examinations is still under investigation. Besides, LV invasion can only be precisely confirmed after ESD by analyzing endoscopically resected specimens.26 Thus, our model may not be applied directly to predict LN metastasis before ESD. Another model should be developed for identifying preoperative LN metastasis using data from available pre-ESD examinations.

The risk assessment for a certain disease relies on predictive models that simultaneously integrate both clinically and statistically significant elements.27 In this study, we identified six parameters as the main independent risk factors of LN metastasis in SESSC. Among these factors, tumor size, invasion depth, histologic grade, and lymph-vessel invasion have already been reported and incorporated in previously published models.12,19,28 Our findings are consistent with the results of previous models. Different from those reported models, we identified another two independent risk factors of LN metastasis in SESSC. Firstly, we identified the preoperative CT as a significant predictor for LN metastasis. CT is one of the imaging modalities for pretreatment assessment of LN metastasis, with a sensitivity of 50% (range, 41–60%) and an accuracy of 63% (range, 53–72%).29,30 In our study, the accuracy of CT was 66.4% with a sensitivity of 51.9%, which is consistent with the previous studies. As an affordable test, CT is also readily available for most patients before treatment. The possibility of inaccurate prediction may due to normal-sized nodes that contain metastatic deposits and benign nodal enlargement arise from inflammation. For the above reasons, LN metastasis cannot be determined by the size evaluated by CT alone. Therefore, we determine CT as one of the six candidate predictors. Besides, we identified the past habit of alcohol taking as a significant predictive factor of LN metastasis. Alcohol has long been considered as a risk factor for development of esophageal cancer.31 Huang et al showed that alcohol taking increased the death hazard of esophageal cancer, and the hazardous effects increase with a dose-dependent manner.32 We found an increased risk of LN metastasis in alcoholic drinkers compared with non-drinkers, and drinkers in our study tends to have more alcohol consumption behavior. Lind et al pointed out that both alcohol consumption behaviour and alcohol dependence status are influenced by a gene called ALDH-1A1.33 This gene is also reported to be highly expressed in human esophageal squamous cell carcinoma, which is significantly associated with lymph node metastasis and poor survival.34 However, there is still no research directly investigating the relationships among alcohol consumption, ALDH-1A1 expression, and lymph node metastasis of esophageal cancer. Further studies are needed to determine whether alcohol consumption leads to a poor prognosis of esophageal cancer by triggering LN metastasis with the influence of ALDH-1A1 expression.

This ANN model showed satisfactory performance through internal cross-validation. The methods of assessing diagnostic test accuracy were further used for model comparison. It is widely known that logistic regression is a simple machine learning algorithm used for binary classification tasks. Although kernelized variants exist, the standard LR model is a linear classifier, which is useful for a dataset where the classes are “linearly separable”. Neural networks are somewhat related to logistic regression. If a logistic algorithm belongs to the generalized linear regression, a neural network can be called a kind of generalized logistic regression.35,36 LR can also be considered as a one-layer neural network without hidden neurons. The neural network has advantages over LR models: the hidden layers facilitate the discovery of more complex and non-linear associations of variables. In this study, we found the ANN model performed better than the LR model, with significantly higher AUC, specificity, PPV, and Accuracy. High specificity means that the ANN model is superior to LR model in reducing the misdiagnosis rate (increasing the true-negative rate of LN metastasis), which is very helpful for clinicians to broaden the indication of endoscopic resection. IDI is another popular tool for evaluating the capacity of a diagnostic test to classify binary outcomes.37 The principle for the IDI is that a better model leads to increased estimated risks of LN metastasis for cases and decreased estimated risks for controls.38 The ANN model outperformed the LR by 23.3%, when assessed by IDI. The larger IDI, the more improved performance of the new model. A potential reason for the better performance of ANN is that the actual algorithm for predicting LN status follows a more complex nonlinear relationship, which can be handled more accurately by ANN than LR. These two models had no statistically significant difference in NPV, which is also an important index for clinicians to know how certain the negative results of LN metastasis predicted by the model are. Both models attained a high NPV. In other words, both models can help clinicians to exclude LN metastasis with high certainty if a patient has a negative predicting result. Thus, patients with negative predicting results may be indicated for endoscopic resection more accurately. NRI is another statistical index commonly used to assess whether one model provides clinically relevant prediction improvements than the other.39 We calculated an NRI of −1.1% with no statistical significance (p>0.05). It implies that no significant improvements in the ANN model for correcting mistakenly classified patients with or without LN metastasis by the LR model. Thus, the classification abilities of both two models are comparative.

The model established in this study is still preliminary. The ANN model was developed from a single-center retrospective cohort; thus, the external validation outside our hospital is further needed to test model performance. The current pattern recognition ANN belongs to a shallow neural network. We expect more improved ANN models, with multiple hidden layers, that can be established by a complicated deep learning algorithm in the future. Other aspects of the model can be explored for improving the model performance, such as adding new candidate predictors, conducting prospective cross-validation to avoid possible bias in other cancer-related factors, or individual factors.

Conclusion

In conclusion, we developed a novel ANN model to predict LN metastasis in patients with SESCC. The ANN model was comparable with the LR model in Sensitivity, NPV, and NRI, and it performed better in Specificity, PPV, Accuracy C-index, and IDI. Therefore, the ANN model is superior to the LR model and may become a valuable tool, especially for providing indications of additional treatment after ESD procedures.

Disclosure

The authors report no conflicts of interest for this work.

References

1. Chen W, Zheng R, Baade PD, et al. Cancer statistics in China, 2015. CA Cancer J Clin. 2016;66(2):115–132. doi:10.3322/caac.21338

2. Lagergren J, Lagergren P. Recent developments in esophageal adenocarcinoma. CA Cancer J Clin. 2013;63:232–248.

3. Tramontano AC, Nipp R, Mercaldo ND, et al. Survival disparities by race and ethnicity in early esophageal cancer. Dig Dis Sci. 2018;63(11):2880–2888. doi:10.1007/s10620-018-5238-6

4. Wei WQ, Chen ZF, He YT, et al. Long-term follow-up of a community assignment, one-time endoscopic screening study of esophageal cancer in China. J Clin Oncol. 2015;10(33):1951–1957. doi:10.1200/JCO.2014.58.0423

5. Jacobs M, Macefield RC, Elbers RG, et al. Meta-analysis shows clinically relevant and long-lasting deterioration in health-related quality of life after esophageal cancer surgery. Qual Life Res. 2014;23(4):1097–1115. doi:10.1007/s11136-013-0545-z

6. Berger A, Rahmi G, Perrod G, et al. Long-term follow-up after endoscopic resection for superficial esophageal squamous cell carcinoma: a multicenter western study. Endoscopy. 2019;51(04):298–306. doi:10.1055/a-0732-5317

7. Qi ZP, Chen T, Li B, et al. Endoscopic submucosal dissection for early esophageal cancer in elderly patients with relative indications for endoscopic treatment. Endoscopy. 2018;50(09):839–845. doi:10.1055/a-0577-2560

8. Tsujii Y, Nishida T, Nishiyama O, et al. Clinical outcomes of endoscopic submucosal dissection for superficial esophageal neoplasms: a multicenter retrospective cohort study. Endoscopy. 2015;47(09):775–783. doi:10.1055/s-0034-1391844

9. Hassanipour S, Ghaem H, Arab-Zozani M, et al. Comparison of artificial neural network and logistic regression models for prediction of outcomes in trauma patients: a systematic review and meta-analysis. Injury. 2019;50(2):244–250. doi:10.1016/j.injury.2019.01.007

10. Fei Y, Gao K, Li WQ. Prediction and evaluation of the severity of acute respiratory distress syndrome following severe acute pancreatitis using an artificial neural network algorithm model. HPB (Oxford). 2019;21(7):891–897. doi:10.1016/j.hpb.2018.11.009

11. Eslamizadeh G, Barati R. Heart murmur detection based on wavelet transformation and a synergy between artificial neural network and modified neighbor annealing methods. Artif Intell Med. 2017;78:23–40. doi:10.1016/j.artmed.2017.05.005

12. Zhou Y, Du J, Wang Y, et al. Prediction of lymph node metastatic status in superficial esophageal squamous cell carcinoma using an assessment model combining clinical characteristics and pathologic results: a retrospective cohort study. Int J Surg. 2019;66:53–61. doi:10.1016/j.ijsu.2019.04.014

13. Zheng H, Tang H, Wang H, et al. Nomogram to predict lymph node metastasis in patients with early oesophageal squamous cell carcinoma. Br J Surg. 2018;105(11):1464–1470. doi:10.1002/bjs.10882

14. Min BH, Yang JW, Min YW, et al. Nomogram for prediction of lymph node metastasis in patients with superficial esophageal squamous cell carcinoma. J Gastroenterol Hepatol. 2019;31.

15. Rice TW, Patil DT, Blackstone EH. 8th edition AJCC/UICC staging of cancers of the esophagus and esophagogastric junction: application to clinical practice. Ann Thorac Cardiovasc Surg. 2017;6(2):119–130. doi:10.21037/acs.2017.03.14

16. Japan Esophageal Society. Japanese classification of esophageal cancer, 11th edition: part Ⅱ and ш. Esophagus. 2017;14(1):37–65. doi:10.1007/s10388-016-0556-2

17. Schwarzinger M, Pollock BG, Hasan OSM, et al. Contribution of alcohol use disorders to the burden of dementia in France 2008–13: a nationwide retrospective cohort study. Lancet Public Health. 2018;3(3):e124–e132. doi:10.1016/S2468-2667(18)30022-7

18. Vastrad C, Vastrad CM. Performance analysis of neural network models for oxazolines and oxazoles derivatives descriptor dataset. IJIST. 2013;3(6):1–15.

19. Ancona E, Rampado S, Cassaro M, et al. Prediction of lymph node status in superficial esophageal carcinoma. Ann Surg Oncol. 2008;15(11):3278–3288. doi:10.1245/s10434-008-0065-1

20. Ma DW, Jung DH, Kim JH, et al. Predicting lymph node metastasis for endoscopic resection of superficial esophageal squamous cell carcinoma. J Thorac Cardiovasc Surg. 2019;157(1):397–402.e1. doi:10.1016/j.jtcvs.2018.07.034

21. Park J, Youn Y, Park J, et al. Clinical outcomes of endoscopic submucosal dissection for superficial esophageal squamous neoplasms. Clin Endosc. 2016;49(2):168–175. doi:10.5946/ce.2015.080

22. Suzuki G, Yamazaki H, Aibe N, et al. Endoscopic submucosal dissection followed by chemoradiotherapy for superficial esophageal cancer: choice of new approach. Radiat Oncol. 2018;13(1):246. doi:10.1186/s13014-018-1195-7

23. Takeuchi M, Suda K, Hamamoto Y, et al. Technical feasibility and oncologic safety of diagnostic endoscopic resection for superficial esophageal cancer. Gastrointest Endosc. 2018;88(3):456–465. doi:10.1016/j.gie.2018.04.2361

24. Yu T, Geng J, Song W, Jiang Z. Diagnostic accuracy of magnifying endoscopy with narrow band imaging and its diagnostic value for invasion depth staging in esophageal squamous cell carcinoma: a systematic review and meta-analysis. Biomed Res Int. 2018;20:8591387.

25. ASGE Standards of Practice Committee, Evans JA, Early DS, Chandraskhara V, et al. The role of endoscopy in the assessment and treatment of esophageal cancer. Gastrointest Endosc. 2013;77(3):328–334. doi:10.1016/j.gie.2012.10.001.

26. Yamada T, Sugiyama H, Ochi D, et al. Risk factors for submucosal and lymphovascular invasion in gastric cancer looking indicative for endoscopic submucosal dissection. Gastric Cancer. 2014;17(4):692–696. doi:10.1007/s10120-013-0323-1

27. Alba AC, Agoritsas T, Walsh M, et al. Discrimination and calibration of clinical prediction models: users’ guides to the medical literature. JAMA. 2017;318(14):1377–1384. doi:10.1001/jama.2017.12126

28. Xue L, Ren L, Zou S, et al. Erratum: parameters predicting lymph node metastasis in patients with superficial esophageal squamous cell carcinoma. Mod Pathol. 2015;28(1):161. doi:10.1038/modpathol.2014.133

29. Choi J, Kim SG, Kim JS, et al. Comparison of endoscopic ultrasonography (EUS), positron emission tomography (PET), and computed tomography (CT) in the preoperative locoregional staging of resectable esophageal cancer. Surg Endosc. 2010;24(6):1380–1386. doi:10.1007/s00464-009-0783-x

30. Van Vliet EP, Heijenbrok-Kal MH, Hunink MG, et al. Staging investigations for oesophageal cancer: a metaanalysis. Br J Cancer. 2008;98:547–557.

31. Lagergren J, Smyth E, Cunningham D, et al. Esophageal cancer. Lancet. 2017;25(390):2383–2396. doi:10.1016/S0140-6736(17)31462-9

32. Huang Q, Luo K, Yang H, et al. Impact of alcohol consumption on survival in patients with esophageal carcinoma: a large cohort with long-term follow-up. Cancer Sci. 2014;105(12):1638–1646. doi:10.1111/cas.12552

33. Lind PA, Eriksson CJ, Wilhelmsen KC. The role of aldehyde dehydrogenase-1 (ALDH1A1) polymorphisms in harmful alcohol consumption in a Finnish population. Hum Genomics. 2008;3(1):24–35. doi:10.1186/1479-7364-3-1-24

34. Wang Y, Zhe H, Gao P, et al. Cancer stem cell marker ALDH1 expression is associated with lymph node metastasis and poor survival in esophageal squamous cell carcinoma: a study from high incidence area of northern China. Dis Esophagus. 2012;25(6):560–565. doi:10.1111/j.1442-2050.2011.01279.x

35. Duda R, Hart P, Stork D. Pattern Classification. 2nd ed. New York: Wiley/Interscience; 2001.

36. Dreiseitl S, Ohno-Machado L. Logistic regression and artificial neural network classification models: a methodology review. J Biomed Inform. 2002;35(5–6):352–359. doi:10.1016/S1532-0464(03)00034-0

37. Pencina MJ, D’Agostino RB Sr., D’Agostino RB, et al. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27(2):157–172. doi:10.1002/sim.2929

38. Kerr KF, McClelland RL, Brown ER, Lumley T. Evaluating the incremental value of new biomarkers with integrated discrimination improvement. Am J Epidemiol. 2011;174(3):364–374. doi:10.1093/aje/kwr086

39. Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128–138. doi:10.1097/EDE.0b013e3181c30fb2

Creative Commons License © 2020 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.