Prognostic Value of Combined Detection of Preoperative Albumin-to-Fibrinogen Ratio and Neutrophil-to-Lymphocyte Ratio in Operable Esophageal Squamous Cell Carcinoma Patients without Neoadjuvant Therapy
Authors Zheng Z, Lin D, Chen Q, Zheng B, Liang M, Chen C, Zheng W
Received 10 December 2020
Accepted for publication 3 February 2021
Published 12 March 2021 Volume 2021:13 Pages 2359—2370
Checked for plagiarism Yes
Review by Single anonymous peer review
Peer reviewer comments 2
Editor who approved publication: Dr Eileen O'Reilly
Zhiyuan Zheng,1,2,* Donghong Lin,2,* Qiaoqian Chen,2,* Bin Zheng,1 Mingqiang Liang,1 Chun Chen,1 Wei Zheng1
1Department of Thoracic Surgery, Fujian Medical University Union Hospital, Fuzhou, Fujian, 350001, People’s Republic of China; 2Medical Technology and Engineering College of Fujian Medical University, Fuzhou, Fujian, 350004, People’s Republic of China
*These authors contributed equally to this work
Correspondence: Wei Zheng; Chun Chen
Department of Thoracic Surgery, Fujian Medical University Union Hospital, No. 29 Xinquan Road, Fuzhou, 350001, People’s Republic of China
Tel +86 591-83357896
Fax +86 591-87113828
Email [email protected]; [email protected]
Background: We retrospectively analyzed the prognostic value of the albumin-to-fibrinogen ratio (AFR)–neutrophil-to-lymphocyte ratio (NLR) score, comprising preoperative AFR and NLR, in esophageal squamous cell carcinoma (ESCC) patients after radical resection.
Patients and Methods: Overall, 215 patients were included. The optimal cutoff value was determined using the receiver operating characteristic (ROC) curve. Based on a low AFR (< 12.06) and high NLR (≥ 1.78), the AFR–NLR score was classified as 2 (both hematological abnormalities present), 1 (one abnormality present), or 0 (both abnormalities absent). Kaplan–Meier curves, Cox regression, and predicted nomogram were used to evaluate the prognostic value of the score.
Results: The prognostic value of the AFR–NLR score was better than that of AFR or NLR alone (P < 0.05). Multivariate analysis showed that a high AFR–NLR score was an independent predictor of poor prognosis for overall survival (P < 0.001). Additionally, in the nomogram including the AFR–NLR score, the net reclassification improvement index increased by 35.5% (P < 0.001), and the integrated discrimination improvement index increased by 9.0% (P < 0.001). The predictive accuracy of the established nomogram model was proved using Harrell’s concordance index (0.811, 95% confidence interval: 0.765– 0.856) and calibration curve. Notably, the decision analysis curve showed that the nomogram had a higher net benefit within most of the threshold probability range, indicating better clinical applicability.
Conclusion: The AFR–NLR score is a useful predictor of the prognosis of ESCC patients after radical resection, and the nomogram established on the basis of this score has a good prognostic value.
Keywords: esophageal squamous cell cancer, AFR–NLR score, prognosis, nomogram
Esophageal cancer is one of the main diseases treated with thoracic surgery, and the number of diagnoses and deaths due to esophageal cancer rank seventh and sixth, respectively, among all cancers in the world.1 In China, esophageal squamous cell carcinoma (ESCC) is the main type of esophageal cancer with atypical clinical symptoms. Most patients are diagnosed with moderate and advanced disease with poor prognosis.2
At present, the TNM stage is the main clinical indicator for evaluating the prognosis of ESCC patients. However, the prognosis of patients with the same stage receiving the same treatment is often individualized. Therefore, it is very important to identify economic and practical blood indicators that predict patient survival.
Since Virchow first discovered the relationship between inflammation and malignancy,3 there has been increasing evidence that tumor progression is not only related to the intrinsic properties of tumor cells but also depends on changes in the tumor microenvironment.4 Hyperfibrinogenemia is frequently observed in patients with malignant tumors and has been shown to be associated with tumor progression, invasion, and metastasis.5 The level of serum albumin, the main serum protein and an indicator of poor nutritional status, has been confirmed to decrease in patients with malignant tumors and is related to poor prognosis.6,7 Inflammation associated with malignant tumors has been confirmed to play an important role in the pathogenesis and metastasis of the disease, while inflammation-related immune cells are biomarkers of systemic inflammation, and their levels can reflect the degree of chronic inflammation in patients.8,9 However, previous studies of these factors have only focused on the interaction of one or two blood indicators. Therefore, we hypothesized that a new scoring system constructed by taking into account changes in multiple blood indicators in the tumor microenvironment may have a higher prognostic value than that of a single blood indicator.
The medical records of 215 ESCC patients who underwent radical resection at the Department of Thoracic Surgery at Fujian Medical University Union Hospital from January 2013 to December 2015 were examined. The inclusion criteria were as follows: no other related anti-tumor treatments were performed before surgery; no other tumors or distant metastases were found in preoperative inspections or intraoperative exploration; no obvious signs of infection before surgery; no vital organ dysfunction; and clinical case data and follow-up records were complete. The exclusion criteria were as follows: previous history of autoimmune disease, blood disease, or chronic inflammatory disease; patients receiving anticoagulant or hormone therapy within 1 month before surgery; and severe complications or death during the perioperative period. The study was approved by the Ethics Committee of Fujian Medical University Union Hospital (2020KY064) and the related research activities strictly followed the Helsinki Declaration. All participants gave informed consent before being included in the study.
Clinical Data Collection
Sex, age, smoking history, drinking history, hypertension, diabetes, TNM stage, tumor site, tumor length, tumor differentiation degree, and postoperative adjuvant therapy were all obtained from medical records. Smoking history was defined as smoking more than 100 cigarettes in their lifetime.10 Drinking history was defined as long-term regular drinking of up to 20 g/day (140 g/week) and up to 10 g/day (70 g/week) in men and women, respectively.11 Peripheral venous blood samples of the patients were collected by a professional examiner within 1 week of admission and sent for examination. Routine blood tests were performed using the Sysmex XN series automatic modular blood fluid analyzer, routine biochemical examination was performed using the Roche Cobas-8000 automatic biochemical analyzer, and blood coagulation function tests were performed using the Stago automatic blood coagulometer. All ESCC patients were diagnosed and confirmed by histopathological examination in accordance with the 8th edition of the American Joint Committee on Cancer TNM staging system.12 Postoperative adjuvant therapy starts within 2–6 weeks after surgery. The regimens of the adjuvant chemotherapy were platinum-based. As for radiation therapy, the patients received conventional radiotherapy at 45–60 Gy with the daily fraction being 1.8–2 Gy.
Calculation and Definition of Relevant Blood Indicators
Based on previous studies, the albumin-to-fibrinogen ratio (AFR) was calculated as the ratio of the absolute albumin and absolute fibrinogen values.13 The neutrophil-to-lymphocyte ratio (NLR) was calculated as the ratio of the absolute neutrophil count to the absolute lymphocyte count.14 The platelet-to-lymphocyte ratio (PLR) was calculated as the ratio of the absolute platelet count to the absolute lymphocyte count.15 The lymphocyte-to-monocyte ratio (LMR) was calculated as the ratio of the absolute lymphocyte count to the absolute monocyte count.16 The prognostic nutrition index (PNI) was calculated as the 10-fold albumin level (g/dL) plus 0.005-times the total lymphocyte count (per mm3).17 The Youden index was calculated using receiver operating characteristic (ROC) analysis to determine an optimal cutoff value for AFR, NLR, PLR, LMR, and PNI in survival analysis. The following optimal cutoff values were identified: AFR, 12.06 (sensitivity: 74.6%, specificity: 53.5%, area under the curve: 0.656); NLR, 1.78 (sensitivity: 78.9%, specificity: 50.0%, area under the curve: 0.652); PLR, 120.6 (sensitivity: 60.6%, specificity: 61.8%, area under the curve: 0.589); LMR, 6.35 (sensitivity: 91.5%, specificity: 25.7%, area under the curve: 0.571); and PNI, 49.93 (sensitivity: 56.0%, specificity: 66.0%, area under the curve: 0.598).
The AFR–NLR score was defined as follows: a high NLR (≥1.78) and a low AFR (<12.06) were both given a score of 1; otherwise, they were scored 0. Individual scores were added to determine the AFR–NLR score (range: 0–2).
Follow-Up and Observation Indicators
Follow-up information was obtained mainly through telephone calls by medical staff or outpatient records (postoperative follow-up was conducted every 3 months for 2 years, and every 6 months from the 3rd year). The primary evaluation indicator was 3-year overall survival (OS). OS was defined as the time between the date of surgery and death of the patient or the last follow-up.
IBM SPSS 25.0 software, MedCalc 19.5 software and R 3.6.1 software were used for statistical analyses. Categorical variables were expressed as frequency and percentage. The optimal cutoff value was determined according to the ROC curve, and the conversion from continuous to categorical variables was performed. The area under the ROC curve was compared using the Z-test on MedCalc. Categorical variables were compared using the chi-square test or Fisher’s exact probability method. Survival curves were plotted using the Kaplan–Meier method and compared using the Log-rank test. Significant variables in univariate analysis were included in the Cox proportional hazard regression model for multi-factor survival analysis. P < 0.05 was considered statistically significant. A predicted prognostic nomogram was constructed using the independent prognostic factors. The accuracy and effectiveness of the nomogram were verified by calculating the net reclassification improvement index, integrated discrimination improvement index, Harrell’s concordance index (C-index), and drawing calibration and decision analysis curves.
The baseline characteristics of the included 215 patients are shown in Table 1. There were 154 men (71.6%) and 61 women (28.4%) with a median age of 60 (range: 34–80) years. The TNM staging was as follows: stage I, 51 patients (24.7%); stage II, 70 patients (32.1%); and stage III, 94 patients (43.2%). A total of 113 patients (52.6%) received postoperative adjuvant treatment (Table 1).
Table 1 Relationships Between the AFR–NLR Score and Clinicopathological Characteristics
The median preoperative AFR, NLR, PLR, LMR, and PNI values were 11.54 (range: 5.61–22.96), 1.93 (range: 0.63–4.36), 116.33 (range: 45.18–405.62), 4.44 (range: 1.1–11.72), and 51.15 (range: 39.6–64.8), respectively. The number of patients with AFR–NLR scores of 0, 1, and 2 were 43 (22.3%), 96 (44.7%), and 76 (33%), respectively.
Univariate and Multivariate Analyses
The median follow-up time was 44.7 (range: 1.6–73) months; 86 patients died due to disease progression during the follow-up period, with an average survival time of 24.77 months. The 1-, 2-, and 3-year OS of all patients were 92.5%, 79.1%, and 67.0%, respectively. Univariate analysis indicated that TNM stage, AFR, NLR, PLR, LMR, PNI, and postoperative adjuvant therapy was prognostic factors for OS. We performed a multivariate analysis of the significant factors in univariate analysis (P <0.05). Multivariate analysis indicated that TNM stage (P < 0.001), postoperative adjuvant therapy (P < 0.001), AFR (P = 0.014), and NLR (P = 0.012) was independent risk factors for OS (Table 2).
Table 2 Univariate and Multivariate Cox Regression Analyses of Overall Survival
Correlation Between AFR–NLR Score and Clinicopathological Characteristics
According to the area under the ROC curve and multivariate analysis results, we constructed the AFR–NLR score. AFR–NLR scores significantly correlated with age (P = 0.026), tumor length (P = 0.017), TNM stage (P < 0.001), AFR (P < 0.001), NLR (P < 0.001), PLR (P < 0.001), LMR (P < 0.001), and PNI (P < 0.001) (Table 1).
Diagnostic Efficacy and Prognostic Value of AFR–NLR Score
The ROC curves for AFR, NLR, PLR, LMR, PNI, and AFR–NLR score were plotted according to the optimal cutoff values for the prediction of patients’ survival. The following areas under the ROC curve were found: AFR–NLR score, 0.712 (95% confidence interval [CI]: 0.646–0.771); AFR, 0.641 (95% CI: 0.573–0.705); NLR, 0.644 (95% CI: 0.576–0.708); PLR, 0.612 (95% CI: 0.543–0.677); LMR, 0.586 (95% CI: 0.517–0.653); and PNI, 0.612 (95% CI: 0.543–0.677) (Figure 1A). We compared the area under the ROC curve for the AFR–NLR score with that for the AFR and NLR, and the results showed that the diagnostic value of the AFR–NLR score for the 3-year OS of patients with ESCC after radical resection was greater than that of a single indicator (P < 0.05) (Table 3).
Table 3 Comparison of AFR–NLR Score with Z Test of AFR and NLR
The 3-year OS in patients with AFR–NLR scores of 0, 1, and 2 was 90.7%, 73.9%, and 44.7%, respectively. We used the Kaplan–Meier method to plot the cumulative survival curve and compared them using the Log-rank test (Figure 1B). Univariate and multivariate analyses indicated that the AFR–NLR score was an independent predictor of OS (Table 4).
Table 4 Univariate and Multivariate Cox Regression Analysis of AFR–NLR Score
Construction and Calibration of Nomograms for OS
Independent predictive factors derived from a multivariate Cox risk regression model were used to establish a nomogram to predict OS in ESCC patients (Figure 2A). The survival probability of an individual can be calculated by adding the scores of the selected risk factors (Table 5).
Table 5 Points for Categorical Variables in Nomogram
The c-index for calculating the nomogram reached 0.811 (95% CI: 0.765–0.856), and the prediction model calibration curve also showed a high degree of coincidence between the predicted survival rate and actual rate of the nomogram. Both results suggest that the nomogram had good predictive accuracy in the modeling group (Figure 2B).
Incremental Predictive Value of AFR–NLR Score on Nomogram
The prediction model including the AFR–NLR showed significantly improved predictive performance, with its net reclassification improvement index (NRI) increased by 35.5% (P < 0.001) and integrated discrimination improvement index (IDI) increased by 9.0% (P < 0.001) (Table 6).
Table 6 Analysis of NRI and IDI on AFR–NLR Score
Clinical Application of Nomogram
The clinical validity of the nomogram was verified using decision curve analysis. The results showed that in most threshold probability ranges, the net benefit of the nomogram was consistently higher than that of other prediction models, indicating that the nomogram was superior to the traditional TNM staging system in predicting the survival of ESCC patients and had good clinical applicability (Figure 3).
In recent years, an increasing volume of evidence shows that tumor progression is not only related to the intrinsic properties of tumor cells but also depends on changes in the tumor microenvironment, which can be reflected by the changes in inflammatory cell counts and corresponding protein levels in serum.18–20
We collected data on relevant blood indicators of patients through the most widely used routine blood, routine biochemical, and coagulation function tests clinically. Based on previous relevant studies, the results were used to determine AFR, NLR, PLR, LMR, and PNI to explore the best indicators for predicting the prognosis of ESCC patients after radical resection.
According to previous studies, the cutoff values for relevant blood values were selected differently.21,22 Since an ROC curve can take into account the sensitivity and specificity of the diagnosis, we plotted it to obtain the Youden index to determine the optimal cutoff value.23 The area under the ROC curve for AFR and NLR was higher than that for PLR, LMR and PNI, suggesting that the former were more effective in predicting the survival of ESCC patients.
Univariate analysis showed that TNM stage, postoperative adjuvant therapy, AFR, NLR, PLR, LMR, and PNI were associated with prognosis. Multivariate analysis showed that TNM stage, postoperative adjuvant therapy, AFR, and NLR were independent risk factors for the prognosis of ESCC patients after radical resection. In view of the results of the ROC curve and multivariate analysis, we constructed a new AFR–NLR score that can simultaneously take into account the patients’ blood coagulation, nutrition, inflammation, and immunity indicators. To explore the diagnostic efficacy of the AFR–NLR score, we compared the AFR–NLR score with AFR and NLR using the Z-test. The results showed that the predictive value of the AFR–NLR score for the 3-year OS of ESCC patients was greater than that of single index detection using either AFR (P = 0.0047) or NLR (P = 0.0102) alone.24 In addition, we found that the AFR–NLR score significantly correlated with the tumor stage and duration of ESCC, suggesting that the AFR–NLR score can predict disease progression and tumor burden.25 The results of univariate and multivariate analyses suggest that the AFR–NLR score (all P <0.001) is an independent predictive factor for the prognosis of ESCC patients.
Nomograms are simple and visual prognostic prediction models that are gradually emerging in the current medical research and clinical practice; they have the advantages of simple use, easy operation, and high predictive accuracy. At present, nomograms are widely used in the prognostic assessment of many diseases, such as nasopharyngeal carcinoma and colorectal cancer.26,27 Since few nomogram prognostic models have been established for ESCC patients after radical resection, we constructed a corresponding model based on the results of the multivariate analysis in this study, which can directly reflect the degree of influence of different factors on the survival prediction for patients. Among them, TNM stage has the highest score in the model, suggesting that it contributed the most to the model. To explore the contribution of the AFR–NLR score to the model in this study, we calculated the NRI and IDI, respectively. The former is based on the quantification of the difference in classification change between two prediction models, while the latter is based on the quantification of the difference in probability between the two prediction models. NRI and IDI were used to calculate the increase in prediction probability for patients and decrease in prediction probability for controls.28,29 The results showed that NRI and IDI increased by 35.5% (P < 0.001) and 9.0% (P < 0.001), respectively, in the prediction model including the AFR–NLR score, and the prediction accuracy was significantly improved.
Note that the nomogram containing the AFR–NLR score showed sufficient discrimination in the primary cohort (C-index: 0.811).30 The established calibration curve also shows that the nomogram-predicted 3-year survival rate is highly consistent with the observed survival rate.31 However, good prediction accuracy is not necessarily related to effectiveness in clinical practice. Therefore, we conducted a decision curve analysis on the prediction model to observe whether the analysis improved prognosis prediction. It is a new method to determine which clinical decision has the greatest net benefit based on threshold probability. Based on the data in this study, we found that the constructed nomogram shows better performance, which indicates that the nomogram is superior to the traditional TNM staging system in clinical prognosis prediction.32,33
In this study, we established a new AFR–NLR score by comprehensively considering the changes in various indicators in the blood and explored its prognostic role in patients with ESCC after radical resection. The constructed nomogram provides a convenient and effective tool for patients and doctors to choose appropriate treatments and accurately predict prognosis. Another important aim is to select patients who might benefit from further care or other additional interventions. However, there are still some limitations to this study. First, as a single-center retrospective study, selection bias and recall bias are inevitable due to the limitations of the data. Second, we excluded patients who had received preoperative chemotherapy and/or radiotherapy, which may have influenced our analysis. Third, the follow-up time of this study was relatively short, so the effect of the AFR–NLR score on disease recurrence or metastasis could not be further explored. Finally, due to the lack of external data, our results were only validated internally. Therefore, a multicenter collaborative prospective study is needed to further confirm our results. In conclusion, this study proposes a nomogram including AFR–NLR score, which shows good performance in predicting OS of ESCC patients after radical resection, which is better than that of the traditional TNM staging system.
This study was supported by the Fujian Key Laboratory of Cardio-Thoracic Surgery (Fujian Medical University).
The authors report no conflicts of interest in this work.
1. Bray F, Ferlay J, Soerjomataram I, Siegel RL, Torre LA, Jemal A. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. 2018;68(6):394–424. doi:10.3322/caac.21492
2. Yang CS, Chen X, Tu S. Etiology and prevention of esophageal cancer. Gastrointest Tumors. 2016;3(1):3–16. doi:10.1159/000443155
3. Balkwill F, Mantovani A. Inflammation and cancer: back to Virchow. Lancet. 2001;357(9255):539–545. doi:10.1016/S0140-6736(00)04046-0
4. Alifano M, Mansuet-Lupo A, Lococo F, et al. Systemic inflammation, nutritional status and tumor immune microenvironment determine outcome of resected non-small cell lung cancer. PLoS One. 2014;9(9):e106914. doi:10.1371/journal.pone.0106914
5. Perisanidis C, Psyrri A, Cohen EE, et al. Prognostic role of pretreatment plasma fibrinogen in patients with solid tumors: a systematic review and meta-analysis. Cancer Treat Rev. 2015;41(10):960–970. doi:10.1016/j.ctrv.2015.10.002
6. Lindenmann J, Fink-Neuboeck N, Koesslbacher M, et al. The influence of elevated levels of C-reactive protein and hypoalbuminemia on survival in patients with advanced inoperable esophageal cancer undergoing palliative treatment. J Surg Oncol. 2014;110(6):645–650. doi:10.1002/jso.23711
7. Matsuda S, Takeuchi H, Kawakubo H, et al. Prognostic impact of change in the fibrinogen and albumin score during preoperative treatment in esophageal cancer patients. World J Surg. 2017;41(11):2788–2795. doi:10.1007/s00268-017-4074-8
8. Muroya T, Wajima N, Ogasawara H, et al. [The impact of neutrophil-lymphocyte ratio on the prognosis in patients who underwent esophagectomy for esophageal cancer]. Gan to Kagaku Ryoho. 2017;44(10):912–914. Japanese.
9. Yodying H, Matsuda A, Miyashita M, et al. Prognostic significance of neutrophil-to-lymphocyte ratio and platelet-to-lymphocyte ratio in oncologic outcomes of esophageal cancer: a systematic review and meta-analysis. Ann Surg Oncol. 2016;23(2):646–654. doi:10.1245/s10434-015-4869-5
10. Regueiro M, Kip KE, Cheung O, Hegazi RA, Plevy S. Cigarette smoking and age at diagnosis of inflammatory bowel disease. Inflamm Bowel Dis. 2005;11(1):42–47. doi:10.1097/00054725-200501000-00006
11. Farrell GC, Chitturi S, Lau GK, Sollano JD; Asia-Pacific Working Party on NAFLD. Guidelines for the assessment and management of non-alcoholic fatty liver disease in the Asia-Pacific region: executive summary. J Gastroenterol Hepatol. 2007;22(6):775–777. doi:10.1111/j.1440-1746.2007.05002.x
12. Donohoe CL, Phillips AW. Cancer of the esophagus and esophagogastric junction: an 8th edition staging primer. J Thorac Dis. 2017;9(3):E282–E284. doi:10.21037/jtd.2017.03.39
13. Hamanaka K, Miura K, Koyama T, et al. The pretreatment circulating albumin-to-fibrinogen ratio in patients with non-small cell lung cancer: a simple, economical, and effective biomarker. J Thorac Dis. 2019;11(Suppl 3):S241–S244. doi:10.21037/jtd.2019.01.83
14. Guo J, Chen S, Chen Y, Li S, Xu D. Combination of CRP and NLR: a better predictor of postoperative survival in patients with gastric cancer. Cancer Manag Res. 2018;10:315–321. doi:10.2147/CMAR.S156071
15. Templeton AJ, Ace O, McNamara MG, et al. Prognostic role of platelet to lymphocyte ratio in solid tumors: a systematic review and meta-analysis. Cancer Epidemiol Biomarkers Prev. 2014;23(7):1204–1212. doi:10.1158/1055-9965.EPI-14-0146
16. Huang Y, Feng JF. Low preoperative lymphocyte to monocyte ratio predicts poor cancer-specific survival in patients with esophageal squamous cell carcinoma. Onco Targets Ther. 2015;8:137–145. doi:10.2147/OTT.S73794
17. Gao ZM, Wang RY, Deng P, et al. TNM-PNI: a novel prognostic scoring system for patients with gastric cancer and curative D2 resection. Cancer Manag Res. 2018;10:2925–2933. doi:10.2147/CMAR.S169206
18. Wuhao H, Shengguang W, Hua Z, Bin Z, Changli W. Prognostic significance of combined fibrinogen concentration and neutrophil-to-lymphocyte ratio in patients with resectable non-small cell lung cancer. Cancer Biol Med. 2018;15(1):88–96. doi:10.20892/j.issn.2095-3941.2017.0124
19. Silva TH, Schilithz A, Peres W, Murad LB. Neutrophil-lymphocyte ratio and nutritional status are clinically useful in predicting prognosis in colorectal cancer patients. Nutr Cancer. 2020;72(8):1345–1354. doi:10.1080/01635581.2019.1679198
20. McMillan DC. An inflammation-based prognostic score and its role in the nutrition-based management of patients with cancer. Proc Nutr Soc. 2008;67(3):257–262. doi:10.1017/S0029665108007131
21. Xue Y, Zhou X, Xue L, Zhou R, Luo J. The role of pretreatment prognostic nutritional index in esophageal cancer: a meta-analysis. J Cell Physiol. 2019;234(11):19655–19662. doi:10.1002/jcp.28565
22. Matsuda S, Takeuchi H, Kawakubo H, et al. Cumulative prognostic scores based on plasma fibrinogen and serum albumin levels in esophageal cancer patients treated with transthoracic esophagectomy: comparison with the Glasgow prognostic score. Ann Surg Oncol. 2015;22(1):302–310. doi:10.1245/s10434-014-3857-5
23. Agarwal S, Graepel T, Herbrich R, Har-Peled S, Roth D, Jordan MI. Generalization bounds for the area under the ROC curve. J Mach Learn Res. 2005;6(2):393–425.
24. Martínez-Camblor P, Carleos C, Corral N. Powerful nonparametric statistics to compare k independent ROC curves. J Appl Stat. 2011;38(7):1317–1332.
25. Gao QF, Qiu JC, Huang XH, et al. The predictive and prognostic role of a novel ADS score in esophageal squamous cell carcinoma patients undergoing esophagectomy. Cancer Cell Int. 2018;18:153. doi:10.1186/s12935-018-0648-2
26. Huang YQ, Liang CH, He L, et al. Development and validation of a radiomics nomogram for preoperative prediction of lymph node metastasis in colorectal cancer. J Clin Oncol. 2016;34(18):2157–2164. doi:10.1200/JCO.2015.65.9128
27. Li XH, Chang H, Xu BQ, et al. An inflammatory biomarker-based nomogram to predict prognosis of patients with nasopharyngeal carcinoma: an analysis of a prospective study. Cancer Med. 2017;6(1):310–319. doi:10.1002/cam4.947
28. Hilden J. Commentary: on NRI, IDI, and “good-looking” statistics with nothing underneath. Epidemiology. 2014;252):265–267. doi:10.1097/EDE.0000000000000063
29. Burch PM, Glaab WE, Holder DJ, Phillips JA, Sauer JM, Walker EG. Net reclassification index and integrated discrimination index are not appropriate for testing whether a biomarker improves predictive performance. Toxicol Sci. 2017;156(1):11–13. doi:10.1093/toxsci/kfw225
30. Van Oirbeek R, Lesaffre E. An application of Harrell’s C-index to PH frailty models. Stat Med. 2010;29(30):3160–3171. doi:10.1002/sim.4058
31. Wan G, Gao F, Chen J, et al. Nomogram prediction of individual prognosis of patients with hepatocellular carcinoma. BMC Cancer. 2017;17(1):91. doi:10.1186/s12885-017-3062-6
32. Vickers AJ, Cronin AM, Elkin EB, Gonen M. Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Med Inform Decis Mak. 2008;8:53. doi:10.1186/1472-6947-8-53
33. Van Calster B, Vickers AJ. Calibration of risk prediction models: impact on decision-analytic performance. Med Decis Making. 2015;35(2):162–169. doi:10.1177/0272989X14547233
This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.Download Article [PDF]