Development and Validation of a Comprehensive Model for Predicting Distant Metastasis of Solid Lung Adenocarcinoma: 3D Radiomics, 2D Radiomics and Clinical Features

Introduction

According to the Global Cancer Statistics Report 2020, lung cancer is the second most common cancer after breast cancer, and its mortality rate ranks first among all cancers.¹ Adenocarcinoma of the lung is the most common pathological form of lung cancer and occurs in the lung tissue distal to the bronchi of the lung segment. Most imaging manifestations are peripheral lung carcinomas, and early symptoms are often not obvious.² Pure ground-glass and partially solid lung adenocarcinomas rarely develop lymph node and distant metastases.^3,4 Lung adenocarcinomas with a ground glass component have excellent survival rates compared with solid lesions.^5,6 Solid lung adenocarcinomas are more prone to distant metastases and are usually unresectable, requiring chemoradiotherapy, targeted therapy, or immunotherapy.⁷ Therefore, accurately determining whether patients with solid lung adenocarcinomas have developed distant metastases prior to treatment is of great importance for clinical staging and treatment decisions.

Histopathologic examination is the gold standard for the diagnosis of distant metastases in lung cancer,⁸ but it is invasive and cannot be used for screening. Many non-invasive imaging modalities, such as computed tomography (CT), magnetic resonance imaging (MRI) and positron emission tomography/computed tomography (PET/CT), are widely used to evaluate distant metastases in lung cancer patients.^9,10 CT and MRI are the most commonly used clinical examination modalities for the diagnosis of distant metastases, but they often require systemic examination of the head, neck, chest and abdomen. The radiation exposure, high cost, and high false-positive rate of PET/CT limit its wide clinical application.¹¹ Therefore, there is an urgent need for a comprehensive, accurate, simple, and noninvasive method to assess distant metastasis in lung cancer patients.

In recent years, with the rapid development of artificial intelligence, radiomics has become established in the field of radiology. Radiomics refers to the extraction and analysis of quantitative image features, invisible to the naked eye, from medical images, combined with machine learning or deep learning algorithms to investigate their possible links to biological and clinical outcomes.¹² Radiomics has been studied in the aspects of pathological detection, diagnosis prediction, gene mutation, treatment strategy, follow-up, efficacy evaluation, and prognosis of lung cancer.¹³ In addition, radiomics has obvious advantages in evaluating tumor heterogeneity and may be able to comprehensively and accurately predict distant metastasis of adenocarcinoma of the lung.

In some previous studies, Chen et al¹⁴ limited themselves to predicting brain metastases in lung adenocarcinomas, and Tao et al¹⁵ did not exclude subsolid lung carcinomas, which rarely have distant metastases. In addition, most previous studies extracted only three-dimensional radiomic features from CT images, and few studies extracted two-dimensional radiomic features. In the present study, we developed and validated a comprehensive model based on contrast-enhanced CT-based 3D and 2D radiomic features combined with clinical parameters to predict distant metastasis in patients with solid lung adenocarcinomas.

Methods

Patients and Clinical Parameters

This study retrospectively analyzed 1848 patients with lung adenocarcinoma who were pathologically diagnosed at our hospital between August 2018 and August 2021. Inclusion criteria: (1) patients with lung adenocarcinoma diagnosed by surgery or biopsy; (2) within 2 weeks before diagnosis, a contrast-enhanced chest CT examination was performed; (3) received brain CT/MRI, abdominal CT/MRI, and bone imaging within 2 weeks after chest examination. Exclusion criteria: (1) pure ground glass or partially solid lung adenocarcinoma; (2) multiple primary lung tumors or other malignant tumors; (3) venous phase images of the chest CT do not meet the conditions for segmentation (tumors with obstructive pneumonia or atelectasis, image artifacts, no thin-slice image or special scanning mode); (4) incomplete clinical data.

Finally, 253 patients with solid adenocarcinoma of the lung were included, and 123 patients (48.6%) developed distant metastases before treatment. A total of 175 patients before November 1, 2020, were classified as the training set, of which 87 patients had distant metastases and 88 patients did not. Seventy-eight patients after November 1, 2020, constituted the test set, of which 36 patients had distant metastases and 42 patients did not. Figure 1 shows the inclusion criteria, exclusion criteria, and grouping of patients.

Figure 1 Patient recruitment process. Abbreviation: DM, distant metastases.

In addition, the clinical parameters of the patients were recorded: Age, sex, smoking status, carcinoembryonic antigen (CEA), carbohydrate antigen 199 (CA199), location and largest diameter of the primary tumor, short diameter of the largest lymph node, clinical TNM stage, and CT semantic features (lobulation, burr, pleural indentation, cavity). The CT semantic features were observed and analyzed by two radiologists (with 4 and 8 years of professional experience, respectively) without knowledge of the results of distant metastases. Lobulation sign: defined as at least three arcuate depressions at the tumor margin; burr sign: defined as a line extending from the tumor margin to the surrounding parenchyma without reaching the pleura; pleural depression sign: referring to the tumor pulling the adjacent pleura; cavity: at least 5mm air shadow inside the tumor. Clinical TNM staging is based on the 8th edition of the TNM staging criteria for lung cancer published by the International Association for the Study of Lung Cancer (IASLC).¹⁶ Distant metastases are diagnosed based on histologic examination or apparent progression or shrinkage of lesions after antitumor treatment.

CT Image Acquisition

All patients underwent enhanced chest CT scan within 2 weeks before diagnosis. Scans were performed with the Siemens SOMATOM Definition Flash CT and GE REVOLUTION HD CT. The patient was examined in supine position. The scan was started after the patient stopped breathing at the end of inspiration, and the scan extended from the base of the lung to the level of the thoracic inlet. The slice thickness during image reconstruction was 1.0 or 1.25 mm. Other relevant scan parameters can be found in Supplementary Table 1.

Segmentation and Feature Extraction

The region of interest (ROI) of the primary tumor was segmented by a radiologist using 3D Slicer software (version 4.11; https://www.slicer.org/), and the ROI included bronchi, blood vessels, necrosis and calcification within the lesion, etc, but not normal lung tissue. Because the included lesions were solid lung adenocarcinomas, delineation was performed at the mediastinal window (window width 350, window level 40) and a 3D ROI was obtained by setting the lower threshold of the mediastinal window (−135HU) to delineate each layer of the lesion layer by layer semi-automatically; in combination with coronal and sagittal images, the largest cross-section of the lesion was selected to outline a 2D ROI. Radiomic feature extraction using PyRadiomics in the 3D slicer extension extracted 851 features per ROI, including 14 shape features, 18 first-order features, 24 Gray Level Co-occurrence Matrix (GLCM) features, 14 Gray Level Dependence Matrix (GLDM) features, 16 Gray Level Run Length Matrix (GLRLM) features, 16 Gray Level Size Zone Matrix (GLSZM) features, 5 Neighboring Gray Tone Difference Matrix (NGTDM) features, and 744 wavelet features (all feature names are listed in Supplementary Table 2).

The settings for image preprocessing are as follows: Gray level discretization (the bin width is fixed at 25), and the image is resampled into voxels of size 1*1*1.

To assess the reproducibility of the radiomics signature, we randomly selected 30 patients who were segmented by another radiologist 1 month later. None of them had information about clinical and pathologic details. Subsequently, the features extracted from the two segmentations were subjected to inter-rater correlation coefficient (ICC) analysis, and the features with an ICC ≥0.75 were selected. Figure 2 is a flowchart of this study.

Figure 2 Flowchart of this study: 3D and 2D ROI segmentation (A), feature extraction (B), feature selection and model development (C), and model evaluation (D).

Feature Selection, and Model Development and Validation

Radiomic features were normalized with the Z-score before feature screening. The 3D and 2D radiomic features were screened separately using the LASSO algorithm. Univariate analysis and multivariate logistic regression analysis were performed to search for meaningful clinical parameters. Training data were used for both feature screening and modeling. A 3D radiomics model (model 1), the 2D radiomics model (model 2), a combined 3D radiomics and 2D radiomics model (model 3), a clinical model (model 4), and a comprehensive model consisting of 3D radiomics, 2D radiomics, and clinical parameters (model 5) were developed.

To evaluate the predictive power of each model, the ROC curve was constructed and AUC, accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated. Delong’s test was used to compare AUC between models. DCA was used to evaluate the clinical utility of each model.^17,18 AIC is used to compare and rank multiple models and emphasizes comparison of the goodness of fit of the models, taking into account the principle of parsimony. In general, the smaller the AIC value, the better the model fit and the simpler the model.¹⁹ The nomogram is used to represent model 5. The value assigned to each variable is scored on a scale from 0 to 100. The total score is obtained by adding the scores for each variable. The risk of distant metastases in patients with solid adenocarcinoma of the lung can be predicted by mapping the total score to the probability of the bottom. The calibration curve is used to evaluate the calibration degree of model 5.

Statistical Analysis

Description of clinical statistical method: a chi-square test or Fisher’s exact test was used for the nominal variable and expressed by the number and proportion. Continuous variables conforming to normal distribution are expressed by two independent samples t-test and mean and standard deviation. For continuous variable with abnormal distribution, Mann–Whitney U-test was used and expressed by median and quartile. All statistical analysis for the present study were performed using R (version 4.0.4; https://www.r-project.org) and Python (version 3.5.6; https://www.python.org). A two-sided p-value <0.05 indicated statistical significance.

Results

Clinical Characteristics

The patient baseline data sheet showed that there were essentially no significant differences in clinical characteristics between the training and test sets (see Supplementary Table 3). The results of the univariate analysis of the training set showed that there were statistically significant differences in burr sign, T stage, N stage, age, CEA, CA199, the largest diameter of the tumor, and the short diameter of the largest lymph node (Table 1). Immediately thereafter, multivariate logistic regression results showed that N stage, age, and CEA were independent predictors of distant metastases (Table 2).

Table 1 Univariate Analysis of the Training Set and the Test Set

Table 2 Multivariate Logistic Regression Analysis of the Training Set

Consistency Test

ICC was applied to measure the consistency between the two image segmentation and feature extraction processes. The ICC cut-off values recommended by Koo et al²⁰ are as follows: poor reliability: ICC < 0.5; moderate reliability: 0.5 ≤ ICC < 0.75; good reliability: 0.75 ≤ ICC < 0.9; excellent reliability: 0.9 ≤ ICC. Atotal of 815 (95.8%) 3D radiomic features and 620 (72.9%) 2D radiomic features had good reliability (ICC ≥0.75), indicating that the extracted radiomic features were well reproducible. The results for ICC are described in more detail in Supplemental Table 4.

Selection of Radiomic Features and Establishment of Five Models

A total of 815 reliable 3D radiomic features were screened using LASSO, and the remaining 3 features were used to create Model 1. These three radiomic features were original_gldm_DependenceEntropy, original_glszm_ZoneEntropy, and wavelet-LLL_glszm_ZoneEntropy.

A total of 620 reliable 2D radiomic features were screened using LASSO (Supplementary Figure 1), and the remaining 6 features were used to build Model 2. These six radiomic features were original_shape_Elongation, original_gldm_DependenceEntropy, original_glrlm_RunEntropy, wavelet-LHL_glcm_Contrast, wavelet-LHL_glrlm_RunVariance, wavelet-LHH_glcm_Imc2.

We combined the above three radiomic features used to construct model 1 and the six radiomic features used to construct model 2 to construct model 3, and calculated the combined radiomics score (c-radscore) according to the weighting coefficient of the features. For more details, see Supplemental Table 5 and Supplemental Figure 2.

N stage, age, and CEA with statistical significance in univariate and multivariate analysis were selected to build the model 4.

Select N stage, age, CEA, and c-radscore to build model 5.

Performance of the Five Models

In the training set and test set, the AUC values of model 1 for predicting distant metastasis were 0.708 (0.645, 0.773) and 0.711 (0.612, 0.804), respectively, and the AUC values of model 2 were 0.746 (0.682, 0.805) and 0.769 (0.681, 0.852), the AUC values of model 3 were 0.761 (0.701, 0.819) and 0.775 (0.686, 0.86), the AUC values of model 4 were 0.819 (0.763, 0.872) and 0.829 (0.752, 0.904), and the AUC values of model 5 were 0.857 (0.808, 0.901) and 0.892 (0.829, 0.949), respectively. The predictive performance of each model is shown in Table 3, and the ROC curve is shown in Figure 3.

Table 3 Performance Evaluation of Five Models in the Training Set and the Test Set

Figure 3 Receiver operating characteristic curves of five models in training set (A) and test set (B).

The Delong test showed that the AUC values of model 5 and model 1 were statistically different in the test set (p=0.001 < 0.05/10, corrected by Bonferroni method²¹), and there was no statistical difference in AUC between the other models (see Supplementary Table 6 for details). DCA was used to evaluate the clinical utility of five predictive models by calculating the net gains at different probability thresholds. DCA (Figure 4) shows that model 5 predicts a higher net benefit for distant metastases than the other four models and all or no treatment strategy in both training and test sets when the threshold probability is approximately between 10% and 90%. The AIC values of the five models were 228.521, 223.886, 224.871, 192.447, and 176.645, with the AIC value of model 5 being the smallest. Based on a comprehensive consideration of DCA, ROC curve and AIC, it is proved that model 5 has better clinical utility, goodness of fit and parsimony.

Figure 4 Decision curve analysis for five models in training set (A) and test set (B). The y-axis represents the net benefit and the x-axis represents the threshold probability. The black, red, green, blue, and purple lines represent the net gains for model 1, model 2, model 3, model 4, and model 5, respectively. The gray line represents the assumption that all patients had distant metastases. The black line represents the assumption that all patients had no distant metastases.

Therefore, we created a nomogram of model 5 to facilitate clinical application (Figure 5). In addition, the calibration curve for model 5 (Figure 6) showed good agreement between its prediction of distant metastases and the actual situation. The Hosmer–Lemeshow test showed that model 5 was a good fit and was not statistically different from the ideal perfect model (p=0.642 > 0.05).

Figure 5 Nomogram for model 5. Abbreviation: c-radscore, radiomic score combining 3D and 2D radiomic features.

Figure 6 Calibration curves of model 5 in training set (A) and test set (B).

Discussion

In this study, we developed five models for predicting distant metastasis in patients with solid lung adenocarcinoma based on 3D radiomics, 2D radiomics of contrast-enhanced CT, and clinical features and evaluated them with the test set. Based on a comprehensive consideration of DCA, the ROC curve and AIC, it is proved that model 5 has better clinical utility, goodness of fit and parsimony than the other four models, and a nomogram is constructed for the convenience of clinicians. In some previous studies, Chen et al¹⁴ retrospectively included 89 patients with T1 lung adenocarcinoma to predict the occurrence of brain metastases. The AUC of the combined model of radiomics and clinical parameters was 0.871, and the sample size of this study was small with only 10-fold cross-validation. Tao et al¹⁵ extracted radiomic features from preoperative CT images of 356 non-small cell lung cancer patients to predict distant metastases. The combined model of radiomic and clinical parameters had an AUC of only 0.77 in the test set. The study did not exclude subsolid lung cancers that rarely develop distant metastases.

Our study follows the recommendations of previous studies to avoid the fragility of radiomics model development²²: (1) The code package for radiomic feature extraction is the Python-based code package PyRadiomics, which has been used by many studies and validated by research.^23,24 (2) We combined 3D radiomic features, 2D radiomic features, and clinical parameters to create a more comprehensive model. (3) This study improved the repeatability of radiomic features by setting threshold for semi-automatic delineation. The studies by Cohen et al²⁵ and Scholten et al²⁶ also confirm the accuracy of segmentation using the threshold method. (4) Multiple segmentations are performed to further check the repeatability of the radiomic features and remove the radiomic features with poor repeatability. (5) Pre-process the images from CT to reduce the influence of different devices and different slice thicknesses. (6) Minimize features in model development to reduce the risk of overfitting. (7) Temporal validation: according to Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD), temporal grouping is better than random grouping for evaluating model performance because it allows non-random variation between the two data sets.^27,28 (8) This study was assessed using the Radiomics Quality Score (RQS), with a score of 19/36, higher than most studies.^29,30

In consistency tests, we found considerable reliability in the extraction of radiomic features. Overall, good interobserver reliability (ICC ≥ 0.75) was achieved for 95.8% of 3D radiomic features and 72.9% of 2D radiomic features. This might reflect to some extent the stability and generalization of the radiomics model. According to the ICC results, the 3D radiomic features were more reproducible than the 2D radiomic features. However, according to the results of the DeLong test, the AUC values of model 1 and model 2 are not statistically different, which could be because the features with good reliability in the 2D radiomic features also have high predictive performance. Yang et al³¹ also demonstrated the predictive performance of 2D texture features in predicting angiolymphatic vessel invasion in lung adenocarcinoma with AUCs of 0.856 and 0.701 for 2D and 3D features in the test set, respectively. Wang et al³² combined the radiomic features of tumor 2D, tumor 3D, and peritumoral regions to predict recurrence-free survival in patients with stage IA solid non-small cell lung cancer. The 3-year and 5-year AUCs were 0.76 and 0.75, respectively, but our study combined 3D and 2D radiomic features and showed good predictive performance in predicting distant metastasis in patients with lung adenocarcinoma (model 3 had an AUC of 0.775 on the test set).

The histopathologic grade and number of lymph node metastases in patients with solid lung adenocarcinoma may be important predictors of distant metastasis. However, this information needs to be confirmed by surgery or biopsy and cannot be used as parameters to predict distant metastasis before treatment. Therefore, we collected as many pre-treatment clinical parameters of patients as possible: age, sex, smoking status, CEA, CA199, location and largest diameter of the primary tumor, short diameter of the largest lymph node, clinical T and N stage, and CT semantic features. The results of univariate analysis of the training set showed that there were statistically significant differences in burr sign, T stage, N stage, age, CEA, CA199, largest diameter of tumor, and short diameter of largest lymph node, which was relatively similar to the results of Tao et al.¹⁵ In multivariate analysis, N stage, age, and CEA were found to be independent predictors of distant metastasis in patients with solid lung adenocarcinomas. The constructed model 4 also showed good predictive performance with an AUC of 0.829 on the test set.

This study has some limitations. First, it is a retrospective study and selection bias may occur. Second, the data are from a single institution and the sample size was not very large, although we chose to use temporal grouping for validation. Third, we segmented only CT venous phase images that were more suitable for tumor boundary segmentation because of the workload. Fourth, the predictive performance of the model in different subgroups was not evaluated. In the future, prospective multicenter studies in more populations are needed to further validate our results.

Conclusion

This study demonstrates that a comprehensive model based on 3D radiomics, 2D radiomics, and clinical features has the potential to predict distant metastasis in patients with solid lung adenocarcinomas.