Nomogram Based on Super-Resolution Ultrasound Images Outperforms in Predicting Benign and Malignant Breast Lesions

Introduction

Ultrasound (US) has become an important tool in assessing breast lesions, with various indications for female patients. It can be used to detect palpable lumps, serve as the first diagnostic option for patients under 40 years old, and evaluate suspicious findings from mammography or magnetic resonance imaging.¹ Breast cancer, the most prevalent malignant disease among women, carries a high mortality rate. In 2010 alone, there were over 1.6 million cases reported worldwide with an estimated projection of 2.1 million by the year 2030.^1–4 Therefore, timely detection of breast masses is crucial for patients.

During ultrasonic imaging diagnosis, doctors rely on assessing microcalcifications, orientation, margin, shape, and echogenicity to identify malignancy prediction in clinical practice. However, despite its high sensitivity,⁵ this method exhibits poor specificity in characterization.⁶ Ultrasonic images are commonly affected by several factors, including speckle noise, low contrast, attenuation, shadow, and signal loss. Image resolution refers to the number of pixels in an image, which directly reflects the information contained within. The higher the resolution, the more vivid and detailed the appearance becomes, allowing for a greater amount of information to be extracted from it.⁷

While good-quality ultrasound images can enhance diagnostic accuracy, subjective evaluations have limitations.^8–10 To overcome these limitations, objective evaluation using computer-based image processing is necessary.^11–13 Radiomics has been shown in previous studies to have the potential to enhance patient management and clinical decision-making by revealing disease characteristics that may not be discernible to the human eye.^14,15 Additionally, the limited spatial resolution of ultrasonic images is frequently encountered due to a range of factors, including constraints imposed by hardware capabilities and the duration required for image acquisition. This constraint can pose challenges in achieving precise diagnosis and formulating effective treatment strategies.

Recently, significant advancements have been made in super-resolution reconstruction techniques, which offer great potential for improving the spatial resolution of medical images without requiring hardware upgrades. Super-resolution reconstruction techniques can be broadly classified into two categories: interpolation-based and learning-based. Interpolation-based techniques use mathematical models to estimate high-resolution images from low-resolution ones, but often yield blurry images with limited improvement in spatial resolution. On the other hand, learning-based techniques employ deep learning models to learn the mapping between low-resolution and high-resolution images. The application of deep learning algorithms in super-resolution reconstruction has led to promising outcomes. These techniques have the ability to significantly improve the level of detail and clarity in medical images, resulting in enhanced spatial resolution.

Therefore, in this retrospective study, we employed a 3D super-resolution reconstruction technique for ultrasound images that employs a generative adversarial network (GAN) as its fundamental architecture.

We attempted to establish various models utilizing diverse machine learning and deep learning algorithms based on the original ultrasound images and super-resolution images, respectively. Ultimately, we developed an effective tool for determining benign and malignant nodules based on the optimal model.

Materials and Methods

Patients

The retrospective study was conducted in accordance with the declaration of Helsinki and received approval from the Institutional Review Boards at The First Affiliated Hospital of Shandong First Medical University & Shandong Provincial Qianfoshan Hospital (YXLL-KY-2023(045)).

A total of 550 patients with 550 breast lesions underwent breast ultrasound examinations at our hospital from January 2020 to July 2022, and ultimately, after screening and selection criteria were applied, a cohort of 333 patients was included for analysis. The inclusion criteria were as follows: (1) Breast US examinations within 2 weeks before surgery or biopsy. (2) All breast lesions with definite pathological diagnosis (3) and no preoperative treatment. (4) Female patients diagnosed with breast lesions. The exclusion criteria were as follows: (1) Patients with biopsy, neoadjuvant chemotherapy, ablation, or radiotherapy before US examination. (2) Incomplete clinical or pathological information. (3) Insufficient image quality for feature extraction. (4) Male patients diagnosed with breast lesions. Study patients recruitment flowchart is shown in Figure 1.

Figure 1 Flowchart illustrating the process of study patients.

Ultrasound Image Acquisition and Processing

Breast ultrasounds were conducted at our hospital using GE machines with identical settings. The patient was positioned supine and instructed to breathe gently, while the instrument sequentially scanned the median, lateral, and medial positions of each breast along with other planes as needed. The maximum diameter of any detected masses was recorded, and then the original images of breast masses were obtained. In this study, we utilized a generative adversarial network (GAN) as the fundamental architecture for our 3D super-resolution reconstruction technique of US images. The objective is to enhance spatial resolution by a factor of two while preserving the original image dimensions. First, Gaussian noise was introduced to the US images in order to downsample the out-plane resolution by a factor of 2, resulting in the generation of novel low-resolution images. Subsequently, these low-resolution images were paired with synthetic high-resolution counterparts and utilized for training a lightweight GAN model. Finally, transfer learning was applied to original images using the trained model. As a result of improved z-resolution, the sagittal GAN-super-resolution images exhibited visual similarity to the original images while displaying reduced blurriness and enhanced texture details as well as sharper edges.

Construction of Machine Learning Model

The overall process of this study is summarized in Figure 2. The training set consisted of 266 patients resulting in a total of 266 images and the testing sets consisted of 67 images, respectively, from the slice that showed the maximum diameter of the tumor. The regions of interest (ROIs) for tumor segmentation were manually delineated on all US images by a radiologist with 5 years of experience using ITK-SNAP version 3.8.0 and subsequently reviewed by another radiologist with 10 years of experience to ensure accuracy. Any discrepancies identified by the two raters were resolved through a process of discussion and consensus-building. A total of 107 handcrafted radiomics features were extracted for each tumor from both the original image and its super-resolution counterpart using PyRadiomics version 3.0.1, respectively. The radiomics features consist of 18 first-order statistics features, 24 gray-level co-occurrence matrix (GLCM) features, 14 gray-level dependence matrix (GLDM) features, 16 gray-level run-length matrix (GLRLM) features, 16 gray-level size-zone matrix (GLSZM) features, 5 neighbouring gray-tone difference matrix (NGTDM) features, and 14 shape features. The radiomic features are described in detail in Table S1, while a comprehensive explanation of all image features can be found on the online platform (http://pyradiomics.readthedocs.io/en/latest/features.html).

Figure 2 The overall procedure of this investigation.

We conducted Mann–Whitney U-test and feature screening on all features, selecting only those with a p-value < 0.05. Subsequently, we standardized all feature lines using the z-score method. In cases where Pearson correlation coefficient > 0.9 existed between two features, one of them was chosen for retention. We used selected radiomics features to develop four models with Logistic Regression (LR) and Support Vector Machine (SVM) algorithms, including two based on original image features and two based on super-resolution image features. The patients were stratified randomly sampled and divided into a training set and a testing set at an 8:2 ratio. The model’s predictive performance was evaluated using sensitivity, specificity, accuracy, negative predictive value (NPV), positive predictive value (PPV), and the area under the receiver operating characteristic curve (AUC).

Development and Interpretability of Deep Learning Models

The dataset was divided into a training set and a testing set using the same splitting strategy as that used for constructing the machine learning model. The transfer learning models utilized in this study comprised ResNet 101 and MobileNet V2, the latter of which is a lightweight convolutional neural network (CNN). The initial weight values were obtained by pretraining all models on the ImageNet dataset. The entire process consisted of two essential steps: the forward computation and the backward propagation. To prepare for training, we extracted rectangular regions of interest (ROIs) from US images using tumor segmentation masks. Subsequently, the deep learning model predicted the malignant probability of the breast lesion image through sequential activation of convolution and pooling layers.

Furthermore, real-time data augmentation was implemented through random horizontal flipping and cropping. The model parameters were updated using the stochastic gradient descent (SGD) optimizer with an initial learning rate of 0.005, which decayed according to the cosine annealing algorithm over a total of 200 epochs consisting of 3400 iteration steps and a batch size of 16. Finally, four deep-learning models were developed: ORResNet101, ORMobileNet_v2, SRResNet101, and SRMobileNet_v2.

Gradient-weighted class activation mapping (Grad-CAM) was applied to visualize the models to make the decision-making process more transparent and investigating its interpretability. The gradient information of the last convolutional layer of CNNs is used for weighted fusion to obtain a class activation map that highlights important regions of the classification target image.¹⁶

Construction of Nomogram

In order to identify the optimal base model for diagnosing benign and malignant breast lesions, we compared the performance of ORResNet101, ORMobileNet_v2, SRResNet101, SRMobileNet_v2, OR_LR, OR_SVM, SR_LR, and SR_SVM models. Upon selecting SRMobileNet_v2 as the most effective basic model, clinical information was subsequently incorporated into the diagnostic model. The result of SRMobileNet_v2 was used deep-learning score for each breast lesion. To construct nomogram, we first evaluated the preoperative clinical characteristics through univariate logistic analysis to identify predictors associated with distinguishing between benign and malignant breast lesions. We then conducted multivariate logistic regression analysis on all statistically significant variables from the univariate analysis. Univariate and multivariate regression analyses of clinical characteristics was shown in Table S2. The findings indicate that both the age of patients and the maximum diameter of tumors are statistically significant variables (all P< 0.05). Finally, we developed a clinical nomogram based on statistically significant clinical variables and subsequently constructed a deep-learning nomogram by integrating the deep-learning score with these variables.

Statistical Analysis

The clinical characteristics of benign and malignant breast lesions were compared using t-tests or Chi-square tests. Machine learning models and deep learning models were evaluated using AUC scores, as well as other metrics such as accuracy, sensitivity, specificity, PPV, and NPV. The deep-learning nomogram was evaluated using ROC curve for discrimination, calibration curve with 1000 resamples for calibration, and Hosmer–Lemeshow (HL) test for goodness-of-fit. The clinical usefulness of the model was assessed through DCA by calculating net benefits. All statistical analyses were conducted using R version 4.2.0 and Python 3.0.1, with all statistics being two-sided and a P-value less than 0.05 deemed to be of significant statistical importance.

Results

Baseline Characters

The clinical characteristics are listed in Table 1, indicating significant differences in age and menopause status between benign and malignant breast lesions in both the training and testing sets. Categorical variables are represented by numbers (n), and p values were calculated using either Student’s t-test or Mann–Whitney U-test. Continuous variables are presented as mean ± standard deviation or median, and p values were calculated using either chi-square test or Fisher’s exact test.

Table 1 The Baseline Clinical Factors of Datasets

Assessment of Traditional Machine Learning Model and the Deep Learning Model

The performance of four traditional machine learning models and four deep learning models is presented in Table 2 and Figure 3 respectively. The models’ performance, based on super-resolution images, generally exceeded that of the original image-based models. Additionally, SRMobileNet_v2 exhibits superior performance among the eight models.

Table 2 The Performance of Different Models

Figure 3 The ROC curves for machine learning models and deep learning models in both training and testing set. (A) ROC curves of machine learning models in training set. (B) ROC curves of machine learning models in testing set. (C) ROC curves of deep learning models in training set. (D) ROC curves of deep learning models in testing set.

ROC curves of machine learning models in training set and testing set are shown in Figure 3A and B. Among the four traditional machine learning models, the logistic regression model based on super-resolution images (SR_LR) exhibits superior performance. The SR_LR model achieved an AUC of 0.921 (95% CI: 0.8909–0.9511), accuracy of 0.835, sensitivity of 0.838, specificity of 0.831, PPV of 0.850, and NPV of 0.817 in the training set; and an AUC of 0.819 (95% CI: 0.7187–0.9194), accuracy of 0.761, sensitivity of 0.706, specificity of 0.818 PPV of 0.800, and NPV of 0.730 in the testing set.

ROC curves of deep learning models in training set and testing set are illustrated in Figure 3C and D. In the four deep learning models, SRMobileNet_v2 s using MobileNet_v2 on super-resolution image had the best prediction performance in distinguishing between benign and malignant breast lesions. In the training set, SRMobileNet_v2 had an AUC of 0.981 (95% CI: 0.9673–0.9939), accuracy of 0.944, sensitivity of 0.915, specificity of 0.976, PPV of 0.977, and NPV of 0.910; while in the testing set, it demonstrated an AUC of 0.851 (95% CI: 0.7556–0.9468), accuracy of 0.821, sensitivity of 0.794, specificity of 0.848 PPV of 0.844, and NPV of 0.800.

SRMobileNet_v2 achieved the minimum loss value, indicating its superior performance in terms of error reduction during training process,¹⁷ and demonstrated faster convergence than any other three deep learning models (Figure 4). To enhance the interpretability of our deep learning model, we made the final convolutional layer transparent, allowing us to gain insights into the diagnosis of benign and malignant breast lesions, as illustrated in Figure 5.

Figure 4 The loss values of various deep learning models exhibited variation across iteration steps in the training set.

Figure 5 Attention regions of diverse deep learning models in ultrasound images of benign and malignant breast lesions: analysis of original and super-resolution images. ORM & SRM, original image & super-resolution image of malignant breast lesions; ORB & SRB, original image and super-resolution image of benign breast lesions.

When examining the attention regions generated by the SRMobileNet_v2 model, we found that they were clear and focused mainly on the boundary and internal regions of the tumor. Interestingly, these attention regions did not show activation in areas adjacent to blood vessels, normal breast tissue, or other soft tissue. This suggests that the SRMobileNet_v2 model effectively narrows its focus to areas directly associated with tumors while excluding surrounding structures that may not be significant indicators of malignancy. By doing so, the model increases its specificity and accuracy in identifying malignant breast lesions.

Performance of Nomogram

The diagnostic performance of the deep-learning nomogram (shown in Figure 6) and the clinical model (constructed without incorporating the deep-learning score using Logistic Regression) were evaluated using ROC curves, as depicted in Figure 7A and B. In the training set, the deep-learning nomogram demonstrated excellent discrimination performance with an AUC of 0.986 (95% CI: 0.9747–0.9964). Even when applied to the testing set, the original model still displayed strong discrimination ability with a slightly reduced AUC of 0.894 (95% CI: 0.8167–0.9712).

Figure 6 The deep-learning nomogram.

Figure 7 The diagnostic performance of the deep learning nomogram and the clinical model. (A) ROC curves of deep Learning nomogram and the clinical model in training set. (B) ROC curves of deep Learning nomogram and the clinical model in testing set. (C) The deep learning nomogram’s calibration curves in training set. (D) The deep learning nomogram’s calibration curves in testing set. (E) DCAs of deep learning nomogram and the clinical model in training set. (F) DCAs of deep learning nomogram and the clinical model in testing set.

On the other hand, in both the training and testing sets, the clinical model had significantly lower discrimination compared to the deep-learning nomogram. In the training set, the clinical model achieved an AUC of 0.862 (95% CI: 0.8174–0.9070), while in the testing set, the AUC was 0.792 (95% CI: 0.6769–0.9069). The deep-learning nomogram outperformed the clinical model in both sets, indicating better predictive efficacy.

The deep-learning nomogram’s calibration curves showed that the predicted and observed malignancy of breast lesions had good agreement in both the training and testing groups (refer to Figure 7C and D). The HL test resulted in a non-significant P value of 0.668 and 0.285 for both cohorts, indicating that the deep-learning nomogram has excellent calibration capability. The DCAs demonstrated that the deep-learning nomogram yields a greater net benefit than the clinical model across the most threshold probability ranges in both training and testing sets (refer to Figure 7E and F).

Discussion

In our current investigation, we have developed an exceedingly precise nomogram for diagnosing benign and malignant breast nodules using deep-learning scores based on 3D super-resolution B-mode US images combined with clinical characteristics. The performance of our nomogram is exceptional, with AUC values of 0.986 in the training set and 0.894 in the testing set.

The efficiency of ultrasound diagnosis depends on the operator’s skill.^18,19 Artificial intelligence is anticipated to enhance the diagnostic performance of breast ultrasound and reduce interobserver variability. Artificial intelligence plays a crucial role in breast cancer screening and detection, alleviating the workload of radiologists while compensating for inexperienced individuals’ lack of experience and skills. Several studies have explored the efficacy of different AI models in the diagnosis of breast nodules.^20–26 Most of these studies have focused on two-dimensional grayscale ultrasound images, ultrasound elastography, and other multimodal images. However, there is a relatively limited number of studies that have specifically utilized super-resolution ultrasound images. Among them, Romeo’s B-ultrasound radiomics approach²⁶ and Zhang’s B-mode ultrasound deep learning method²⁴ have demonstrated high performance in diagnosing benign and malignant breast diseases. In the testing set, they achieved AUC values of 0.82 and 0.846, respectively. Our developed nomogram, based on B-mode ultrasound images, outperforms the previous studies with an AUC of 0.894 in the testing set. This signifies its superiority in diagnosing benign and malignant breast nodules.

In addition, we employed both deep learning and traditional machine learning techniques to construct diagnostic models. Eight diagnostic models were developed, and their performance was compared (showed in Table 2). Remarkably, the top-performing deep learning model surpassed the performance of the best traditional machine learning models in our dataset. The logistic regression model based on super-resolution images (SR_LR) achieved an AUC of 0.921 in the training set and an AUC of 0.819 in the testing set, while SRMobileNet_v2 utilizing. MobileNet_v2 on super-resolution image obtained an AUC of 0.981 in the training set and an AUC of 0.851 in the testing set.

To better demonstrate the performance of various deep learning models, we utilized heatmaps. These heatmaps have revolutionized medical diagnosis by providing invaluable insights into the nature of malignant lesions and highlighting salient regions that help clinicians make informed decisions about patient care.

The development of medical image clarity is crucial as it serves as the fundamental basis for experts to diagnose patients’ conditions. Super-resolution techniques have demonstrated their efficacy in improving image resolution and quality in various medical imaging modalities.²⁷ Super-resolution reconstruction is mainly widely used in practical medical applications for MR and CT images,^28–31 but it is rarely used for US images.^32,33 In the present study, we applied super-resolution reconstruction to B-mode ultrasound images with the expectation of achieving finer details and improved visualization of breast nodules. This has the potential to lead to more accurate and reliable diagnoses. In line with the current trend, researchers will continue to propose more effective super-resolution reconstruction algorithms in order to enhance the quality of medical images. Looking ahead, it may become feasible to directly acquire high-definition medical images through low-cost software algorithms. With the increasing prevalence of super-resolution technology in clinical practice, a greater number of radiologists will reap its benefits.

Our research is subject to certain limitations. First, as this study is retrospective in nature, further prospective studies are necessary to validate its findings before they can be translated into clinical practice. Second, the 3D super-resolution reconstruction technique we provided can increase the spatial resolution by 2 times while maintaining the original image size. In future studies, we aim to increase the spatial resolution even further for comparative studies. Third, the primary objective of this study was solely focused on super-resolution reconstruction of B-mode ultrasound images. Our aim is to fully implement super-resolution techniques on ultrasound images acquired in various modes, with the ultimate goal of establishing a universal diagnostic model for distinguishing between benign and malignant breast lesions. This will enhance the applicability of our tool and assist clinicians in their discrimination tasks. Fourth, due to the retrospective design of this study, we did not establish such a combined model based on US, Mammography, and MR images of patients with breast nodules. However, exploring such a comprehensive model will be an important direction for our future research endeavors.

Conclusion

We have successfully developed a nomogram by integrating clinical parameters with the Deep Learning Radiomics technique, based on high-resolution ultrasound images of breast nodules. This innovative approach offers a noninvasive and highly practical solution for accurately predicting the benign or malignant nature of these nodules before surgery. Our nomogram not only enhances diagnostic accuracy but also provides clinicians with an advanced tool to make informed decisions regarding patient care.