Improving Non-Invasive Prediction of Thyroid Nodule Malignancy: A Machine Learning-Based Clinical Approach

Introduction

Thyroid cancer (TC) is the most frequently diagnosed endocrine carcinoma, ranking seventh in global incidence according to 2022 GLOBOCAN statistics.¹ This high prevalence underscores the importance of accurate diagnostic methods that can effectively identify malignancies and minimize unnecessary interventions. Currently, ultrasound (US) combined with fine-needle aspiration biopsy (FNAB) constitutes the gold standard for the diagnostic evaluation of TC.² Ultrasonography serves as the first step in TC diagnosis, providing detailed images of nodules and enabling the assessment of critical characteristics such as shape, margins, microcalcifications, echogenicity, and vascularity. However, interpretation becomes difficult when features typically associated with malignancy appear in benign nodules, or when malignant nodules present with benign characteristics.³ Based on ultrasound characteristics, clinicians determine whether a nodule requires further investigation with FNAB, which enables cytological assessment of the cells within the nodule. The results of FNAB are classified according to the Bethesda System for Reporting Thyroid Cytopathology (TBSRTC), with categories ranging from benign (Bethesda II) to malignant (Bethesda VI).⁴ A significant diagnostic challenge emerges with Bethesda categories III (atypia of undetermined significance), IV (follicular neoplasm), and V (suspicious for malignancy), which represent indeterminate or inconclusive cytological findings.^4,5 According to the third edition of TBSRTC, these categories account for approximately 1/3 of all FNAB results and are associated with a malignancy risk of 22% for category III, 30% for category IV, and 74% for category V, posing a substantial clinical dilemma, especially category III, which is the most heterogeneous.⁵ Given this diagnostic uncertainty, many patients with indeterminate cytology undergo thyroidectomy to establish a definitive diagnosis.⁶

A recent study by Mavromati et al found that many thyroidectomies performed for nodules with indeterminate cytology ultimately demonstrated benign pathology on final histological examination. Surgery was deemed unnecessary in 56%, 68%, and 21% of patients with Bethesda III, IV, and V nodules, respectively.⁷ These unnecessary surgical interventions carry significant consequences for patients, including permanent hormone replacement and potential complications such as hypoparathyroidism or recurrent laryngeal nerve palsy, and are associated with high costs.^8–10 Cancer diagnosis and treatment may also have a negative effect on patients’ psychological health and decrease their quality of life.¹¹ This high rate of unnecessary surgeries highlights a critical clinical need to improve pre-surgical diagnostic precision for thyroid nodules. Improving diagnostic accuracy could reduce the number of surgical interventions and their associated risks and costs.

Recent advances in artificial intelligence (AI), particularly machine learning (ML), have drawn increasing attention to the potential applications of AI and ML in medical diagnostics.^12,13 ML models require training on datasets to correctly identify complex patterns and make predictions, such as classifying thyroid nodules as malignant or benign.^14–17 Research has shown that ML models may outperform human judgment.¹⁵ In this study, we aimed to evaluate the clinical performance and limitations of commonly used, interpretable ML models for pre-surgical malignancy risk assessment of TNs, based exclusively on routinely available, non-invasive ultrasound features in a large real-world cohort. Importantly, the aim of this study was not to develop novel ML algorithms.

Materials and Methods

All procedures were conducted in accordance with the ethical standards of the institutional and/or national research committee and the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards. Our retrospective study was approved by the Bioethics Committee of the Wroclaw Medical University, Poland. Signature Number: KB-241/2023; 24 February 2023. Every patient participating in the study provided informed consent at admission, stating that the results may be used for research purposes. Data from established medical records were analyzed retrospectively and anonymously. Due to data anonymization, the authors did not have direct access to the study participants.

Data Collection

The present study analyzed the medical records of 5986 patients admitted to the Department of General Surgery at Wroclaw Medical University during the study period (between January 2008 and December 2023). All of these patients underwent thyroid ultrasound examinations and subsequently had total or partial thyroidectomy due to the presence of either single or multiple thyroid nodules (TNs). The ultrasonography equipment used at our center was consistently of the highest clinical standard, with regular updates to maintain state-of-the-art capabilities. It was performed by two experienced ultrasonographers, each with over 20 years of experience in thyroid imaging. Postoperative samples were collected in each case, and histopathology reports were obtained. Only patients with benign TNs or differentiated thyroid cancer (TC) were considered. Differentiated TC included papillary thyroid carcinoma (PTC) and follicular thyroid carcinoma (FTC).¹⁸ Due to changes in histopathological classification over the study period, Hürthle cell TC could not be consistently distinguished from FTC in this retrospective cohort and was therefore analyzed collectively within the malignant group. Fifty-five patients with either poorly differentiated TC, anaplastic TC, medullary TC, secondary TC, sarcoma, lymphoma, extramedullary plasmacytoma, or squamous cell carcinoma were excluded from this study. In total, 5928 records were used in this research; thus, 3 patients were excluded from the study due to missing data. The selection process is illustrated in a flow diagram in Figure 1.

Figure 1 Selection of the study group from the records of 5986 patients admitted to the Department of General Surgery during the study period. All selected patients underwent thyroid ultrasonography prior to thyroidectomy. In each case histopathology results were obtained. Only cases of differentiated thyroid cancer or benign thyroid nodules were analyzed.

Among these individuals, 4954 were women (83.6%), and 974 were men (16.4%). The mean age was 51.7 ± 14.5 years. TN characteristics, such as (1) hypoechogenicity, (2) microcalcifications, (3) shape, (4) margins and (5) vascularity, were gathered from ultrasonography imaging. The selected ultrasonographic features correspond to key risk descriptors incorporated in structured reporting systems such as EU-TIRADS, in which high-risk nodules (with an estimated malignancy risk of approximately 26–87%) are characterized by hypoechogenicity, microcalcifications, irregular shape, and irregular or ill-defined margins. Vascularity pattern represents an additional relevant factor, as malignant TNs are more likely to exhibit marked intranodular vascular flow.¹⁹ Only categorical ultrasound variables were included, as they are routinely assessed in daily clinical practice, less dependent on equipment-specific settings, and more robust to inter-operator variability, thereby facilitating model interpretability and potential clinical implementation. Table 1 provides an overall summary of the study population.

Table 1 A General Summary of the Overall Study Population, Including Sex Distribution, Mean Age, and Key Ultrasound Features (Hypoechogenicity, Microcalcifications, Shape, Margins and Vascularity)

Table 2 presents considered ultrasonographic features along with their occurrence typically associated with benign or malignant TNs.^19–21 However, it is important to note that these characteristics often overlap. Features typically associated with malignancy may appear in benign TNs, or conversely, malignant nodules may present with benign characteristics. In Figure 2, there are example ultrasound scans of the thyroid with nodules, illustrating the features discussed above.

Table 2 Features Commonly Associated with Thyroid Nodule Malignancy (Hypoechogenicity, Microcalcifications, Shape, Margins and Vascularity) and Their Encoding for Machine Learning Model Development

Figure 2 Ultrasound images of a benign and malignant thyroid nodules. (A) A benign isoechoic nodule of a regular shape with sharpen margins. (B) A malignant nodule with massive peripheral calcifications. (C) A malignant hypoechogenic nodule of irregular shape with blurred margins. (D) A malignant hypoechogenic nodule of irregular shape with irregular vascularity. Abbreviation: n, nodule.

All 5 features, presented in Table 3, were further used to build ML models predicting the malignancy of TNs.

Table 3 Distribution of Clinical and Ultrasound Features in Benign and Malignant Thyroid Nodules Among 5928 Patients. Data Include Key Ultrasound Characteristics Analyzed in This Study (Hypoechogenicity, Microcalcifications, Shape, Margins and Vascularity)

Data Preprocessing

Among the 5928 cases eligible for this study, 5305 were classified as benign (89.5%) and 623 as malignant (10.5%) based on histopathology reports. The dataset was then randomly divided into two subsets: a training cohort (80%) and a test cohort (20%). To deal with data imbalance, the Synthetic Minority Oversampling Technique (SMOTE) was used in the training dataset. In the test cohort, however, we intentionally preserved the original class distribution (10.5% malignant cases) to realistically simulate clinical conditions and evaluate the model’s true diagnostic performance, as this setup reflects the actual prevalence encountered in routine practice and does not influence model learning, which was conducted on a balanced training set.

Machine Learning Methods

Five machine learning models were taught to classify thyroid tumors as benign (0) or malignant (1) based on data from the training dataset. The program Orange3 provided algorithms for the machine learning models and was used to conduct the training and evaluation process. The models’ training and evaluation process is presented in Figure 3.

Figure 3 Workflow of the study. The dataset (n = 5928) was split into training (80%) and test (20%) subsets. Synthetic Minority Oversampling Technique (SMOTE) was applied to the training data, and five machine learning models were developed and evaluated. The best-performing model was further evaluated on the test dataset, and feature importance analysis was conducted. Abbreviation: ROC-AUC, receiver operating characteristic-area under the curve.

The models selected for comparison were Random Forest, Logistic Regression, Neural Network Multilayer Perceptron (MLP), Gradient Boosting Machines (GBMs), and Decision Tree. Machine learning models were implemented in Orange3 with fixed hyperparameter settings. Random Forest was trained with 300 trees, maximum tree depth limited to 5, and a minimum node size of 10 instances required for further splitting. Gradient Boosting (scikit-learn implementation) used 300 estimators with a learning rate of 0.05, maximum tree depth of 3, a minimum split size of 10 instances, and subsampling of 0.70 of training instances; replicable training was enabled. Logistic Regression used L2 (ridge) regularization with C = 0.5. A single Decision Tree was induced as a binary tree with maximum depth 5, minimum leaf size of 5 instances, minimum split size of 10 instances, and early stopping when majority reached 95%. The Neural Network classifier used two hidden layers (30 and 10 neurons), ReLU activation, the Adam optimizer, L2 regularization α = 0.001, and a maximum of 500 training iterations; replicable training was enabled. Ten-fold cross-validation was applied to obtain reliable performance metrics and enable unbiased comparison between classifiers.²²

The models’ performance was evaluated using five measurements: classification accuracy (CA), precision (Prec), recall (Rec), F1 score, and receiver operating characteristic area under the curve (ROC-AUC). Accuracy measures overall correctness, precision reflects the proportion of correctly identified malignant cases among all cases classified as malignant, recall (sensitivity) indicates the proportion of actual malignant cases correctly detected, specificity measures correct identification of benign cases, F1 score balances precision and recall, and ROC-AUC quantifies discriminatory ability between classes.^23,24 For each model and metric, performance was estimated using ten-fold cross-validation and reported as mean ± standard deviation across folds. Statistical comparisons between models were conducted using paired Wilcoxon signed-rank tests applied to fold-wise results, with Holm-Bonferroni correction for multiple comparisons (α = 0.05). Following cross-validation and statistical comparison of model performance, one model was selected for final evaluation on the held-out test dataset.

Results

Random Forest, Gradient Boosting Machines (GBMs), and the Neural Network (MLP) achieved comparable classification accuracy (≈0.79) and ROC-AUC values (≈0.88), indicating similar overall discriminatory performance. The MLP obtained the highest precision, whereas Logistic Regression achieved the highest recall, reflecting differences in sensitivity-specificity trade-offs across models. Statistical analysis across cross-validation folds showed no single classifier to be consistently superior across all performance metrics. Given its stable performance, high specificity, and interpretability, Random Forest was selected for further evaluation on the independent test dataset. The detailed results are presented in Table 4.

Table 4 Results Achieved by Machine Learning Models on the Training Dataset, Assessed Using Standard Performance Metrics

The test cohort (20% of the original dataset) consisted of 1061 benign (0) and 125 malignant (1) cases. The confusion matrix revealed that the model correctly detected 77 out of 125 malignant TNs and 996 out of 1061 benign TNs. Figure 4 illustrates the confusion matrix of Random Forest.

Figure 4 Confusion matrix of the Random Forest model evaluated on the test dataset. Notes: 0 – benign, 1 – malignant.

Random Forest’s performance was then reassessed on the test dataset using the same metrics. As presented in Table 5, the model achieved a classification accuracy of 0.905 (95% CI: 0.887–0.920), precision of 0.577 (95% CI: 0.460–0.622), specificity of 0.939 (95% CI: 0.923–0.952), recall of 0.616 (95% CI: 0.528–0.697), F1 score of 0.577 (95% CI: 0.498–0.644), and ROC-AUC of 0.843 (95% CI: 0.734–0.819). The Random Forest model achieved a Brier score of 0.114, indicating good overall calibration and reliable probability estimates for malignancy prediction. These results confirmed the model’s strong performance in correctly identifying benign cases, although its ability to detect malignant nodules was comparatively lower. The observed decrease in precision compared to the training set may be explained by several factors related to dataset composition and model training strategy. First, the test cohort preserved the natural class distribution observed in clinical practice, with malignant cases constituting only 10.5% of the population, whereas the training dataset was balanced using SMOTE. In low-prevalence settings, precision is particularly sensitive to false positive predictions, such that even a modest increase in false positives can lead to a substantial reduction in precision values.²⁵ Second, SMOTE generates synthetic malignant samples through interpolation between existing minority-class observations. While this approach improves class balance during training, it does not fully reproduce the diversity of real malignant nodules observed in clinical practice.¹⁹ As a result, the model, trained on a dataset containing a large proportion of synthetic cases, was exposed during testing to a broader spectrum of malignant and borderline presentations that had not been fully represented in the training phase. Consequently, this mismatch between the synthetic training distribution and the more heterogeneous, imbalanced, real-world test data led to an increased number of benign nodules being misclassified as malignant, thereby reducing precision.

Table 5 Results Achieved by the Random Forest Model on the Test Dataset, Assessed Using Standard Performance Metrics

Feature importance analysis was conducted to evaluate the contribution of each variable to the predictions made by the Random Forest model using the permutation method. The greater the decrease in ROC-AUC after excluding each variable, the larger the impact a feature has on distinguishing between benign and malignant TNs. As shown in Figure 5, nodule vascularity emerged as the most influential factor in decision-making. Microcalcifications, irregular margins, and hypoechogenicity also played a significant role in determining the malignancy of a thyroid nodule.

Figure 5 Feature importance analysis identified vascularity, followed by the presence of microcalcifications and the type of nodule margin, as the most decisive factors in determining nodule malignancy. Abbreviations: TC, thyroid cancer, AUC, area under the curve.

Discussion

In this study, we analyzed the effectiveness of various models in diagnosing TC based on data extracted from ultrasonography imaging and patients’ medical histories. This study aimed to investigate whether accurate TC diagnosis is possible using only non-invasive diagnostic techniques supported by ML models, which would be less burdensome for the patient than the classic diagnostic approach consisting of ultrasound and FNAB.

To validate the effectiveness of our method, we compared our results with findings from recent publications that explored the diagnostic accuracy of TC using ultrasonography and FNAB. In the study by Alhajlan et al, the authors reported a recall of 83.33%, an ROC-AUC of 87.8%, and a specificity of 79.14%.²⁶ Another study by Mehanna et al reported a recall of 87% and a notably lower specificity of 67%.²⁷

In contrast, our Random Forest model achieved a specificity of 93.9%, an ROC-AUC of 84.3%, and a recall of 61.6%. These results indicate that our approach substantially improves specificity and presents a notably lower recall compared to the cited studies. The higher specificity suggests that our model is more effective in correctly identifying benign cases, which is crucial for reducing unnecessary biopsies. The slightly lower ROC-AUC in our case may be attributed to the sole use of non-invasive input features, which supports the clinical significance of our findings: high diagnostic accuracy can be achieved without relying on invasive procedures. A lower recall score indicates that our model performed worse in terms of correctly identifying malignant nodules, which could be attributed to data class imbalance and the application of the SMOTE.

Malignant cases occur much less frequently than benign ones, with an overall malignancy rate reported between 7% and 15% in the general population.²⁸ In our dataset, which consists of 5928 patients, only 10.5% (n = 623) of cases were malignant, confirming the naturally occurring class imbalance. This skewed distribution can negatively affect the performance of classification models, often resulting in models that are biased toward the majority class, benign TNs, and thus less reliable for identifying malignant cases.²⁹

One way to address this issue is by undersampling the majority class to achieve balance, however, this approach sacrifices a large portion of the data, potentially reducing variability and limiting the model’s ability to generalize.³⁰ In our study, we applied the SMOTE to balance the training dataset. The SMOTE addresses class imbalance by generating synthetic samples of the minority class, thereby improving predictive accuracy without discarding valuable data.²⁹ This method has been successfully applied in other domains facing similar imbalance challenges, such as breast cancer detection and cervical cancer classification.^31,32 This approach is particularly effective when working with low-dimensional data, where the number of features is relatively small compared to the number of samples.²⁹ Although research has shown that oversampling techniques like the SMOTE often outperform undersampling methods in terms of classification outcomes, in our case, the models’ ability to correctly classify malignant nodules was lower than in previously mentioned studies, despite the use of the SMOTE.^26,27,32,33

The size of the dataset is widely recognized as having a key influence on the models’ performance and generalizability. Larger cohorts tend to improve classification accuracy and reduce the risk of overfitting.^28,34 In our study, we analyzed data from a cohort of 5928 patients, which provided an extensive and diverse set of features for model training. This large-scale dataset allowed the algorithm to learn from a wide variety of clinical presentations and imaging patterns, thereby enhancing its ability to correctly diagnose new, unseen cases. Therefore, our best-performing model classified thyroid nodules with 90.5% accuracy, slightly higher than the accuracy reported in similar studies based on smaller datasets; for instance, Xi et al achieved 79.3% accuracy using a cohort of 724 patients.¹⁵ The substantial volume of data used in our work is thus a major strength, supporting the reliability of the obtained results and the clinical relevance of the proposed approach. Nevertheless, while overall accuracy was high, it was largely driven by the model’s ability to correctly detect benign nodules. Improving sensitivity remains a critical challenge to ensure malignant cases are reliably identified. Future efforts should focus on increasing recall through integration of additional clinical and imaging data, such as cytology results or more detailed vascular and structural ultrasound features. Strengthening the model’s sensitivity without compromising specificity would help avoid missed cancers while still preventing overtreatment of benign lesions.

Recent studies, including multicenter analyses based on raw ultrasound images, have demonstrated the potential of deep learning and convolutional neural networks for TN classification with comparable or superior performance to that of radiologists in internal and external test sets.^35,36 However, such approaches often require large, well-annotated image datasets, are less interpretable, and remain difficult to integrate into routine clinical workflows.^37,38 In contrast, the present study deliberately focuses on feature-based ML models using routinely assessed ultrasound descriptors, prioritizing interpretability and real-world applicability over maximal standalone diagnostic performance. Our study lies not in algorithmic innovation, but in the large-scale, real-world evaluation of commonly used ML models applied to a limited and clinically interpretable set of ultrasound features, highlighting both their potential utility and their current limitations in preoperative risk stratification.

In this research, the following ultrasonographic characteristics of nodules, typically associated with malignancy, were taken into account: hypoechogenicity, presence of microcalcifications, irregular shape, irregular margins, and high vascularity.^39–41 By conducting feature importance analysis, we wanted to find out which of these characteristics play a more significant role in determining the malignancy of nodules. The results of our study identified high vascularity and microcalcifications as the most critical determinants in diagnosing TC using the tested models. Our results align with Xi et al’s findings, who also identified these features as highly predictive.¹⁵ However, a study by Li et al reports that the presence of irregular margins contributed the most to the models’ judgement.⁴² Identifying the most significant features of TC is key to improving diagnostic accuracy and efficiency. Importantly, the moderate sensitivity observed in this study underscores the challenges of relying solely on a limited set of ultrasound features for malignancy prediction and supports the role of ML models as decision-support tools rather than stand-alone diagnostic systems.

In TC diagnostics, missed malignancies carry significant clinical risks including disease progression, potential metastasis, and delayed treatment.⁴³ With 48 out of 125 malignant cases undetected in our test cohort, the observed sensitivity remains insufficient for stand-alone clinical decision-making. Therefore, this approach should be viewed as complementary to, rather than replacement for, established risk stratification frameworks. Current guidelines including EU-TIRADS and ATA recommendations provide structured approaches to nodule evaluation based on multiple sonographic features.^44,45 Our model could potentially integrate with these systems as an adjunctive decision-support tool, particularly when clinical-sonographic assessment yields indeterminate results. For instance, nodules classified as EU-TIRADS 4 (intermediate risk, 6–17% malignancy) or Bethesda III/IV cytology might benefit from ML-based probability estimates to refine surgical decision-making. However, given the current sensitivity limitations, the model’s primary utility lies in confirming benign classification (high specificity: 93.9%) rather than ruling out malignancy.

Beyond conventional ultrasonography, emerging imaging modalities have been investigated for thyroid nodule diagnosis, aiming to capture tissue characteristics not accessible with standard ultrasound. Second harmonic generation (SHG) microscopy has demonstrated the ability to visualize microstructural features of thyroid nodules, including heterogeneity of collagen organization associated with malignant pathology.^46,47 Recent studies further showed that combining such advanced imaging data with supervised machine learning can improve diagnostic discrimination by explicitly leveraging high-dimensional, image-derived features.¹⁹ While these approaches remain outside routine clinical practice, they illustrate the broader applicability of machine learning as an analytical framework for integrating diverse imaging modalities. In this context, the machine learning strategy presented in the current study can be viewed as a clinically accessible implementation of this approach, based on routinely available ultrasound features.

Limitations of the Study

This study has several limitations. First, the US characteristics of the nodules were extracted manually by clinicians, which may introduce bias and affect data reliability.⁴⁸ Recent studies have demonstrated the potential of deep learning models, such as convolutional neural networks (CNNs), to automatically identify TNs from ultrasound images and correctly classify them.^49–51 These algorithms analyze complex image features that are beyond human perception. When compared to experienced radiologists, these models have shown comparable accuracy and even superior diagnostic performance in detecting benign nodules.⁵² Second, we acknowledge the potential issue of era bias due to the 15-year study period, as changes in ultrasound technology and evolving diagnostic standards (eg., EU-TIRADS) over time may have introduced variability affecting data interpretation and model performance. Moreover, we recognize that the use of vascularity as a simplified binary feature (high vs. low) may not fully capture the complexity of blood flow patterns within thyroid nodules; this approach, though consistent with clinical practice, could introduce bias and affect feature importance ranking, highlighting the need for more detailed, prospective, multi-center data in future studies. Additionally, the retrospective nature of the study and the long observation period may introduce variability related to operator experience. The analysis was also based on a limited set of routinely assessed ultrasound features, which may have constrained model sensitivity but was intentionally chosen to reflect real-world clinical practice. These limitations are unavoidable in retrospective study designs. Finally, all patient records analyzed in this study came from a single medical center. Future studies should focus on external validation by training and testing these models on datasets from multiple centers to enhance accuracy and minimize random diagnostic errors.

Recent studies highlight the importance of a multimodal diagnostic approach in TN assessment, as reliance on a single diagnostic modality is associated with relevant clinical uncertainty, particularly in indeterminate nodules.⁵³ US-based evaluation, although fundamental in initial risk stratification, is limited by substantial overlap between benign and malignant sonographic features and by inter-observer variability, which restricts its standalone diagnostic value. These limitations contribute to the persistent clinical dilemma of indeterminate nodules, where neither ultrasound nor cytology alone can reliably exclude malignancy. In this context, the assessment and integration of potential markers or risk factors of malignancy are considered essential for improving diagnostic accuracy.^54–56 Consequently, current evidence supports the use of ultrasound-derived features as part of an integrated diagnostic framework rather than as an isolated decision-making tool.⁵⁷

Conclusion

Our study demonstrates that feature-based ML models can support the diagnostic assessment of TNs using non-invasive ultrasound features. The proposed approach achieved satisfactory performance in identifying benign lesions, which is clinically relevant for reducing overtreatment and avoiding unnecessary surgical procedures. However, the current sensitivity of the model remains insufficient for stand-alone clinical decision-making, particularly with respect to reliable detection of malignant nodules. Therefore, the model should be considered a decision-support tool rather than an independent method replacing established diagnostic pathways. Further improvements will require expansion of the dataset – especially with a higher proportion of malignant cases – and integration of additional diagnostic inputs, including extended ultrasound characteristics and cytological data from FNAB. Such multimodal approaches may enhance diagnostic reliability and, in the longer term, contribute to more cost-effective and individualized management of patients with TNs.