Machine Learning-Based Objective Evaluation Model of CTPA Image Quality: A Multi-Center Study

Introduction

Pulmonary embolism (PE) is a critical and potentially fatal condition caused by the blockage of pulmonary arteries or their branches, typically resulting from blood clots that often originate in deep vein thrombosis.^1–3 Computed tomographic pulmonary angiography (CTPA) is currently the most commonly used and effective method for diagnosing PE, with PE typically appearing as a filling defect in CTPA images.^4,5 However, the accuracy of CTPA in diagnosing PE can be compromised by several factors, including improper scan timing and the presence of artifacts, which may degrade image quality. Such degradation can negatively affect post-processing and hinder the accurate diagnosis of PE. Therefore, evaluating image quality is a critical aspect of quality control in imaging systems for PE diagnosis.^6–8

Currently, there are two main methods for distinguishing between artifacts and lesions: one is based on subjective image quality assessment, and the other involves objective evaluation models designed for specific needs.⁹ However, subjective image quality assessment has several drawbacks: it is time-consuming, highly influenced by the evaluator’s personal perception, and dependent on the image viewing equipment and display environment. For these reasons, solely relying on physicians’ subjective impressions for effective medical image evaluation is not the optimal approach.^10,11

Most studies use signal-to-noise ratio (SNR) and contrast-to-noise ratio (CNR) as standard metrics for objective image quality assessment in CTPA image quality comparisons. Their research evaluates new technologies, contrast agent injection protocols, and scanning protocols through objective image quality assessment.^12–15 While these metrics reflect image quality to some extent, they are unable to fully assess the quality based on the specific characteristics of CTPA images.

With the development of artificial intelligence, it has been widely applied in various aspects of medicine, such as predicting treatment outcomes, cancer classification, and diagnostic assistance.^16–18 However, its application in the field of quality assessment is limited, mainly due to the “black-box” nature of deep learning technologies, which makes them lack interpretability and thus affects the reliability of the models. This study proposes a machine learning model tailored to the unique characteristics of CTPA images, utilizing interpretable features to accurately reflect image quality. This approach aims to improve the efficiency of image quality control, facilitate more effective quality control practices, and promote the standardization of CTPA image quality control.

Materials and Methods

Dataset

This study was approved by the Medical Ethics Committee of the Affiliated Hospital of Xuzhou Medical University (XYFY2024-KL392). All procedures performed in this study involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 helsinki declaration and its later amendments or comparable ethical standards. This study utilizes medical records obtained from previous clinical diagnosis and treatment, thus exempting the need for informed consent from patients, and all data have been anonymized.

This study retrospectively collected three datasets: an internal dataset (99 cases) and two public datasets, FUMPE (21 cases) and CAD-PE (30 cases).^19,20 The internal dataset includes 99 patients who underwent CTPA between March 2022 and January 2023 at our hospital. Inclusion criteria were: (1) clinically diagnosed with acute pulmonary embolism and subsequently undergoing CTPA; (2) aged 18 years or older. Exclusion criteria were: (1) patients allergic to iodine contrast agents; (2) patients with severe cardiac, liver, or renal dysfunction; (3) pregnant or lactating women. A multi-center dataset consisting of 150 cases was created by combining the three datasets. The data were randomly divided into a training set (105 cases) and a test set (45 cases) in a 7:3 ratio.

Image Processing

The images were reviewed by a radiologist with over three years of experience, under the guidance of a radiologist with more than seven years of experience, and the former delineated the regions of interest (ROIs) on a GE Advantage Workstation 4.7 (GE Healthcare, USA). Circular ROIs were placed on the main pulmonary artery, left and right main pulmonary arteries, distal arteries of the left and right lower lobes, superior vena cava, ascending aorta, left and right main pulmonary veins, left atrium, and latissimus dorsi. The largest possible area was selected while avoiding interference from thrombus, calcification, plaques, and other factors on the vessel walls. For each ROI, the CT value and standard deviation (SD) were recorded.

Image Analysis

Objective Image Quality Evaluation

SNR and CNR were used as standard metrics for objective image quality assessment, measured according to the formulas (1) and (2) proposed by Szucs-Farkas et al,²¹ as shown below:

(1)

the SI_Vessels represents the average signal intensity of the pulmonary vessels, defined as the mean CT value obtained from five different levels: the main pulmonary artery, left main pulmonary artery, right main pulmonary artery, and distal arteries of the left and right lower lobes. Noise is defined as the standard deviation of the signal in the latissimus dorsi muscle. SI_Muscle refers to the CT value of the latissimus dorsi muscle. Additionally, the difference between the noise values of the right and left main pulmonary arteries(DIFF) is calculated to assess the impact of artifact effects.

Subjective Image Quality Evaluation

Two radiologists, different from those who delineated the ROIs and with over five - year experience in CTPA diagnosis, independently assessed the images anonymously and randomly on standardized display equipment, providing subjective opinion score based on the image quality. The subjective evaluation criteria for CTPA images are as follows: 5 = excellent quality, clear details, no artifacts, 4 = good quality, clear boundaries, minimal artifacts, suitable for clinical diagnosis, 3 = moderate image quality, with some quality issues, but still sufficient for basic diagnostic purposes, 2 = poor image quality with noticeable artifacts or blurring, reducing diagnostic confidence, 1 = very poor image quality, almost unrecognizable, containing no diagnostic information.²² The average of these subjective ratings for all images was calculated as the mean opinion score (MOS).

Feature Selection and Model Construction

The CT values and SD values of the measured ROIs, along with SI_Vessels and DIFF, were standardized as feature values. Initial feature selection was performed using Least Absolute Shrinkage and Selection Operator (LASSO) on the training set, followed by further filtering based on the Pearson correlation coefficient, retaining only features with a correlation coefficient <0.85. A random forest regression model was then constructed based on the selected features, with MOS serving as the reference standard for the model. The model’s predicted values were output and compared with SNR and CNR. The model’s predicted results are classified as follows: if at least one radiologist rates an image higher than 3, it is considered a high-quality image; otherwise, it is classified as a low-quality image. The classification results are then evaluated using a confusion matrix to ensure the rigor and comprehensiveness of the assessment.

Statistical Analysis

All data were analyzed using Python 3.8 and SPSS 26.0 software. Continuous data were expressed as mean ± standard deviation (x ± s), and Inter-group differences were analyzed using independent sample t-tests, with P ≤ 0.05 considered statistically significant. For the subjective analysis, interobserver agreement was calculated using the kappa statistic to assess the level of agreement between the two readers. A kappa value of 0.81–1.00 indicates excellent agreement, 0.61–0.80 represents substantial agreement, 0.41–0.60 denotes moderate agreement, 0.21–0.40 indicates fair agreement, and 0.00–0.20 reflects poor agreement.²³ The performance of the model was evaluated using mean squared error (MSE), R2_Score, Pearson linear correlation coefficient (PLCC), Spearman rank-order correlation coefficient (SRCC), and Kendall rank-order correlation coefficient (KRCC). MSE reflects the deviation between the model’s predicted values and the actual values, R2_score describes the model’s explanatory power for variations in the dependent variable, and the correlation coefficients (PLCC, SRCC, and KRCC) measure the degree of similarity between the two datasets.^24,25

Results

Clinical Characteristics

A total of 150 CTPA cases were included across three datasets. The internal dataset consisted of 99 patients, comprising 46 males and 53 females, with an average age of 64.5 ± 17.1 years. The FUMPE dataset included 21 cases, consisting of 10 males and 11 females, with an average age of 55.24 ± 19.32 years. The CAD-PE dataset contained 30 cases, although it lacked basic patient information. Consistency analysis of the subjective opinion scores from the two radiologists for all images in the three datasets was conducted, with the Kappa values for the three datasets being 0.648 for FUMPE dataset, 0.646 for CAD-PE dataset, and 0.664 for Internal dataset (n=99). For the combined dataset (n=150), the Kappa value was 0.673, as shown in Table 1.The average subjective opinion scores from the two radiologists were used as the MOS for subsequent analysis. MOS for the training and test sets were 3.60 and 3.68, respectively, with no statistically significant difference between them (p > 0.05).

Table 1 Interobserver Agreement of Subjective Opinion Scores Across Three Datasets

Feature Selection

A total of 24 features were measured based on CTPA images. As shown in Table 2, no statistically significant differences were found in these features between the training and test sets (P > 0.05). After feature selection, among the 150 cases, three features were retained: the CT value of the main pulmonary artery (MPA_CT), the CT value of the ascending aorta (AAO_CT), and the difference in noise values between the right and left main pulmonary arteries (DIFF). The Pearson correlation coefficients between MPA_CT and AAO_CT, MPA_CT and DIFF, and AAO_CT and DIFF were −0.14, −0.11, and 0.18, respectively, indicating no correlation among the selected features, as shown in Figures 1 and 2.

Table 2 Comparison of Feature Values Between the Training Set and the Test Set

Figure 1 The process of feature selection using LASSO: (a) Ten-fold cross-validation for tuning parameter selection in the LASSO model; (b) The path plot of the coefficients corresponding to different λ (Lambda) values in LASSO regression demonstrates how the feature coefficients change as λ varies.

Figure 2 The heatmap of CTPA image quality correlations displays the correlation analysis between MOS and the CT values and SD values of the ROIs. Positive values indicate a positive correlation, while negative values indicate a negative correlation. The greater the absolute value, the stronger the correlation between the features.

Model Evaluation

The results from the test set indicate that the random forest regression model outperformed SNR and CNR in reflecting the MOS. As shown in Table 3, the obtained MSE, R2_Score, PLCC, SRCC, and KRCC were 0.45, 0.67, 0.87, 0.87, and 0.74, respectively. As shown in Figure 3, the random forest model achieved 10 (12) true positives (TP) and 29 (33) true negatives (TN) in the evaluation. Figure 4 displays nine images representing different levels of CTPA image quality. The random forest regression model demonstrated good predictive performance across images with varying MOS scores.

Table 3 Comparison of Random Forest Regression and Traditional Metrics in CTPA Image Quality Evaluation

Figure 3 The confusion matrix for the results of three models. (a – c) represent the confusion matrix results for SNR, CNR, and Random Forest, respectively, regarding subjective image quality. Images rated higher than 3 by at least one radiologist are classified as high-quality images, while all other images are classified as low-quality images.

Figure 4 Comparison between CTPA image quality prediction results and MOS. RF, random forest regression.

Discussion

In recent years, the importance of image quality assessment has become increasingly prominent in medical imaging, with more and more studies attempting to integrate artificial intelligence (AI) technologies into this field. For example, Edenbrandt et al used deep learning to conduct a classification study on seven attributes of CT scans, such as whether they contained the corresponding positions, across eight different datasets. The accuracy rates for the anatomical regions and hip prostheses ranged from 98.4% to 100%.²⁶ Similarly, Belue et al evaluated prostate MRI T2-weighted imaging quality and demonstrated model interpretability through heatmaps, but this study was limited to single-center data.²⁷ Qi et al utilized a convolutional neural network to evaluate 173 patients who underwent PET/CT. While maintaining high - quality evaluation results, the speed was 200 times faster than that of manual evaluation.²⁸ Kashyap et al used DenseNet121 to classify X-ray films into diagnosable and non-diagnosable data. The area under the receiver operating characteristic curve was 0.93, and many X-ray films with diagnostic quality problems were removed. To date, no research has proposed an interpretable model for CTPA image quality assessments.²⁹

This study builds an interpretable machine learning model using multi-center data. Among the features included in the model, the CT value of the main pulmonary artery (MPA_CT) showed a positive correlation with MOS. Pearson correlation analysis also revealed that the CT values of the main pulmonary artery and its left and right branches were highly correlated, suggesting strong consistency in image quality assessment. However, the CT value of the ascending aorta showed a negative correlation with both CTPA image quality and MOS, which may be due to the influence of pulmonary venous filling on the observation of the pulmonary arteries, affecting the quality of 3D reconstruction. Additionally, DIFF also showed a negative correlation, likely related to radial artifact caused by excessive contrast agent concentration in the superior vena cava, leading to unclear visualization of the right pulmonary artery and increasing the risk of false-positive pulmonary embolism. The three features selected for this model are highly relevant to clinical practice and demonstrate good interpretability.

In evaluating the model’s performance, MSE reflects the deviation between the predicted and actual values, with values closer to 0 indicating better model performance. The R2_Score describes the model’s ability to explain the variation in the dependent variable, with values closer to 1 indicating better performance.²⁴ The correlation coefficients (such as PLCC, SRCC, and KRCC) measure the degree of similarity between two datasets. In this study, these coefficients showed a strong correlation, indicating that the model effectively reflects the subjective scoring results. Experimental results show that, compared to traditional SNR and CNR, the random forest regression model provides a more accurate and clinically relevant objective assessment of CTPA image quality.

With the continuous development of CT technology, research on image quality has been increasing. However, objective image quality evaluation methods in these studies still predominantly rely on SNR and CNR.^12–15 The model developed in this study offers a more accurate reflection of CTPA image quality compared to SNR and CNR, and Its’ MSE, R2_Score, PLCC, SRCC, KRCC all increased by more than 0.3. The objective metrics selected by this model are not only easy to measure but also demonstrate good consistency with subjective ratings. The application of this model can simplify the image quality evaluation process, providing an innovative approach to image quality control and offering a more reliable objective evaluation method for comparative image studies.

Another advantage of this model lies in its effective support for downstream tasks in deep learning. In recent years, deep learning techniques have been widely applied in CTPA for tasks such as PE detection, classification, and segmentation.^30,31 For example, Lanza et al carried out tasks such as pulmonary embolism detection and thrombus volume measurement on the RSPECT dataset. This study utilized the dataset to complete multiple tasks including detection and segmentation.³² However, Low-quality images can significantly impact the performance of deep learning models and increase training difficulty. The model designed in this study can effectively mitigate the negative impact of low-quality images on deep learning algorithms during the training phase, thus providing strong support for the application of deep learning technologies in the CTPA domain.

This study also has limitations. Firstly, although a multi-center dataset was used, it only included data from three centers, two of which were public datasets. One of the datasets lacked detailed patient background information, and the total sample size of 150 cases is still relatively small. Future research should expand the data sources and conduct larger-scale multi-center studies to enhance the generalizability and practical value of the model. Specifically, we should include more publicly available datasets from other regions. Secondly, this study mainly relied on subjective scoring by radiologists from a single center. For future research, it is necessary to seek multi-center cooperation and adopt a method of scoring by radiologists from multiple centers to determine the MOS. Additionally, the impact of respiratory artifacts on image quality was not sufficiently considered, and future research could incorporate algorithms that automatically identify respiratory artifacts.

Conclusion

In conclusion, this study successfully developed an interpretable machine learning model for the objective assessment of CTPA image quality. The model effectively reflects traditional subjective evaluation results and objectively reveals the quality level of CTPA images. This helps improve the efficiency of image quality control, facilitating better quality control practices and further enhancing image quality.