Interobserver Variability in Hysterosalpingography Interpretation: Radiologists vs Gynecologists

Introduction

Hysterosalpingography (HSG) is a well-established, minimally invasive diagnostic imaging technique used for decades to evaluate the female reproductive system, specifically the uterine cavity and fallopian tubes.¹ This procedure primarily investigates the cause of infertility, recurrent pregnancy loss, and other gynecological conditions.² HSG involves the injection of radiopaque contrast material into the uterine cavity and fallopian tubes, followed by fluoroscopic or radiographic imaging to assess tubal patency and uterine abnormalities.³

The interpretation of HSG examinations can be subjective and may be influenced by the observer’s experience level, training, and specialization.⁴ Interobserver variability has been identified as a significant issue in assessing HSG examinations, potentially leading to discrepancies in clinical management and patient care.⁵ To address this concern, it is essential to understand the extent of interobserver variability between different medical professionals involved in interpreting HSG images, such as radiologists and gynecologists.

Although several studies have examined interobserver variability in HSG interpretations, most of these investigations have compared the agreement between radiologists or gynecologists.^4,6 Few studies have directly compared the interpretations of radiologists and gynecologists or investigated the potential impact of interdisciplinary collaboration on HSG assessment accuracy.^5,6 Therefore, the primary aim of this manuscript is to evaluate and compare the interobserver variability between radiologists, gynecologists, and radiologists vs gynecologists in the interpretation of HSG examinations.

Recent guidelines continue to recognize hysterosalpingography as an important first-line investigation in the infertility work-up, particularly in resource-limited settings where advanced modalities are not universally available.⁷ At the same time, these guidelines emphasize the increasing role of newer sonographic approaches, such as hysterosalpingo-contrast sonography (HyCoSy) and three-dimensional (3D) sonohysterography, which provide high diagnostic accuracy for tubal patency and uterine abnormalities while reducing patient discomfort and avoiding ionizing radiation.⁸ These advances highlight the evolving landscape of reproductive imaging: while HSG remains widely used and accessible worldwide, modern sonographic alternatives are increasingly recommended as complementary or even preferential techniques in contemporary infertility practice. Against this backdrop, understanding the variability and reliability of HSG interpretation among different specialists remains highly relevant, as it continues to impact clinical decision-making and patient outcomes on a global scale.

Material and Methods

Written informed consent from the patients and ethical approval of the hospital committee was obtained (2021–147) for this prospective, multicentric study. Six radiologists, six gynecologists, two referees (A.K. and B.G)., one combiner of data, a statistical analyzer, a writer and corresponding author of the study (C.K), and a total of 15 specialists joined the investigation.

The study was approved by the Ethics Committee of Antalya Training and Research Hospital (Approval No: 2021–147) and conducted in accordance with the principles of the Declaration of Helsinki.

100 HSG patients, performed in a single tertiary care referral center with the same imaging equipment and team in 2021, were transferred to the gynecologists and radiologists in different hospitals and cities via the internet in Dicom format. The patient’s written informed consent and clinical histories were added to the transferring process.

Six radiologists (median 19 years’ experience, range 12–25 years) with routine reporting of HSG, and six gynecologists (median 16 years’ infertility practice, range 12–22 years) familiar with HSG interpretation participated. No study-specific training was provided to preserve real-world variability.

Specialists were asked to evaluate the images of 100 patients with about 10 probable predefined findings/diagnoses and allowed to add a new finding out of the probabilities if they exist. Predefined findings/diagnoses were presented as: 1) No finding or comment, 2) Normal findings, 3) Fimbrial phimosis, 4) Hydrosalpinx (Uni or bilateral), 5) Isthmocele, 6) Cavitary filling defects, 7) Tubal occlusion (Uni or bilateral), 8) Congenital uterine anomaly (septum, bicornuat/unicornuat uterus etc.), 9) Additional imaging suggestion, 10) Additional finding out of the list, and 11) The quality of the examination.

The images of the patients were evaluated for 10 predefined findings above, and the examination quality by 12 specialists. Each predefined finding and the quality of examination were evaluated separately for interobserver variability between radiologists, gynecologists, and radiologists vs gynecologists.

Each specialist completed the examination of the HSG exams in a period and transferred the findings in Excel format to the first referee (A.K.), who numbered the specialist’s report and transferred the blinded report to the second referee (B.G.), who checked the reports about compatibility with the instruction and forwarded to C.K., who combined, analyzed and written the study results. C.K. transferred the manuscript to both referees for review before submitting it to the journal.

Statistical Analysis

Sample Size Calculation

We calculated the sample size using a confidence-interval approach for Fleiss’ κ with multiple raters, as recommended by Rotondi and Donner.⁹ Assuming 12 raters, binary outcomes, an anticipated κ of 0.45, marginal category prevalence of 0.50, and two-sided α=0.05, the Rotondi–Donner framework indicates that approximately 96–104 subjects are required to achieve a 95% CI half-width of ≈0.06–0.07 (ie, an expected lower 95% CI bound ≥0.35). Accordingly, our sample of 100 consecutive HSG cases provides adequate precision for the primary agreement analyses.

We assessed the interobserver agreement among multiple raters using Fleiss’ kappa coefficient (κ) for various categorical variables. Data were collected from 12 raters who independently evaluated 100 cases. The categorical variables under consideration included [list variables]. All statistical analyses were performed using the Statistical Package for the Social Sciences (SPSS) version 25. To compute Fleiss’ kappa in SPSS, we first prepared a data matrix in which each row represented a case, and each column represented a rater. We then used the “Reliability Analysis” function in SPSS to calculate Fleiss’ kappa coefficient for each categorical variable. Fleiss’ kappa coefficient (κ) ranges from −1 to 1, with values above 0.8 indicating excellent agreement, 0.61–0.8 substantial agreement, 0.41–0.6 moderate agreement, 0.21–0.4 fair agreement, and below 0.2 poor agreement. For each category, Fleiss’ κ and 95% confidence intervals were calculated for radiologists, gynecologists, and cross-specialty agreement.

Results

Fleiss’ κ coefficients were calculated to assess interobserver agreement for 100 hysterosalpingography (HSG) examinations interpreted by six radiologists, six gynecologists, and across both specialties. Overall, agreement ranged from poor to moderate depending on the diagnostic category.

Cross-Specialty Agreement (Radiologists vs Gynecologists)

Agreement between the two specialties was highest for tubal occlusion (κ=0.508, 95% CI: 0.484–0.532), followed by uterine anomalies (κ=0.444, 95% CI: 0.420–0.468) and hydrosalpinx (κ=0.373, 95% CI: 0.349–0.397). In contrast, subjective variables such as additional findings (κ=0.036), additional examination recommendation (κ=0.087), and overall image quality (κ=0.000) demonstrated very poor reproducibility, indicating a lack of consensus beyond chance levels (Table 1).

Table 1 Interobserver Variability Between Gynecologists and Radiologists

Agreement Among Gynecologists

Gynecologists showed their strongest consistency for hydrosalpinx (κ=0.501, 95% CI: 0.450–0.551), uterine anomalies (κ=0.496, 95% CI: 0.445–0.546), and tubal occlusion (κ=0.460, 95% CI: 0.409–0.511). Similar to cross-specialty findings, subjective categories such as additional findings (κ=−0.015) and quality of examination (κ=−0.043) showed disagreement below chance, reflecting high variability in interpretation (Table 2).

Table 2 Interobserver Variability Between Gynecologists

Agreement Among Radiologists

Radiologists achieved their highest agreement for tubal occlusion (κ=0.536, 95% CI: 0.486–0.587), followed by uterine anomalies (κ=0.386, 95% CI: 0.335–0.436) and cavity filling defects (κ=0.266, 95% CI: 0.215–0.316). Consistency was again poor for subjective variables, with κ values close to zero for image quality (κ=0.023) and additional recommendations (κ=0.084) (Table 3).

Table 3 Interobserver Variability Between Radiologists

Overall Patterns

Across all groups, structured and anatomically defined findings (tubal occlusion, uterine anomalies, hydrosalpinx) demonstrated the highest reproducibility, with κ values in the moderate range. Conversely, parameters relying on subjective judgment (eg, examination quality, additional findings, or need for further imaging) consistently showed poor or negative agreement, highlighting their unreliability. These findings underline that interobserver variability is most pronounced when interpretation depends on subjective assessment rather than structured diagnostic criteria.

Discussion

This prospective multicenter study reveals significant interobserver variability in the interpretation of hysterosalpingography (HSG) examinations among radiologists, gynecologists, and across both specialties. Overall, interobserver agreement ranged from poor to moderate, with the highest concordance observed for more structured and anatomically defined findings such as tubal occlusion (κ = 0.508), uterine anomalies (κ = 0.444), and hydrosalpinx (κ = 0.373). Conversely, findings requiring subjective judgment, such as overall image quality and recommendations for further imaging, demonstrated very low or even negative kappa values, underscoring the inconsistency in interpreting such parameters.

The observed variability has direct clinical implications for infertility management. Discrepancies in interpreting key findings such as tubal occlusion or uterine anomalies may lead to misdiagnosis, unnecessary invasive procedures, or delayed treatment. For example, overdiagnosis of tubal occlusion could prompt unwarranted laparoscopy, whereas underdiagnosis may postpone assisted reproductive techniques. Our findings emphasize that structured reporting and interdisciplinary collaboration are essential to minimize such risks.

These findings are in alignment with previous studies emphasizing the variability of HSG interpretation. Glastein et al first reported considerable observer variability in diagnosing and managing HSG examinations, which could significantly influence treatment decisions in infertility workups.⁴ Similarly, Renbaum et al found that clinicians and radiologists frequently disagreed on key HSG findings, such as tubal blockage and uterine filling defects, leading to potential disparities in patient management.⁶

Our study contributes additional evidence supporting these concerns by providing a comparative analysis among radiologists, gynecologists, and across both specialties. Agreement was slightly higher among radiologists than among gynecologists for most structured criteria. For instance, radiologists demonstrated the highest interobserver agreement for tubal occlusion (κ = 0.536), while gynecologists showed strongest agreement for hydrosalpinx (κ = 0.501) and uterine anomalies (κ = 0.496). This may reflect differences in training emphasis—radiologists may be more attuned to subtle image-based signs of tubal pathology, while gynecologists may better recognize clinically significant uterine variants.

Nonetheless, the consistently poor agreement on subjective parameters such as “quality of the examination” (κ = 0.000 among radiologists vs gynecologists; −0.043 among gynecologists) highlights the inherent subjectivity and the lack of standardized criteria for such assessments. Similar conclusions were drawn by Mol et al, who noted that lack of standardization in HSG reporting leads to variability even among experienced readers.¹⁰

Structured reporting systems may offer a solution to this problem. Standardized templates using consistent terminology—already employed successfully in radiology for modalities like breast imaging (BI-RADS) and CT colonography—have been shown to enhance interobserver agreement and diagnostic confidence.^11,12 Güler Okyay et al previously emphasized the potential benefits of similar standardization in HSG evaluation.⁵ Incorporating such structured approaches could unify interpretations across specialties.

Another strategy to improve diagnostic consistency involves multimodal imaging. Three-dimensional (3D) sonohysterography and hysterosalpingo-contrast sonography (HyCoSy) have shown promise in enhancing diagnostic accuracy and interobserver agreement, particularly for uterine anomalies and tubal patency.^13,14 Studies by Deutch and Abuhamad found that 3D ultrasound surpasses HSG in differentiating septate from bicornuate uteri, reducing diagnostic ambiguity.¹³

Technological advancements, particularly artificial intelligence (AI)-based diagnostic tools, also hold promise for enhancing objectivity and reducing variability. AI has demonstrated robust performance in pattern recognition and decision support across various imaging modalities.¹⁵ Although not yet widely applied in HSG, early studies suggest that machine learning algorithms may aid in the automated assessment of tubal patency and cavity morphology, offering reproducible results with minimal observer bias.¹⁶

It is also essential to promote interdisciplinary training and collaboration to reduce variability in diagnostic practices. Integrating AI-based decision support systems may help standardize interpretation criteria and minimize diagnostic errors, in line with recommendations from diagnostic accuracy reporting guidelines such as STARD 2015.¹⁷ Future research should focus on evaluating the effectiveness of these collaborative and technology-assisted approaches, particularly in the context of complex infertility assessments.

The main strengths of this study include its prospective, multicenter design and the relatively large number of raters (six radiologists and six gynecologists) evaluating 100 consecutive HSG cases. Compared with prior reports, this represents one of the largest cross-specialty assessments of interobserver variability in HSG interpretation. The use of predefined diagnostic categories and blinded evaluation further reduced bias and allowed for a systematic comparison. Finally, the direct analysis of agreement both within and between specialties provides novel insights into the complementary perspectives of radiologists and gynecologists in infertility diagnostics.

Several limitations should be noted. First, although our sample size was statistically adequate for κ estimation, all HSG examinations were obtained from a single center, which may limit generalizability to different clinical environments. Second, while we included specialists from multiple institutions, the absence of standardized training protocols across radiologists and gynecologists likely contributed to variability, as highlighted by the negative κ values for subjective parameters. Third, potential institutional bias and the lack of stratification by individual reader experience represent additional limitations.

Regarding the potential role of artificial intelligence, our findings support the concept that AI-assisted tools may help reduce subjectivity and improve reproducibility; however, such systems remain experimental in HSG imaging and have not yet been validated in large-scale prospective studies. Their clinical integration will require careful evaluation, including cost-effectiveness and workflow impact.

From a clinical standpoint, the variability observed in this study has direct consequences for infertility management. Misclassification of tubal occlusion, for example, could lead to unnecessary laparoscopic procedures, while underdiagnosis might delay appropriate assisted reproductive interventions. These findings emphasize the urgent need for structured reporting templates and interdisciplinary collaboration to ensure consistent and accurate HSG interpretation in routine practice.

Conclusion

Our findings underscore considerable interobserver variability in the interpretation of HSG, particularly between radiologists and gynecologists. While structured and objective findings such as tubal occlusion and uterine anomalies yield moderate agreement, subjective assessments show poor reproducibility. These results highlight the risk of misdiagnosis, unnecessary interventions, or delayed treatment in infertility care if HSG interpretations are not standardized. To minimize variability, structured reporting systems and interdisciplinary training should be implemented, enabling radiologists and gynecologists to combine their complementary expertise.

Moreover, while artificial intelligence holds promise as a supportive tool to reduce subjectivity, its integration into HSG practice requires further validation through large-scale, prospective studies. Future research should focus on evaluating such technology alongside multimodal imaging approaches, with the ultimate goal of improving diagnostic consistency, clinical decision-making, and patient outcomes in infertility management.