PAIR: Evaluating the Limits of Agreement Among Non-Retinal Specialist Using PathFinder Artificial Intelligence Tool for Retinal Disease Referrals: A Prospective Observational Study

Introduction

Artificial intelligence (AI) has emerged as a powerful tool in ophthalmic imaging, particularly for the automated interpretation of optical coherence tomography (OCT) scans.¹ By reducing reliance on subjective human interpretation, AI enables faster and more consistent detection of retinal pathologies such as choroidal neovascularization (CNV), diabetic macular edema (DME), and age-related macular degeneration (AMD). Traditional OCT analysis can be time-consuming and prone to interobserver variability, whereas deep learning (DL) based models have shown remarkable success in automating tasks such as image classification, layer segmentation, and fluid detection.^2,3

Several studies have demonstrated that convolutional neural networks (CNNs) can accurately classify retinal diseases on OCT images and even outperform conventional methods. Nigjeh et al⁴ reported enhanced accuracy for CNV and DME classification using VGG16 and ResNet18 models, while Umair et al developed OculusNet, an explainable AI framework capable of identifying complex pathological patterns.⁵ Similarly, Ying et al showed that AI-based segmentation of hyperreflective foci improves diagnostic precision across multiple retinal diseases.⁶ Beyond classification, AI systems also enhance image quality through noise reduction and super-resolution, improving interpretability and diagnostic workflow.⁷

Automated OCT interpretation is particularly valuable in settings with limited access to retina specialists. Studies have shown that machine-learning-based systems can reliably identify vision-threatening diseases and assist general ophthalmologists in triage and referral decisions.^8,9 These advances signal a paradigm shift toward integrated, clinician-assistive imaging technologies.

The CIRRUS PathFinder AI (Carl Zeiss Meditec AG, Germany) represents one such system, embedded within the OCT console to provide real-time, automated analysis. Trained on more than 75,000 OCT B-scans and validated by expert retina specialists, PathFinder demonstrates approximately 88% sensitivity and 93% specificity in detecting macular findings such as subretinal and intraretinal fluid, retinal pigment epithelium (RPE) changes, and photoreceptor disruption.¹⁰ It also facilitates longitudinal monitoring of the fellow eye by automatically tracking anatomical changes over time.

Despite extensive research on AI-based OCT interpretation, most existing models are device-agnostic, and their accuracy varies across platforms due to differences in imaging technology. Unlike most device-agnostic OCT AI models evaluated retrospectively, PathFinder is an on-device, real-time decision-support system integrated directly into the CIRRUS OCT workflow, providing image quality assessment and pathology flagging at the point of acquisition and interpretation. However, its real-world diagnostic performance and clinical utility have not been independently validated. The present study aims to evaluate the agreement between non-retina specialists assisted by PathFinder AI and fellowship-trained retina specialists in diagnosing macular pathologies and determining referral decisions in a real-world setting. This work seeks to establish whether on-console and real-time AI assistance can enhance diagnostic consistency and streamline referral workflows in everyday clinical practice.

Methods

This was a cross-sectional diagnostic agreement study that included all consecutive patients undergoing OCT evaluation for suspected or confirmed retinal pathology during the April to July 2025 study period. The study adhered to the tenets of the Declaration of Helsinki and received approval from OasisEye Specialists Medical Advisory Board Research Ethics Committee (Approval number: OASIS-PF-025). The OasisEye Specialists Medical Advisory Board Research Ethics Committee determined that patient consent to review their medical record was not required as it was a non-invasive prospective study, and the patients’ data maintains confidentiality.

Each patient’s demographic and clinical information, including age, gender, presenting complaints, best-corrected visual acuity (BCVA), refraction, intraocular pressure, slit-lamp and fundus examination findings, and relevant systemic comorbidities, was recorded.

OCT scans were acquired using the Cirrus OCT platform (Carl Zeiss Meditec AG, Germany) following a standardised 6×6 mm macular raster scan protocol. Only scans of high image quality were included for analysis, defined as those with a signal strength ≥80%. Images below this threshold or those affected by segmentation artefacts were excluded. Additionally, eyes with significant media opacity resulting in poor scan quality (eg., dense cataract or corneal scarring) were excluded, as these could compromise OCT signal strength and limit reliable AI-assisted interpretation. We also excluded eyes with primary pathologies outside the 6×6 mm macular raster scan protocol (eg., optic nerve head disorders or advanced glaucomatous damage), as these conditions fall beyond the anatomical scope of the PathFinder macular analysis.

The PathFinder AI module (Carl Zeiss Meditec AG, Germany), integrated within the Cirrus OCT platform, employs deep-learning algorithms to automatically assess OCT image quality and detect areas of potential pathology within the macular cube. It functions by acquiring dense 512×128 Smart Cube scans of the macula, automatically centered on the fovea, and subsequently qualifying each scan for image quality using neural networks trained to distinguish good from poor-quality images. During analysis, the software detects the presence of structural abnormalities consistent with eight predefined retinal pathologies or conditions and visually flags them for the operator, yellow indicators for poor-quality scans and red markers for B-scans containing suspicious findings. The system is fully embedded within the Cirrus review software and can be activated or deactivated as needed on the reviewer console.

All OCT scans were independently reviewed by three fellowship-trained retina specialists (RS1, RS2, RS3) and three non-retina specialists (NRS1, NRS2, NRS3). Each observer provided a diagnosis and a referral opinion on whether the patient should be referred to a retina specialist. The retina specialists were masked to all clinical and demographic details, including age, gender, BCVA, refractive status, intraocular pressure, cataract status, presenting complaints, and fundus findings, and based their evaluations solely on the OCT images. They did not have access to the Pathfinder AI software either.

The non-retina specialists had access to the full clinical profile, including BCVA, and systemic information, and made their diagnoses using PathFinder assistance. They were asked to record their confidence level in each diagnosis using a five-point Likert scale (Not confident, Slightly confident, Moderately confident, Very confident, and Completely confident). The time taken by the non-retina specialists with PathFinder AI assistance to indicate the probable pathology was recorded and categorised into six predefined intervals (<5, 6–10, 11–15, 16–20, 21–25, 26–30, and >30 seconds).

The primary outcome measure of this study was the level of agreement between the three non-retina specialists and the retina specialists (taken as the gold standard) in two key domains: (1) the diagnosis assigned based on OCT interpretation, and (2) the referral decision, ie., whether the case warranted referral to a retina specialist.

Statistical Analysis

Descriptive statistics were used to summarise the data. Continuous variables (eg., age) were reported as means with standard deviations, while categorical variables (eg., sex, diagnostic categories, referral status) were summarised as frequencies and percentages (n, %).

When retina specialists differed in their assessments, the gold standard for both diagnosis and referral was defined as the category on which at least two of the three specialists agreed. If all three specialists disagreed, the diagnosis or referral status assigned by the most experienced specialist (KF) was taken as the reference standard. The majority rule was also applied to the NRS group to determine the final NRS diagnosis for each case, ensuring true diagnostic discordance rates between the three RS and three NRS graders.

Inter-rater agreement among the three retina specialists was assessed using Cohen’s kappa (κ) with 95% confidence intervals (Wilson’s Method) for pairwise comparisons and Fleiss’ kappa for overall agreement. Agreement between each non-retina specialist and the gold standard was also quantified using Cohen’s kappa.

For the four most common diagnostic categories, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for each non-retina specialist relative to the gold standard. Discordance rates in referral decisions were determined by comparing each non-retina specialist’s decision with the gold standard. To explore determinants of discordant referral, logistic regression analysis was performed with discordance as the dependent variable and age, baseline visual acuity, diagnosis made by the non-retina specialist, their confidence in the diagnosis and time taken by the Pathfinder assistant to arrive at the diagnosis as predictors. Model performance was evaluated through goodness-of-fit (Hosmer–Lemeshow test), discrimination (area under the ROC curve), and calibration.

All analyses were performed using R (version 4.3.3, packages: psych, irr, epiR, dplyr, tidyr, ggplot2). Statistical significance was set at p < 0.05.

Results

Participant Demographics

A total of 202 participants were included, with a mean age of 62.7 ± 12.3 years (median 64, IQR 56–72, range 30–91 years). There were 96 males (48%) and 106 females (52%). Common systemic comorbidities included hypertension (n=81, 40%), hyperlipidemia (n=75, 37%) and diabetes mellitus (n=47, 23%). The mean visual acuity at time of assessment was 0.44 ± 0.47 logMAR (6/15 Snellen’s equivalent) (median = 0.3 logMAR (IQR=0.14 to 0.5logMAR, range= 0 to 2 logMAR). The commonest gold-standard diagnosis, as agreed by at least 2 retina specialists, was epiretinal membrane (n=86, 43%), followed by AMD (n=35, 17%), normal OCT (n=30, 15%), macular hole (n=10, 5%) and macular oedema (n=9, 4.5%).

Agreement Between Retina Specialists

For diagnosis (Table 1), pairwise agreement between the three retina specialists ranged from moderate to substantial, with an overall Fleiss’ kappa of 0.59, indicating moderate agreement. When considering the ratings of the three retina specialists, complete agreement across all three was seen in 111 cases (55%), while partial agreement (two specialists agreeing) was observed in 74 cases (38%). In 14 cases (7%), all three specialists disagreed. Using the majority rule (at least two raters in agreement), the overall agreement was 93% (95% CI: 88–96%). Amongst the individual disease entities, the highest agreement was seen in ERM (k=0.86) and the lowest in AMD (k=0.56) (Figure 1).

Table 1 Agreement Between the Three Retina Specialists for Diagnosis and Referral (Kappa, 95% CI)

Figure 1 Alluvial diagram showing inter-rater agreement among the three retina specialists (RS1, RS2, RS3) for the top five primary diagnoses and “other” category.

For referral decisions, the agreement was consistently moderate, with an overall Fleiss’ kappa of 0.47 (Table 1). In absolute terms, the three retina specialists showed total agreement in 157 cases (78%) and partial agreement (two specialists agreeing) in 45 cases (22%). There were no instances of complete disagreement across all three raters. When applying the majority rule (at least two specialists agreeing), this translated into 100% agreement on referral versus non-referral decisions.

Agreement of Non-Retina Specialists with the Gold-Standard Diagnosis

When compared with the gold-standard diagnosis, agreement varied across the three non-retina specialists. NRS1 demonstrated substantial agreement (κ = 0.78, Table 2), whereas NRS2 showed moderate agreement (κ = 0.64). NRS3 had the lowest performance, with moderate agreement at the lower end (κ = 0.54). Overall, non-retina specialists expressed high levels of confidence when using PathFinder for diagnosis. Both NRS1 and NRS3 reported being very confident in the vast majority of cases (98% and 96% respectively), while NRS2 showed lower confidence, with just over half of the cases (55%) rated as very confident (Table 2).

Table 2 Agreement of Non-Retina Specialists to Gold-Standard Diagnosis and Referral

The time taken by non-retina specialists to arrive at a diagnosis with PathFinder was generally short. For all three specialists, the majority of cases were interpreted within 11–20 seconds, with smaller proportions requiring either shorter (<10 seconds) or longer (>20 seconds) durations. Very rapid interpretations (<5 seconds) were uncommon, while only a minority of cases extended beyond 30 seconds. Overall, this indicates that the software enabled relatively quick decision-making across users.

Diagnostic Accuracy of Non-Retina Specialists

The ability of non-retina specialists to detect specific retinal pathologies using PathFinder varied across diseases and raters (Table 2). Sensitivity was generally high for cystoid macular edema and epiretinal membrane, but comparatively lower for age-related macular degeneration and macular hole, particularly for some raters (Figure 2). Specificity was consistently high across all conditions, often exceeding 90%, indicating a strong ability to identify normal cases correctly. Positive predictive values and kappa statistics, however, showed more variability, with lower values for certain raters in conditions such as cystoid macular edema and macular hole, reflecting occasional false positives and inconsistent diagnostic agreement with the gold standard. Of note, rater-level analysis revealed marked variability in positive predictive value for macular hole. NRS3 demonstrated a substantially lower PPV (0.39) for macular hole compared with the other non-retina specialists. Closer examination showed that NRS3 overdiagnosed macular hole in 14 cases, which were classified as epiretinal membrane (n = 12) or vitreomacular traction (n = 2) by majority retina specialist consensus. Representative OCT images of these cases show ERM-associated pseudoholes in these cases (Figure 3).

Figure 2 Diagnostic performance of non-retina specialists (NRS1–NRS3) compared with the gold-standard diagnosis established by retina specialists for the top four diagnostic categories.

Figure 3 An OCT image with PathFinder evaluation of an ERM with pseudohole diagnosed as macular hole by NRS3 and referred but it was not in agreement to RS.

Referral Agreement of Non-Retina Specialists with the Gold Standard

Referral patterns showed greater variability across non-retina specialists compared to diagnostic agreement. NRS1 demonstrated substantial agreement with the gold standard (κ = 0.68), while both NRS2 (κ = 0.24) and NRS3 (κ = 0.34) achieved only poor agreement in referral decisions (Figure 4). The lower referral agreement for NRS3 was primarily driven by false-positive diagnoses of macular hole, which led to over-referral in cases that were classified as epiretinal membrane or vitreomacular traction by the retina specialist panel.

Figure 4 An OCT image of an Epiretinal Membrane with 20/20 vision with PathFinder evaluation that NRS diagnosed in agreement to RS but did not refer.

Predictors of Discordance for Referral

Given that NRS2 had the lowest agreement with the gold standard for referrals, a logistic regression analysis was performed to identify predictors of discordance (Table 3) with NRS2. The only significant factor was visual acuity, where each 1-line worsening in vision was associated with a 34% lower odds of discordance in referral (OR 0.66, 95% CI 0.55–0.80, p < 0.001), indicating that poorer vision increased the likelihood of concordance with the gold-standard referral. Other factors, including age, gender, underlying retinal diagnosis (AMD, CME, ERM), self-reported confidence level, and interpretation time with PathFinder, were not significantly associated with referral discordance. The logistic regression model showed good fit and discrimination (Hosmer-Lemeshow p = 0.84, AUC = 0.80, McFadden’s R² = 0.24, Somers’ D = 0.61), with good calibration.

Table 3 Logistic Regression for Predictors of Discordance Between the Referral Preference by NRS2 and the Majority Referral Patterns Expected by Retina Specialists

Discordance of Gold Standard Graded Normal and Non-Retina Specialists Graded Pathology

The most significant finding of discordance in diagnosis for RS graded normal eyes is the high frequency of epiretinal membrane, accounting for 40.0% of all false-positive diagnoses (Table 4). This is followed by PED at 20.0% and AMD at 13.3%. These results are yielded across thirteen eyes with thirty false-positive diagnoses made by the three PathFinder assisted NRS, that were truly normal according to the retina specialists (Figure 5).

Table 4 Discordance of Gold Standard Graded Normal and Non-Retina Specialists Graded Pathology

Figure 5 An OCT image with PathFinder evaluation of a case ERM that one NRS got right and two got wrong.

Discussion

In this study, we evaluated the diagnostic and referral performance of non-retina specialists using the PathFinder AI assistant, benchmarked against gold-standard diagnoses and referrals derived from retina specialists. Overall, retina specialists demonstrated moderate agreement with each other for both diagnosis and referral, though majority-rule analysis where at least two agreed, yielded very high agreement. Non-retina specialists, when aided by PathFinder, achieved substantial to moderate agreement with the gold-standard diagnosis, with performance varying across raters and disease categories. Importantly, specificity remained consistently high across conditions, while sensitivity and predictive values showed greater variability. Confidence in diagnosis was generally high and interpretation times were short, underscoring the usability of the software in real-world settings. However, referral agreement was notably weaker, particularly for two non-retina specialists. Further analysis suggested that NRS2, and likely NRS3, relied more on visual acuity than OCT characteristics when deciding on referrals. Since retina specialists in our study were masked to visual status, this difference in approach likely contributed to higher discordance rates. A key concern is that patients with preserved vision but with pathologies such as ERM or AMD could be under-referred, despite retina specialists considering referral appropriate. It is also noted that subtle and early signs of pathologies such as ERM, PED, and AMD, can be picked up by PathFinder. This is seen in the diagnosis discordance of non-retina specialists to those graded normal by gold standard. This is due to retina specialists who would consider these findings clinically insignificant.

Artificial-intelligence applications in ophthalmic imaging have evolved rapidly, particularly in OCT interpretation, where large volumes of data demand efficient and reproducible analysis. Early work by De Fauw et al¹¹ demonstrated that a deep-learning system could identify retinal diseases and generate referral recommendations directly from OCT volumes, matching the diagnostic performance of expert ophthalmologists. Similarly, Schlegl et al¹² developed a fully automated pipeline for fluid detection and quantification on OCT, capable of distinguishing intraretinal and subretinal fluid in neovascular AMD. Later studies refined these approaches, such as Cao et al,¹³ who used deep-learning segmentation to monitor treatment response in DME, and Yim et al,¹⁴ who built a model predicting future conversion from dry to neovascular AMD using baseline OCT features. Collectively, these investigations established that AI can interpret OCT images with diagnostic accuracy comparable to human graders while also enabling longitudinal disease tracking.

Most of these systems, however, were device-agnostic research models requiring external data processing and dedicated computational resources. They typically functioned offline or via cloud-based platforms and were not integrated into the OCT acquisition workflow. In contrast, the PathFinder AI is embedded directly within the Cirrus OCT console, performing real-time image-quality assessment, retinal-layer segmentation, and pathology detection during scan acquisition. This design eliminates the need for data export or external analysis, enabling immediate feedback to technicians and clinicians. The present study is, to our knowledge, among the first to evaluate such a commercially available, on-device OCT-AI system in a real-world clinical setting.

Our results show that non-retina specialists, when assisted by PathFinder, could diagnose common macular pathologies such as epiretinal membrane and cystoid macular edema with high accuracy and confidence. Specificity across observers remained consistently high, suggesting that the AI tool effectively reduces false positives and helps rule out normal scans. Sensitivity was lower for subtle or early manifestations of age-related macular degeneration and macular holes, indicating that AI assistance complements but does not replace clinical expertise. Importantly, the substantial interobserver confidence observed in this study suggests that AI integration not only enhances diagnostic accuracy but also reinforces clinician assurance, an essential factor for widespread adoption. However, isolated deviations in diagnostic performance for certain conditions and users (for example, a higher false-positive rate for macular hole in one rater) warrant caution. Closer examination revealed that most of these cases represented pseudoholes secondary to ERM rather than true full-thickness macular holes, reflecting classification challenges within the vitreomacular interface spectrum rather than systematic AI error. This suggests that optimal use of AI assistance may benefit from additional training in OCT interpretation and effective use of AI-based decision-support tools.

The implications are particularly relevant for resource-constrained or high-volume clinical environments, where retina specialists may not be readily available. Integrating AI-enabled OCT devices at the point of care could allow comprehensive ophthalmologists, optometrists, or even trained technicians to detect pathology earlier and make more accurate referral decisions. Such an approach could transform routine imaging centers into front-line retinal screening hubs, improving accessibility to specialized care.

Previous studies of OCT-based AI systems have reported very high diagnostic performance. For example, in a community screening platform by Liu et al,¹⁵ the OCT-AI model achieved a sensitivity of 96.6% (95% CI 91.8–98.7%) and specificity of 98.8% (95% CI 98.0–99.3%) for detecting urgent retinal cases, and 98.5% sensitivity (95% CI 96.5–99.4%) with 96.2% specificity (95% CI 94.6–97.3%) when both urgent and routine cases were included. Similarly, Bai et al¹⁶ evaluated AI-assisted OCT for detection of 15 retinal disorders and found sensitivities ranging from 87.65% to 100% and specificities from 80.12% to 99.41%. In comparison, our study using the PathFinder AI assisted by non-retina specialists demonstrated similarly high specificity across conditions, but somewhat lower sensitivity for certain pathologies (for example, our sensitivity for AMD and macular hole was below those ranges). Several factors likely contribute to this difference in sensitivity: in our real-world clinical workflow, non-retina specialists were making diagnostic and referral decisions rather than the AI operating in isolation; the pathology spectrum included subtle and mixed presentations rather than screening-only cases; the PathFinder is device-embedded and optimized for the CIRRUS OCT environment rather than a retrospective algorithm trained on multiple devices; and non-retina specialists may rely differently on clinical information in conjunction with AI output, potentially affecting detection thresholds, particularly for referral decisions that are influenced by visual acuity, symptom burden, anticipated treatment benefit, local management capability, and cost considerations. This is also substantiated by the fact that even the three retina specialists agreed only moderately on the final diagnosis and referral decisions. These differences suggest that while PathFinder performs well in a practical setting with high specificity, careful attention is required to sensitivity limitations when deploying in non-specialist hands. Additionally, known methodological challenges such as the lack of large-image training datasets,¹⁷ inconsistent reporting metrics, non-standardised imaging or post-processing protocols, and limited graphics processing unit capabilities for exploiting 3-dimensional features can be addressed to improve the performance of future iterations of the PathFinder AI.

Concerns regarding AI-induced over-diagnosis are increasingly relevant in clinical imaging. In the present study, however, false positive rates were low and PPV values were high across most disease categories, indicating that AI-assisted interpretations rarely labeled normal eyes incorrectly. These findings suggest that PathFinder-assisted OCT interpretation favours specificity over sensitivity, with missed pathology rather than over-calling representing the dominant response, and highlighting the need for human oversight. Importantly, this pattern underscores the role of human oversight, as non-retina specialists integrated clinical context, particularly visual acuity, when making referral decisions, rather than relying on AI outputs in isolation. Taken together, this pattern highlights both the strengths and the current boundaries of AI-assisted OCT interpretation in real-world clinical workflows.

This study has certain limitations. It was conducted at a single tertiary center with a limited sample size, which may restrict generalizability. The diagnostic distribution was heavily weighted toward ERM, reflecting real-world OCT practice with consecutive case inclusion; however, the relatively small number of other pathologies, particularly CME and macular hole, limits the precision with which PathFinder’s diagnostic performance for these less frequent conditions can be assessed. The AI model evaluated was proprietary to the Cirrus OCT system and cannot be extrapolated to other devices or image formats. Furthermore, the retina specialists were Furthermore, retina specialists were intentionally masked to clinical data to establish an OCT-based reference standard, whereas non-retina specialists had full access to these details, a design choice that may introduce contextual differences but reflects real-world AI-assisted decision-making. Finally, PathFinder currently identifies predefined macular pathologies and does not yet quantify lesion burden or disease activity, features that would further enhance clinical utility.

Future work should include multicenter studies with larger and more diverse populations, cross-platform validation to test device independence, and longitudinal designs evaluating the impact of AI-assisted triage on patient outcomes. Integrating predictive modules, such as AI-based disease progression forecasting, could further expand its utility from diagnostic support to comprehensive retinal management.

Conclusion

In summary, this study provides early clinical evidence that PathFinder AI-assisted OCT interpretation can enhance the diagnostic and referral capability of non-retina specialists, in selected clinical contexts. By embedding deep-learning algorithms directly within the OCT console, PathFinder bridges the gap between imaging acquisition and interpretation, offering a practical and scalable solution for extending retina-level clinical decision support to general ophthalmic practice. With further validation and improving iterations, such device-integrated AI systems may play a pivotal role in improving access to retinal care, especially in resource-limited settings.