Comparison of Clinician Review Time and Confidence with Optical Coherence Tomography Digital versus Print-Based Workflows: A Pilot Observer Study

Introduction

Intravitreal injection (IVI) is one of the most commonly performed ophthalmologic procedures worldwide and has revolutionized the management of vision-threatening retinal diseases, such as age-related macular degeneration (AMD), diabetic retinopathy (DR), diabetic macular edema (DME), retinal vein occlusion (RVO), and uveitis. The number of IVI treatments administered annually worldwide is estimated to exceed 20 million and is rapidly growing.^1–5 Quality of life and psychometric studies have determined that IVI, while sight-saving, can impose substantial temporal and financial burdens on patients in the form of frequent clinic visits, diagnostic testing, and treatment costs, in addition to anxiety and stress.^5–8

From the healthcare provider’s (HCP) perspective, IVI treatments require significant resources in the form of time and energy.^9,10 Studies to improve clinic efficiency and reduce cost of care have been conducted, such as those exploring the risks and benefits of same day, bilateral IVI and task-shifting of IVI to nurses and nurse practitioners.^11,12 There are few studies that examined methods of streamlining and extracting efficiencies from the diagnostic workflows of IVI management.

The use of optical coherence tomography (OCT) is currently sine qua non in the management of macular diseases.^13,14 OCT is utilized to determine disease activity, identify prognostic biomarkers, gauge response to treatment, and determine optimal dosing schedule.¹⁵ OCT technology has been continuously evolving over the past few decades and has resulted in higher image resolution, faster scanning speeds, reduced imaging artifacts, and ability to identify and measure blood vessel flow.^15,16 In parallel, HCPs are also transitioning from viewing results using analog paper printouts to fully digital formats. While becoming increasingly important in retinal care, OCT machines with advanced image management software (IMS) are not available in all practices especially in resource-challenged settings, and thus clinicians may rely on analog and printed reports. The adoption of digital workflows may facilitate the faster analysis of imaging data, better understanding of treatment responses, and more effective patient education. Advances in digital retinal workflow optimization include the ability to consolidate multimodal imaging data from various input devices, simultaneous viewing of multiple images on a single screen, graphical analysis of structural outcomes, correlation between treatment administration and clinical outcomes, and calculation of treatment duration intervals.

Few studies have assessed the clinical efficiency of digital versus analog workflows in ophthalmology. This pilot study evaluated and compared late software version OCT digital retina workflows with conventional analog/paper printout workflows in terms of time efficiency, reviewer confidence, ease of use, and rater reliability when these workflows were utilized to evaluate eyes for IVI treatment.

Materials and Methods

Study Design

This was an observer based workflow comparison study. Ten retinal specialists reviewed the OCT image sets in print and digital formats obtained from 30 eyes with various retinal pathologies that had previously received IVI. For each image set, the date of OCT imaging and IVI administration, drug type, and adverse events are provided to each reviewer. All images were anonymized to prevent patient identification. Patients have previously signed an informed consent for study purposes. The review and analysis of retrospectively anonymized data were approved by the Peregrine Eye and Laser Institute Institutional Review Board (PELI IRB Protocol No. 2023-03). This study adhered to the tenets of the Declaration of Helsinki.

Image Selection and Review

The cases consisted of a consecutive series of image sets from patients at the Peregrine Eye and Laser Institute in Makati City, Philippines. Inclusion criteria included the following: 1) patients with at least biannual clinic visits with the latest visit between April 1, 2024, to June 30, 2024, 2) diagnosed with posterior segment disease resulting in central foveal fluid accumulation (DME, RVO, nAMD, uveitis or pathologic myopia), 3) underwent at least 2 past treatments with IVI of anti-VEGF and/or corticosteroid injections, 4) underwent at least 3 spectral-domain OCT (SD-OCT) imaging studies with 100 kilohertz, high definition SD-OCT (Cirrus 6000, Carl Zeiss Meditec AG, Jena, Germany) equipped with image data management software (Retina Workplace, Carl Zeiss Meditec AG, Jena, Germany), 5) minimum SD-OCT signal strength of “5” for all visits, 6) OCT imaging data imported into a digital imaging data management software (Retina Workplace, Carl Zeiss Meditec AG, Jena, Germany). The eyes were further grouped by disease chronicity according to the number of clinic visits where SD-OCT scans were performed, as follows: acute eyes, 3–6 clinic visits; intermediate eyes, 7–12 visits; and chronic eyes, 13 or more visits. Eyes were consecutively added to the study until 10 image sets for each of the 3 chronicity groups were selected for a total of 30 eyes.

To minimize hindsight bias, a crossover design was adopted wherein half of the reviewers were randomly assigned to evaluate print image sets first, followed by digital images, while the other half evaluated digital image sets first, followed by printed ones. The same digital and print images were eventually evaluated by all reviewers. The reviewers were assigned to first review either the printed or digital images using an online random number generator. Each reviewer then rendered a simulated treatment decision (treat or not treat) for the most recent image of each set which was meant to reflect actual treatment decisions on real cases.

Analog or Print Workflow

For paper or analog images, an image set for each eye was created by printing OCT reports from each patient visit date. Each printout was anonymized and coded, such that no patient identification details were visible. For each image set, printouts were arranged chronologically, with the earliest image on the first page and the most recent image on the last page. Each OCT report page displayed the following parameters: 1) simultaneous view of fovea-centered scanning laser scanning ophthalmoscope (SLO) image, OCT fundus image, retinal thickness map, layer maps, and OCT image display; 2) fovea-centered ETDRS grid has sectoral retinal thickness values (microns) that are color-coded in comparison to normative data; 3) OCT fundus image; 4) three-dimensional (3D) topographical macular thickness map; 5) segmented ILM map; 6) segmented RPE map; 7) fovea-centered horizontal and vertical cross sectional 6 millimeter B Scan images; and 8) tabulated macular parameters color-coded in comparison to normative data (central subfield thickness in microns, cube volume in cubic millimeters, and cube average thickness in microns) (Figure 1).

Figure 1 Paper printout of optical coherence tomography macular thickness analysis result of anonymized eye with diabetic macular edema which received aflibercept injection on the given exam date.

For visits where IVI medications were administered, the medication given (eg bevacizumab 1.5 mg, ranibizumab 0.5 mg, brolucizumab 6 mg, faricimab 6 mg, triamcinolone acetonide 2 mg, or dexamethasone implant 0.7 mg) were indicated (Figure 1).

The print image sets were presented to the reviewer in random order until all 30 image sets were reviewed, and the reviewer responses were recorded.

Digital Workflow

Digital image navigation and data summary software (Retina Workplace, RWP) (Carl Zeiss Meditec, Jena, Germany) were used to display a set of OCT images from each study eye. The reviewers were given a standard orientation for the basic navigation of the software. The reviewer was masked to patient identities and identification codes. The software was set to the macular thickness analysis display screen, which demonstrated the following: 1) simultaneous view of fovea-centered scanning laser ophthalmoscope image, OCT fundus image, retinal thickness map, layer maps, and OCT image display (top third of screen); 2) 6-millimeter fovea-centered horizontal B-scan images; 3) line graph display of serial central subfield thicknesses; 4) line graph display of serial macular volumes; and 5) IVI medication treatment type and dates. From a temporal standpoint, by default, RWP software displayed images from the earliest visit, penultimate visit, and most recent visits for the above parameters 1 to 2. For parameters 3–5, the RWP software displayed values from the initial visit to the most recent visit when the mouse pointer was hovered over the visit dates, which are indicated by dots on the line graphs (Figure 2).

Figure 2 Digital workflow image of same eye with diabetic macular edema. Top panels feature longitudinal retinal thickness maps that demonstrate disappearance of DME by the last visit (right most images). Middle panels demonstrate serial B Scan images. Lower panels feature a chart view that demonstrates how the average cube thickness (green line) and central subfield thickness (yellow line) respond to intravitreal aflibercept injections (syringe icons) on various treatment dates (dots).

In addition to the above default images, reviewers were given the freedom to select images from any available patient visits using the drag and drop function of the software. The digital image sets were presented to the reviewer in random order until all 30 image sets were completed, and the reviewer responses were duly recorded.

Outcome Measures

The primary outcome measure was the image review time (RT). Secondary outcome measures included self-rated treatment decision confidence, ease of use, and inter- and intrarater reliability.

Time Efficiency

The RT for each image set was measured using a timer. For the print workflow, the RT was measured from the handoff of each eye’s complete image setup to the moment the treatment decision was verbalized. For digital workflows, RT was measured from the moment the screen was navigated to the macular thickness analysis screen page, until the treatment decision was verbalized. As previously indicated, the image sets were classified into three imaging frequency groups (acute, intermediate, and chronic) to determine whether RT was influenced by disease chronicity. The mean print and digital RT for the entire dataset and each chronicity group were determined.

Ease of Use and Reviewer Confidence

After each image set was reviewed and treatment decision was verbalized and recorded, the reviewer provided a self-rated treatment decision confidence level on a scale from zero, indicating no confidence, to ten, representing absolute confidence. Once all 30 images in each workflow were completed, the reviewer provided an ease-of-use rating for the paper and digital workflows on a scale from zero, indicating extremely easy to use, to ten, indicating extremely difficult to use.

Reliability

Inter-rater agreement using the Fleiss kappa coefficient and intra-rater reliability using Cohen’s k coefficient among reviewers were determined for each workflow to determine differences in treatment decision-making, in addition to the percent agreement rate.

Statistical Analysis

A power test was conducted to ensure validity of the primary objective of statistical testing. Data were collated as percentages, means, and standard deviations (SD). Normality of data distribution was evaluated using the Shapiro–Wilk W-test, which was performed for print and digital review time data. The two workflows were compared using the t-test for continuous data and Fisher’s exact test for categorical data (QuickCalcs, GraphPad Software 2024). A post hoc analysis using the Z-test was performed to assess the differences in inter-rater reliability between the two workflows. The level of significance was set at p < 0.05.

The power test with parameter alpha = 0.05, the mean review time of digital and print, and the difference in standard deviation resulted in a power of 0.8723, which indicates that the hypothesis test/paired means test conducted on digital and print review times is sufficiently powered for statistical relevance.

Results

The study involved ten retina specialists who served as image raters. Table 1 summarizes the raters’ profiles. The mean age was 39.1 ± 7.9 years old, with mean practice duration of 6.2 ± 7.4 years. Half (50%) primarily used print workflows for OCT review in clinical practice, whereas the other half (50%) mostly utilized digital workflows. The majority (90%) used a Zeiss HD-OCT machine in their practice. Only 50% of the specialists had prior experience with the RWP, with an average usage duration of 1.4 ± 2.2 years. Prior to the workflow comparison, all reviewers were oriented to the Zeiss RWP system and were given as much time as desired to practice navigation within the RWP environment (Table 1).

Table 1 Demographic and Practice Profile of Raters

Sixty OCT image sets from 30 eyes (30 printed and 30 digital) of 28 patients were reviewed. Of the study eyes, the mean patient age was 73.2 ± 13.0 years (range, 51 to 98). Disease etiology included neovascular age-related macular degeneration (70%), diabetic macular edema (20%), retinal vein occlusion (7%), and uveitic choroidal neovascularization (3%). IVI comprised anti-VEGF medications in 90% of eyes and dexamethasone implants in 10% of eyes.

The mean print workflow RT for each image set was 55.6 ± 37.2 seconds while the mean digital workflow RT for each image set was 28.6 ± 16.5 (p = 0.007). Using a digital workflow reduced the overall mean RT by 48.5%.

Grouped according to chronicity, the mean print and digital workflow RT for acute eyes (n = 10) were: 36.1 ± 26.9 and 23.4 ± 20.1 seconds, respectively (p = 0.2472); for intermediate eyes (n = 10), 50.4 ± 37.5 and 30.1 ± 29.2 seconds, respectively (p = 0.1935); and, for chronic eyes (n = 10), 80.2 ± 69.4 and 32.4 ± 21.3 (p = 0.0619). Digital workflow reduced the mean RT by 35.2%, 40.2%, and 59.6% for acute, intermediate, and chronic eyes, respectively.

The mean reviewer-rated ease-of-use scores for print and digital workflows were 6.3 ± 1.7 and 9.5 ± 0.5, respectively (p = 0.0008). The mean self-rated treatment-decision confidence levels using print and digital workflows were 8.0 ± 0.9 and 8.5 ± 0.8, respectively (p = 0.0015) (Table 2).

Table 2 Comparison of Image Review Time, Self-Rated Treatment Decision Confidence Level and Ease of Use Using Print and Digital Workflow

The mean inter-rater percentage agreement rates for treatment decisions of print and digital workflows were similar at 81 ± 17% and 79 ± 15%, respectively (p = 0.998). The Fleiss’ kappa values were 0.412 (95% CI: 0.36–0.47; SE = 0.027) for the print workflow, indicating moderate agreement, and 0.332 (95% CI: 0.28–0.39; SE = 0.027) for the digital workflow, indicating fair agreement. The Z-test showed that the Fleiss’ kappa scores for the tests were not significantly different (Z-score = 0.175; p = 0.861). In addition, the mean intra-rater treatment-decision reliability rate was 71.0 ± 6.1% (range, 63.3 to 80.0) while the computed Cohen’s kappa was 0.368 (95% CI: 0.259–0.519), indicating fair agreement.

Discussion

OCT imaging plays a crucial role in the management of common retinal diseases by providing quantitative, high-resolution, cross-sectional images of the retina that aid in determining disease activity as well as treatment response.¹⁷ Imaging data are essential for determining the need to start IVI therapy, monitoring treatment response, and adjusting dosing regimen.¹⁸ The alignment of OCT imaging with IVI treatment protocols has greatly improved patient outcomes by enabling personalized, data-informed disease management.^19,20

The trend towards increased dependence on OCT imaging has resulted in a substantial imposition of temporal and physical burdens on physicians reviewing large volumes of OCT scans. The interpretation of these images requires considerable expertise, time, and attention. In a busy clinical setting, ophthalmologists often face review overload, which can lead to diagnostic lapses, mental fatigue, and an increased risk of management oversight. The integration of electronic and artificial intelligence (AI)-based tools for automated OCT analysis has shown promise in helping physicians reduce their workload while improving their diagnostic accuracy.^21–24

Traditionally, diagnostic workflows involving image acquisition, review, and interpretation rely on printed hard copies that are then physically transmitted to the clinician for review, whereupon an action plan is developed.^25,26 This process can be cumbersome and may result in treatment delays and decreased productivity. Digital workflows involving review software aided by AI algorithms have great potential to augment image review processes and have been incorporated into several image review software (ie Heidelberg HEYEX2, Nidek NAVIS-EX, Topcon ImageNet6, Zeiss Retina Workplace). In previous studies comparing digital and print workflows for cataract surgery, specific steps of interest (time for preoperative assessments, IOL power calculation, IOL axis marking, and alignment) were reported to be significantly shorter using the digital workflow.^27–31

A pilot study was conducted to determine whether a research design is practical or workable on a larger scale, to identify flaws or inefficiencies in the study protocol, and to estimate the resources for a larger research program. The methods in this study clearly demonstrated that using a digital workflow was more time efficient than using a traditional print or analog workflow when reviewing OCT images, with a trend towards greater RT reduction with increasing disease chronicity. Image reviewers also reported higher confidence levels in therapeutic decision making and increased ease of use when using a digital workflow. The therapeutic decision agreement rates among the reviewers appeared to be similar using either workflow. Finally, there was fair agreement in the therapeutic decisions of each reviewer using either workflow.

In this retinal workflow study, we used direct observation and semi-structured interview approaches to determine RT.³² The actual time savings generated by using a digital workflow are likely to be higher because the study did not account for necessary print workflow-related activities such as printing and collating hard copies. Navigation of digital workflows is faster, as the rater needs only drag and drop the image from the data of interest available on the software platform. By contrast, a rater using printed images must physically shuffle pages to visualize the results from different visit dates.

Digital workflows have several advantages that may augment the quality and efficiency of the image review process, including higher-resolution images, an overview of sequential images, and reviewer-controlled image enhancements (eg magnification, contrast), and anatomical parameter summary (for example, mean central subfield thickness, macular volume), image overlays, annotation of clinical findings and notes, incorporation of treatment and medical events with structural outcomes, promotion of patient education, and facilitation of scan selection by the reviewers. These potential advantages require validation through qualitative and quantitative studies.

Currently, most clinical practices indicate an increasing need for access to medical information to guide management decisions. This has resulted in the continuous development of ophthalmology information systems (OIS) and picture-archiving and communication systems (PACS). An OIS allows different actors to input medical information into a common platform (eg refraction, visual acuity, intraocular pressure) while the PACS allows multimodal images to be visualized by ophthalmologists and reading center technicians.^25,26,33 As PACS evolve, new functionalities are built in to enable and facilitate actions that enhance information access and communications. Digital workflow systems have been reported to enable more efficient intraocular lens selection, surgical planning, and implantation in the cataract space. The paucity of studies examining the impact of PACS on retinal practice workflows has prompted this initial study.

The adoption of health information technologies (HIT) has been instrumental in improving HCP productivity, decreasing treatment delays, reducing errors, facilitating health-related financial transactions, and providing large datasets for health research. Data or image management software (DIMS), which allows images from various capture devices (eg fundus camera, OCT machine, and automated perimetry) to be consolidated into one viewing platform, has streamlined the review procedure for ophthalmologists. The current DIMS not only allows for the presentation of data but also incorporates clinical events (eg IVI, laser treatment, and medical adverse events) and displays their temporal relationships. Aside from this, DIMS allows for remote access and virtual monitoring of macular diseases via telemedicine.³⁴ Digitalization also lays the groundwork for artificial intelligence to analyze, interpret, and recommend clinical management strategies in a drive towards personalized medicine. Our results suggest that practices with medium-to-high volumes of OCT imaging and IVI may benefit from DIMS integration.

While DIMS was introduced a decade ago, many institutions still use rudimentary versions which require physicians to either review images on the OCT machine screen or manually select images for viewing on another computer screen. The latest IMS versions can automatically select and present curated or physician-defined imaging data and may further enhance practice efficiency but may face regulatory or budgetary barriers for adoption. The results of this study may aid in overcoming these obstacles for late version software implementation.

Our study limitations include the small sample size of the image sets and reviewers and unequal digital workflow experience of the reviewers. Additionally, our study used OCT images alone and was restricted to a specific, though widely available, software platform, which may have limited the generalizability of our findings. The outcome measures mainly involved RT and did not evaluate the causes of variability in diagnostic agreement rates. The study likewise looked into workflow efficiency and did not pursue whether treatment decisions against a particular gold standard. Future studies should include a more detailed time analysis of different workflow steps to fully elucidate the time-efficiency impact of digital systems. The cost-effectiveness of transitioning to digital workflow may be another area of investigation. Future studies should incorporate clearer OCT grading guidelines to increase reviewer agreement rates and consider adding a gold standard to provide additional clinical value to the results.

This pilot study demonstrated only fair to moderate agreement in treatment decision-making among the different raters, despite the uniformity of the review environment. This study included 10 raters with varying experiences in reviewing images from eyes with disparate diseases. The achieved suboptimal concordance identifies potential factors that can influence image interpretation and decision-making, including rater experience with the digital system, rater experience in managing retinal diseases, and wide variance in disease pathology. Learning from this research, a more expansive study is being designed to include raters with similar extensive digital workflow experience and to limit it to a single disease entity (for example, diabetic macular edema) to better elucidate the value of the digital workflow.

In summary, our small pilot observational study appears to support that digital workflows appear to facilitate OCT image review, enhance time efficiency, and improve physician confidence among retinal specialists evaluating patients requiring IVI medications. In an appropriate setting, the adoption of digital workflows can be considered to enhance practical productivity and patient experience. Future studies involving more clinicians, a larger number of cases, and with additional cost-analysis and efficiency metrics are encouraged.