Comparative Validation of PlusoptiX and AI-Optic Photoscreeners in Children with High Amblyopia Risk Factor Prevalence

Introduction

Photoscreening is a valid method for detection of amblyopia risk factors with a limited endorsement for children older than 3 years,¹ but capable for toddlers and infants even younger than 1 year.² Photoscreening is designed to identify the refractive amblyopia risk factors anisometropia, hyperopia, astigmatism and also myopia.³ In addition, photoscreeners can also target strabismus and deprivational amblyopia risk factors. The Vision Screening Committee of the American Association for Pediatric Ophthalmology and Strabismus (AAPOS) has published confirmatory exam failure levels for amblyopia risk factors starting in 2003³ and updated with three age ranges in 2013⁴ with even further refinement in 2021.⁵

A given photoscreener manufacturer or vision screening clinic needs to select and set instrument referral criteria so the device can target AAPOS uniform guidelines risk factor levels with appropriate sensitivity and specificity. Some devices employ a single pre-selected instrument referral criteria (IRC) which, when selected to be sensitive for manufacturer liability often results in reduced specificity- and over-referral from the viewpoint of the receiving ophthalmologist. To help address the disparate screening program goals, Plusoptix (Nuremberg, Germany) has adopted a user-selectable group of published instrument referral criteria options targeting the 2013 AAPOS guidelines.

The Plusoptix family of photoscreeners (S04, s08, s09, s-12, s-16) has had extensive validation compared with eye chart visual acuity,^6,7 digital flash cameras,^7–9 the MTI photoscreener,^9,10 the iScreen,^9,11–13 Suresight,^9,14–16 compared with SPOT,^{12,13,17–21} 2WIN,^17,21–26 GoCheck Kids,¹³ blinq,^25,26 with Retinomax^{9,23,24,27,28} and table-top autorefractors,^22,29 community screening^30,31 and examination.^28,32–44 The refractive accuracy has been compared.^{18,22,28,43,45} PlusoptiX is useful on special-needs children.⁴⁶

Various alternative IRC have been compared.³⁹ Evidence-based instrument referral guidelines with sensitive and specific alternatives for the three leading infrared photoscreeners PlusoptiX, SPOT (Welch-Allyn, Schenectady, New York) and the Adaptica 2WIN (Padova, Italy) targeting the 2021 revision in AAPOS guidelines⁵ have been published.²¹

Cost has been a barrier in the adoption of photoscreening for some pediatric clinics. In Spring 2022, a new infrared photoscreener with centralized, machine-learning-based interpretation was marketed called the AI-Optic Vision Screener (Beijing Fantasy Forest Intelligent Technology Company, Ltd, Beijing, China, www.ai-optic.com). The AI Optic photoscreener was priced below the cost of several existing photoscreening devices. The Alaska Blind Child Discovery (ABCD) promptly obtained a new device. The purpose of this paper is to describe the features of the new AI Optic device and to subject it to rigorous head-to-head validation compared to a conventional PlusoptiX a-12 in a pediatric ophthalmology practice. Practical use of the new device is described in detail. The AI Optic and the PlusoptiX a-12 are shown in Figure 1.

Figure 1 The AI Optic (left) and the PlusoptiX a-12 photoscreeners.

Materials and Methods

The prospective evaluation of a screening test as a part of the Alaska Blind Child Discovery was approved by the institutional review of Providence Alaska Medical Center in compliance with the Declaration of Helsinki and the Health Insurance Portability and Accountability Act. Parents provided written informed consent and older children provided assent. De-identified data are provided: https://www.abcd-vision.org/references/AI%20Optics%20PlusoptiX%20de-ID%202022.pdf.

Released in 2022, the more recent device is called the AI Optic Vision Screener (version 0.8.3). AI Optics is a company since 2020 with address 145 Lexington Ave APT 10 New York, NY, 10016–8365 United States. From the product description, it is a multi-radial, Infrared (850 nm) photoscreener measuring at a distance of 100 ± 5 cm through pupils with diameter 4.0–9.0 mm, covering an interpupillary distance 50–70 mm over refractive range spherical power −8D to +8D and cylindrical power −3.50D to +1.0D. The intended age range is 3–50 years. The AI Optic offers no ability to select + or minus cylinder notation, but the output is mainly minus cylinder, but sometimes presents without the (-). Historical data stored in device can be individually or batch emailed but may, at times, not transmit with error message “Failed to login!”

AI Optic allows user selection of three instrument referral criteria (IRC): the pre-determined default is (1) “Arnold, (sensitivity 92% specificity 90% Indication of source: www.abcd-vision.org, Arnold is best at maximizing sensitivity and specificity for the latest AAPOS referral criteria.¹² You can choose Arnold for routine check”. Other IRC are (2³⁴) and (3³²). The actual anisometropia/hyperopia/cylinder/myopia cut-offs for school-age for AI Optic IRC option (1) are 1.25/2.00/1.50/1.50, for option (2) are 1.00/3.50/1.25/3.00 and for IRC option (3) 1.25/1.00/1.25/1.00.

Inclusion criteria were children undergoing pediatric eye examination with favorable parental or legal guardian consent. Excluded were children either too young or not able to gaze at the photoscreeners. Each patient had an eye exam including best corrected visual acuity (for age), ocular motility including alternating prism cover test, stereopsis, anterior and posterior segment exams. Refraction was done with accommodation relaxed with cyclopentolate 1% 30 minutes before refraction in children under age 8 and with fogging at the phoropter for those older than 8. From the exam, refractive, strabismic and deprivational amblyopia risk factors meeting AAPOS criteria were identified.

SAMPLE SIZE CALCULATION: Prior experience with infrared photoscreener refraction comparison demonstrated the ABCD ellipsoid mean about 2.0 with standard deviation 2.0.²¹ Therefore, to detect a difference of 0.5, with beta 0.8 and alpha 0.05, a sample size of 126 would be needed. If 2/3 of the patients were predicted older than 4 years, then about 180 patients should be recruited expecting some to be inconclusive or out-of-range.

The ability of each device to screen for given risk factor targets for amblyopia are compared using receiver operating characteristic curves⁴⁷ from which an area under the curve (AUC) was calculated.

From a range of individual subcomponent refractive and strabismic amblyopia instrument referral criteria that correlated between each device and the confirmatory eye examination, receiver operating characteristic curves were generated comparing the devices targeting 2013 and 2021 AAPOS Uniform guideline with failure criteria that recognize anisometropia, meridional hyperopia, astigmatism, myopia and strabismus. To a set of IRC called “regular”, “sensitive” and “specific” and extra level for “extra specific” was added to extend observed range on the ROC curves (Table 1).

Table 1 Instrument Referral Criteria (IRC) for PlusoptiX a-12 and AI Optic Photoscreeners Targeting 2013 and 2021 AAPOS Uniform Guideline Refractive Amblyopia Risk Factors (ARF). Cut Off Ranges Include Anisometropia (A), Meridional Hyperopia (H), Myopia (M) and Astigmatism (C; cylinder)

The ability of each device to estimate age-based sphero-cylinder refractive error is compared using the ABCD ellipsoid method.²⁴ The ABCD ellipsoid is derived as an oval-solid region on a 3 dimensional plot with J0 (x-axis), J45 (y-axis) and M spherical equivalent (z-axis) the displacement between two sphero-cylinder refractions leading to a single measure related to the degree of blur produced by the refractive difference.⁴⁸

Results

PATIENTS: A total of 200 children aged 0.4 to 18 (mean 7.6 ±4.7 years) undergoing new or follow up pediatric ophthalmology examinations were concomitantly screened with the AI Optic and the PlusoptiX a12 devices (Table 2). 113 had age-based 2021 AAPOS refractive and/or strabismic amblyopia risk factors. The ethnic/racial/gender mix was 14 Asian, 18 Black, 24 Hispanic, 16 Alaska Native, 15 Pacific Islander and 113 White; 106 were male. Seventy-one had some form of neuro-developmental delay including 16 with autism, 11 premature, 7 with attention deficit (ADHD) and 6 with trisomy 21 and of whom cooperation was judged as poor in 9 and “fair” in 12. Sixteen were referred by community pediatric photoscreening and 15 had prior strabismus surgery.

Table 2 Characteristics of Patients. Counts and Measured Variables are Quantified by Mean, Standard Deviation (S.D), Median, Minimum (min) and Maximum (max). There were 71 with Developmental Delay such as Autism, Trisomy 21, Marked Prematurity and Attention Deficit. Actual Refraction is Compared to Estimates by the PlusoptiX a-12 and AI Optic Photoscreeners Featuring Sphere Right and left (R and L) and Cylinder (Cyl)

PRACTICAL EXPERIENCE WITH THE DEVICES: The following are observations regarding advantages and disadvantages of each device when screening cooperative and developmentally delayed children. The AI Optic in 2022 US dollars costs less than $1000 while the other traditional infrared, multi-radial photoscreeners cost $6-8000.

The AI Optic requires WiFi for remote, automatic intelligence interpretation (takes 10–70 seconds) and charges in about 2 hours with USB-C affording 2 hours of cordless use for over 10–20 screenings. The AI Optic comes with a 13-page brief Product Instruction and an online training video which recommend fixing device to a tripod instead of hand holding. Presumably the tripod would be for adult and older-children refractions whereas actual pediatric photoscreening benefits from quick, creative camera focus and re-alignment. Therefore, for our validation in children, we used the device hand-held.

The display on this early release of AI Optics still has some English typos such as: “proceesing”, “enter your email to see you result” The instrument referral criteria we employed on the AI Optic matched one available on the PlusoptiX and was called “Arnold.”¹²

The PlusoptiX has extensive clinical experience with product and software updates.⁴⁹ The replaceable AA batteries or rechargeable option are helpful. The user interface selecting patient age only affords quick screening and the internal computer interpretation is usually very fast (2–5 seconds image acquisition and 3 seconds interpretation). Focus is indicated by a green ring around the pupils. For children with complex refractions, blocked or non-round pupils, the PlusoptiX will not “guess” a quick interpretation but might take 30 seconds or more attempting to acquire reliable infrared pupil crescent information. If PlusoptiX interprets refractions out of range, instead of a presenting a sphero-cylinder refraction estimate, it will instead state “HYP” for excess hyperopia or hyperopic astigmatism and “MYO” for excess myopic refraction estimation. For age-dependent target conditions, Plusoptix uses a simple touch screen choice of years of age before enabling image acquisition.

The AI Optic has a brief user’s manual and a 13-minute online video so certain aspects of screening with this device were intuitive, but others required discovery with use. For instance, patient demographics name, gender and birthdate must be entered first, then image acquisition triggered by pressing a whirly-circle emblem on the touch screen; in a dim room the patient’s face and pupils were not visible until a second emblem- not specifically mentioned on the user manual, was activated initially. Focus is based on image clarity on the screen and then “Image proceesing” shows as the number of seconds for WiFi and internet transmission to the central reading center followed by return of a “refer” or “pass” color-coded interpretation with sphero-cylinder estimate mainly in minus cylinder form (however range of cylinder for the device listed as “from −3 to 1 diopter). Details about AI Optic proprietary central reading process, such as location, data storage and/or computer methods are not given in the accompanying printed product instructions or on the website. Almost always the left eye spherical equivalent is 0.75 to 1.0 diopters more hyperopic than the right eye. Stored screening results can be reviewed, selected individually or as a group and emailed however the device was able to email the initial 50 screenings, but gave an error when attempting to email as the group got larger. The ability to aim the camera and hold it steady on a stable, or a moving patient worked well hand-held by an experienced screener and the tripod was not needed.

The PlusoptiX A-12 (version 7.2.5.0, Nuremberg, Germany) originally designed in 2012 is an industry-standard, multi-radial infrared photorefractive screener with extensive validation. Plusoptix can work independently from internet with digital interpretation real-time through the device; Plusoptix does allow WiFi connectivity to electronic medical records, however. An S12 is for screening while the A-12 provides full sphero-cylinder refraction with similar red reflex analysis from the prior S04, S08 and S09⁴⁶ models. These PlusoptiX devices use standard or rechargeable AA batteries. We employed IRC rating number 4 out of five options for specificity¹² (anisometropia 1.25, hyperopia 2.00, cylinder 1.50 and myopia 1.50) when this device was employed hand-held in a dim exam room. A range of instrument referral criteria previously reported for PlusoptiX²¹ are employed and new regular, sensitive and specific and extra-specific alternatives are generated from adjusted ARF component contributions to preliminary AUC, particularly for the recently released AI Optics device are given in Table 1.

VALIDATION: The PlusoptiX was unable to give an interpretation in 25 cases usually due to strabismus and/or poor cooperation, in 9 excess hyperopia was noted in both eyes, in 3 excess myopia was noted and in 19 an excess of hyperopia was noted in one eye with sphero-cylinder estimate in the contralateral eye. For the AI Optic, 16 were unreadable yielding no refractive results of which two were patients with strabismus and two others had high hyperopia (5 diopters), but sphero-cylinder estimates were obtained for the remaining 184 whether strabismic or not. Of the 25 inconclusive interpretations with Plusoptix, 22 (88%) had 2021 AAPOS ARF while of the 16 inconclusive AI Optic, 10 (63%) had 2021 AAPOS ARF.

Receiver operating characteristic (ROC) curves demonstrate the predictive value targeting amblyopia risk factors comparing the two devices. AAPOS Uniform Guideline 2013 older age group (>47 months; Figure 2) and the more recent 2021 guidelines for preschool (<4 years) and also for children ≥ 4 years are presented (Figure 3). The actual “refer” or “pass” results from the PlusoptiX using device selected IRC option 4 of 5, and the Arnold default IRC for AI Optics are highlighted on each ROC curve with a green arrow (Figures 2 and 3). Estimates of the Area Under the Curve for ROC curves comparing devices in this high-prevalence cohort for school age 2013 and 2021 guidelines and for preschool 2021 are given in Table 3. For children over 4 years of age targeting 2021 refractive plus strabismic amblyopia risk factors, the AI Optics had AUC of 0.58 compared to 0.74 for PlusoptiX a-12. The corresponding instrument results between AI Optics and the PlusoptiX a-12 photoscreener influenced by developmental delay is shown in Table 4. For school age 2021 refractive plus strabismus, the Plusoptix “regular” IRC had sensitivity/specificity of 92%/82% for patients with developmental delay and 73%/81% for patients with normal development. For the same patients, targeting strabismus plus refractive 2021, AI Optic had sensitivity/specificity of 69%/28% for patients with developmental delay and 74%/48% for patients with normal neuro-development.

Table 3 Area Under the Curve (AUC) for Receiver Operating Characteristic (ROC) Curves Validating the PlusoptiX a-12 and the AI Optic Photoscreeners Targeting American Association of Pediatric Ophthalmology and Strabismus (AAPOS) Refractive Alone and Refractive Plus Strabismic Amblyopia Risk Factors (ARF). The total Number of Children for Each Age Range (n) and The Prevalence of Those Above the Examination Failure Levels are Given. Validation Sensitivity (sens) and Specificity (spec) from the ABCD Criteria (Arnold) for School age Targeting Refractive ± Strabismic Amblyopia Risk Factors are Given. Inconclusive Device Results (i) are Reported as if They Were a “Refer (i = r), and also as if They Are Not Included in the Calculations (Exclude i)

Table 4 Corresponding Results Between AI Optic and PlusoptiX a-12 Photoscreening Accounting for the Number (in Parentheses) with Developmental Delays

Figure 2 The receiver operating characteristic (ROC) curves comparing AI Optic to the Plusoptix a-12 in children over 4 years of age targeting the 2013 AAPOS amblyopia risk factors for strabismus and refractive error with inconclusive results counted as a “refer” and Instrument referral criteria “Arnold, ABCD” chosen indicated by the green arrows.

Figure 3 Receiver operating characteristic (ROC) curves targeting AAPOS 2021 refractive and strabismic amblyopia risk factors comparing school age (left column) with preschool (right column). Inconclusive results (i) are either counted as a “refer” (I = refer) or excluded from calculations of sensitivity and specificity. Instrument referral criteria “Arnold, ABCD” chosen indicated by the green arrows.

REFRACTIVE COMPARISON: The accuracy of each device’s photoscreen estimation of refraction was compared to actual refraction using the ABCD ellipsoid method.²⁴ Of the 200 patients, 134 had sphero-cylinder estimates in both AI Optics and PlusoptiX for comparison. ABCD ellipsoid mean (±S.D) for PlusoptiX right eye 2.20±1.5 and left eye 2.34±1.6 differed from AI Optics right eye (3.15±2.3) and left eye (3.63±2.6) using Student’s t-test with right eye (t(266) = 4.0 and left eye t(266) = 4.9 both being significant (p<0.001). However, the Shapiro–Wilk test for normal distribution for ABCD ellipse right eye (z=6.0) and ellipse left eye (z=6.8) PlusoptiX and for IA Optics right eye (z=6.3) and left eye (z=4.9) all were not normally distributed (p<0.001). Therefore, the ABCD ellipsoid median with inter-quartiles (Q1,Q3) for PlusoptiX right eye 1.88 (1.12,2.86) and left eye 2.10 (1.26,3.04) differed from AI Optics right eye 2.53 (1.54,4.01) and left eye 3.05 (1.83,5.00) demonstrated by Mann–Whitney z=3.7 right eye and z=4.2 left eye (Figure 4; p<0.001 each).

Figure 4 The ABCD ellipsoid compares spherical cylindrical refractions from Plusoptix and AI Optic with actual (cycloplegic) refraction. Top metric box plots with interquartile ranges (yellow and purple) highlights values of 1 correspond to one line visual acuity blur (grade A) while value of 2 implies 3 logMAR lines blur (grade B) and value of 3 6 logMAR lines blur (grade C). Bottom metric shows proportion with each grade (colors purple, Orange and green) blur and those that are a poorer spectacle match than grade C (teal color) (more than 6 lines visual acuity blur).

The ABCD ellipsoid for PlusoptiX did not correlate with AI Optics for right eye (Pearson Product-Moment Correlation r(132) = 0.107, p=0.22) or for left eye (Pearson Correlation r(132) = 0.07, p=0.42).

Discussion

The 2022 released, less-expensive AI Optic was compared to the similar shaped and configured 2012 released PlusoptiX a-12 photoscreener on 200 children from a pediatric eye practice. Since the 2021 guidelines have only recently been published, each device still utilized Instrument Referral Criteria that target 2013 AAPOS guidelines.⁴ Approximately 1/3 of the younger children less than 4 years old and about 2/3 of the older children had amblyopia risk factors defined by the 2021 AAPOS guidelines.⁵ For the 2013 and 2021 Guidelines, the extensively utilized PlusoptiX a-12 outperformed the newer AI Optic in terms of amblyopia risk factor screening and refractive estimation.

A strength of this study is a sufficient number of ethnically-diverse, older children with high prevalence of amblyopia risk factors. On the other hand, subgroup analysis of the younger group less than 4 years of age was limited by sample size of just 52. Developmentally delayed children were not excluded. Uniform guideline recent target amblyopia risk factors were analyzed in addition to the former 2013 exam failure levels for which each device has software to target. Refractive comparison utilized validated, single metric ABCD ellipsoid values and grades with sufficient sample size.

PlusoptiX had more inconclusive results than AI Optic. A refractive estimate was delivered in 175 of the PlusoptiX screenings and 184 of the AI Optic screenings in this high risk cohort with 32 patients with a tropia of greater than 8 prism diopters. Due to variations in pupillary red reflex crescent during ocular misalignment, the Plusoptix with real-time, in-device interpretation is programmed to not give a refractive estimate when substantial strabismus is suspected. On the other hand, AI Optic with remotely AI-interpreted images produced refractive estimates sometimes in the presence of constant strabismus. Eighteen inconclusive Plusoptix interpretations were regarded as a refer by AI Optic while 5 patients were inconclusive on both devices (Table 4).

A weakness of the study is that it is not performed in community cohort with typical risk factor prevalence then combined with uniform exam definition of refractive error, strabismus, media opacity and age-appropriate visual acuity from which amblyopia itself can be diagnosed. Instead of highly experienced screener as in this study, the experience level of the screener could have been more typical such as studying a lay screener or pediatric technician or nurse. A thorough cost-benefit analysis was not done to compare the devices; instead price range and validation data are presented.

Plusoptix had been the only photoscreener to offer a user a choice of five instrument referral criteria³⁹ but now AI Optics has been designed to offer three of these also.

With further clinical experience and validation, the accuracy of this new AI Optics with artificial intelligent interpretation might have the potential to get even better.

Conclusion

The inexpensive AI Optic resembles the Plusoptix in form and configuration including user-selected instrument referral guidelines, however it does not yet achieve Plusoptix’s high level of performance targeting uniform amblyopia risk factor guidelines or estimates of sphero-cylinder refractive error. With further validation and AI, the performance of the new AI Optic might be increased.