Validation of a Modified National Eye Institute Grading Scale for Corneal Fluorescein Staining

Introduction

Application of vital stains or dyes to the corneal and conjunctival tissues are commonly used to diagnostically evaluate the integrity of the ocular surface. Sodium fluorescein, lissamine green, and rose bengal are commonly used stains for differentially diagnosing ocular surface conditions including dry eye disease, meibomian gland dysfunction, and other conditions that may impact normal tear film functioning (eg, blepharitis, Sjögren’s syndrome) or mechanical positioning of the eyelids and lashes (eg, trichiasis, entropion).^1–8 With rose bengal being associated with patient discomfort, it has lost favor in recent years since it yields staining characteristics similar to lissamine green.^6,7 Sodium fluorescein is known to provide better corneal lesion visualization than lissamine green; thus, sodium fluorescein is more commonly used for evaluating corneal disease than lissamine green.

Evaluation of the presence, severity, and pattern of corneal and/or conjunctival staining following application of sodium fluorescein or lissamine green to the cornea may be evaluated as part of clinical research in addition to their use as a diagnostic clinical tool. Monitoring changes in the severity of ocular surface staining in response to interventional treatments is often used as a common endpoint in clinical research for conditions such as dry eye disease and meibomian gland dysfunction. Depending on the design of the study, ocular surface staining may be used as a primary endpoint in trials designed to assess the efficacy of new drugs or devices.^1,8–12

Grading scales designed to provide a standardized method to quantify the nature and severity of ocular surface staining have been developed and are widely used in clinical research. Despite extensive use of grading scales for corneal and conjunctival staining in the diagnosis and evaluation of patients with ocular surface diseases, no single grading scale for staining has been universally adopted. Commonly used grading scales for corneal fluorescein staining include the National Eye Institute (NEI)/Industry Workshop grading scale, the Oxford scale, and the van Bijsterveld scale. These scales and a range of others have been developed, and modified in some cases, to assess the severity of ocular surface staining with respect to staining intensity, density, and characteristic features, including filaments, patches, confluence, and coalescence.^{1,9,10,13–15}

A modified version of the original NEI staining scale (0 to 3 integer scale for each region of the cornea), presented in 1995 as part of the NEI Industry Workshop, has been developed by Lexitas Pharma Services Inc. (Durham, NC, USA) for use in clinical research in response to a need for more precise assessment of greater degrees of corneal fluorescein staining. This report presents the results of a validation process evaluating the reliability and repeatability of the Lexitas modified NEI staining scale in comparison to the original NEI staining scale based on the density of staining and a structured version of the NEI staining scale that quantifies punctate staining.¹⁴

Methods

Study Images

This validation study developed a series of images for evaluation and quantification of the staining scales. No patient medical records or other information was used, and no contact with patients occurred during the study. No interventional treatments were involved in this study. All images used in the validation process were either medical illustrations or images selected from a proprietary database of photographs that did not contain any identifying information. No internal review board or ethics committee review approval was sought given the nature of the source data. The images used in the validation process included a series of illustrations and clinical photographs depicting corneal fluorescein staining. The illustrations depicting corneal staining were prepared using Adobe Illustrator software, through version 26.4 (Adobe, San Jose, CA, USA). Illustrations were designed to display a left eye, with a grid overlay to divide the cornea into 5 regions: central, inferior, nasal, temporal, and superior. The five-region grid was overlayed onto the cornea with the central region of the grid representing one third the size of the corneal diameter. The width of the surrounding quadrants was likewise one third the diameter of the cornea. This grid system is consistent with the three grading scales being evaluated in this study. A full description of the grading scales can be found in Figures 1 and Figure 2. Each illustration displayed varying degrees of fluorescein staining in one or more regions of the cornea. A total of 18 illustrations were used in the study, with 5 regions scored for each image. A series of 14 clinical photographs of corneal fluorescein staining were also included in the series of images to be graded. The clinical photographs of staining were selected from an internal company library of images used for training. The illustrations were designed and clinical photographs were chosen to provide a full range of staining severity across the grading scales used in the study. Like the illustrations, a grid was overlayed onto the cornea, creating 5 regions to be graded. All regions were graded, when possible; however, a portion of the superior region of the cornea was obscured by the upper eyelid in 11 of the 14 photographs. The superior region was not graded in those images.

Figure 1 Corneal fluorescein staining scales used in the validation of the Lexitas modified NEI scale. Descriptions for each grade of the NEI scale (density), NEI scale (dot count) and the Lexitas modified NEI scale are presented. *Confluence defined as closely adjacent multiple dots, but not merging into a solid area of staining. **Coalescence defined as merging of dots into a solid area of staining. Abbreviations: NEI, National Eye Institute; TNT, too numerous to count.

Figure 2 Reference illustrations provided to the independent examiners as examples for each grade level of the (A) original NEI density, (B) structured NEI dot count, (C) Lexitas modified NEI (score range 0.0–2.0), and (D) Lexitas modified NEI (score 2.0–4.0) staining scales.

The images were grouped as illustrations or photographs, with each being displayed in a random order within the group with a unique identifier that was not provided to the examiners. Display of the images for grading was standardized using identical computer monitors with no modifications to the factory settings at each site (Samsung 24” QHD; Samsung Electronics, Suwon-si, South Korea).

Examiners

The study images were evaluated independently by 3 examiners. The examiners were all clinical practitioners with extensive experience with grading corneal fluorescein staining using multiple established grading scales. Each examiner was trained on the study grading scales by providing them with a detailed description of the scales and meeting with them prior to the grading session to review any questions. The graders were also required to complete a pilot run of scoring the staining severity of 7 images (4 illustrations and 3 clinical photographs) prior to the full validation assessments.

Grading of the Images

The corneal fluorescein staining on each image in the pilot series and validation sets was graded based on 3 different grading scales: the original NEI staining scale (density-based scoring),¹⁴ a structured version of the NEI staining scale (dot-count scoring) that has been used in clinical research (unpublished data), and the more categorized Lexitas modified NEI staining scale. Images were viewed via a PowerPoint slide deck (Microsoft, Redmond, WA, USA), and all images were graded with one scale before proceeding to the next scale. Each image included the needed grading scale for reference during the grading procedure to help prevent grader confusion. In addition to the descriptors for each step of the grading scales to be used, the examiners were provided with a reference illustration showing an example of staining corresponding to each step of the grading scales (Figure 2). A single reference image showing different degrees of staining severity (0–3 range) was used for the original NEI staining scale (Figure 2A) and the NEI dot-count staining scale (Figure 2B), and 2 reference images (0–4 range) were shown for the Lexitas modified NEI staining scale (Figure 2C and D). Each examiner proceeded through the set of images for the pilot or validation runs until grades had been recorded for each grading scale on all the images. The grading of the full validation image sets was separated by a period of 2 to 3 weeks for each examiner to allow for determination of intra-examiner repeatability. The images were randomly reorganized between the first and second grading of the validation sets.

Statistical Methods

The images used in this study were presented to the examiners with no identification markings. Images for the second read were reordered to assist with masking between the two reading timepoints. Data from graded images were exported into an Excel data set (Microsoft, Redmond, WA, USA). The data was reviewed for discrepancies against the original images. The dataset included the examiner’s last name, image number, region (central, inferior, nasal, superior, temporal), scale, and notation for the first or second read.

Summary statistics for each grading scale were calculated by type (illustration or photograph) and region based on the scores for each of the three examiners (labeled A, B, C) for each corneal region. Intra-examiner reliability between the two time points was assessed two ways: using the unweighted or simple kappa statistic to assess exact agreement and the weighted kappa statistic to assess partial agreement or association. Average intra-examiner kappas (simple and weighted) were computed for each examiner as the mean of each kappa statistic across the 5 regions to get an overall assessment of performance. Inter-examiner reliability utilized the scores from the first read of each examiner, and all three pairs of examiners (A-B, A-C, B-C) were assessed for agreement using the kappa statistics described above. Average inter-examiner kappas (simple and weighted) were computed for each pair of examiners as the mean of each kappa statistic across the 5 regions to get an overall assessment of performance. Note that due to limited data, kappas for the superior region for clinical photos are not produced. Therefore, the average intra- and inter-examiner kappas are based on the average of kappas for the 4 available regions.

Computations for all results were performed using PROC FREQ in SAS (2013, 2020; SAS Institute, Cary, NC, USA).^16,17 Plots were produced using JMP 16 (Cary, NC, USA), and summarize the simple and weighted kappa for each examiner or pair of examiners. Means and standard deviations (SD) of the kappa averages are used to summarize results across the 3 examiners or pairs of examiners. Note that the SDs are likely an underestimate of the true inter-examiner kappa averages since each examiner contributes to 2 pairs.

Results

Data was analyzed from the scores provided by the 3 independent examiners from each gradable corneal region on 32 images (18 illustrations and 14 photographs). A total of 154 corneal regions were gradable across the 3 grading scales (NEI density, NEI dot count, and the Lexitas modified NEI staining scales), for each examiner for each of the 2 rounds of the validation assessment. A total of 8 regions (0.9%) were not scored that were predetermined to be gradable during the assessments. Missing data for one or both scores for a particular region and scale were excluded from the intra-examiner and inter-examiner analysis.

The average intra-examiner kappa statistics for all 5 regions of the cornea for the illustrations and 4 regions for the clinical photographs is plotted by examiner, kappa type, and grading scale in Figure 3. The intra-examiner kappa statistics for the illustrations is presented in Figure 3A and the photographs is presented in Figure 3B. The mean (SD) intra-examiner simple kappa values for the graded illustrations using the NEI density, NEI dot count, and the Lexitas modified NEI staining scales were 0.67 (0.20), 0.91 (0.03), and 0.80 (0.20), respectively, and the weighted kappa values were 0.72 (0.17), 0.94 (0.03), and 0.92 (0.08). The corresponding mean (SD) intra-examiner simple kappa values for the graded photographs of corneal staining using the NEI density, NEI dot count, and the Lexitas modified NEI staining scales were 0.83 (0.04), 0.76 (0.07), and 0.77 (0.16), respectively, with the weighted kappa values were 0.88 (0.04), 0.85 (0.05), and 0.88 (0.14). A higher degree of agreement, based on the kappa statistic values was observed for the weighted kappa analysis. The mean intra-examiner weighted kappa values for all three grading scales for the photograph set, and the NEI dot count and Lexitas modified NEI staining scales for the illustration set were within the near perfect agreement category (0.8–1.0; Table 1). The mean intra-examiner weighted kappa value for the NEI density staining scale was in the substantial agreement category for the illustration data set (0.6–0.8; Table 1).

Table 1 Correlation of Kappa Values and the Degree of Agreement

Figure 3 The average intra-examiner kappa statistics (across all regions of the cornea) are presented by examiner (A, B, C), staining scale (NEI density, NEI dot count, Lexitas modified NEI), and kappa statistic type (simple, weighted) for the corneal fluorescein staining images, with the kappa values being presented for the illustrations in (A), and the photographs in (B).

The intra-examiner kappa statistics for each individual region of the cornea (center, inferior, nasal, temporal, and superior) is plotted by examiner, kappa type, and grading scale for the illustrations in Figure 4A. The intra-examiner kappa statistics for the center, inferior, nasal, and temporal regions of the cornea is plotted by examiner, kappa type, and grading scale for the photographs in Figure 4B. Similar to the intra-examiner kappa statistics for the average across all regions, the weighted kappa values indicated a higher degree of agreement across the regions, by examiner, and for each grading scale.

Figure 4 The intra-examiner kappa statistics for each region of the cornea are presented by examiner (A, B, C), staining scale (NEI density, NEI dot count, Lexitas modified NEI), and kappa statistic type (simple, weighted) for the corneal fluorescein staining images, with the kappa values being presented for the illustrations in (A), and the photographs in (B).

The average inter-examiner kappa statistics for all 5 regions of the cornea for the illustrations and 4 regions for the clinical photographs is plotted by paired examiner, kappa type, and grading scale in Figure 5. Paired inter-examiner comparisons were analyzed for data from examiners A – B, A – C, and B – C. The inter-examiner kappa statistics for the illustrations is presented in Figure 5A and the photographs in Figure 5B. The mean (SD) inter-examiner simple kappa values for the graded illustrations using the NEI density, NEI dot count, and the Lexitas modified NEI staining scales were 0.59 (0.06), 0.86 (0.03), and 0.78 (0.09), respectively, and the weighted kappa values were 0.65 (0.05), 0.90 (0.02), and 0.91 (0.03). Mean (SD) inter-examiner simple kappa values for the graded photographs of corneal staining using the NEI density, NEI dot count, and the Lexitas modified NEI staining scales were 0.80 (0.05), 0.84 (0.08), 0.69 (0.08), respectively, and weighted kappa values of 0.88 (0.03), 0.89 (0.06), and 0.88 (0.05). The mean inter-examiner weighted kappa values for all three grading scales for the photograph set, as well as the NEI dot count and Lexitas modified NEI staining scales for the illustration set indicated a high level of concordance in scores between examiners, with the weighted kappa values in the near perfect agreement category (0.8–1.0; Table 1). The mean intra-examiner weighted kappa value for the NEI density staining scale for the illustration set corresponded to the substantial agreement category (0.6–0.8; Table 1).

Figure 5 The average inter-examiner kappa statistics (across all regions of the cornea) are presented by paired examiners (A-B, A-C, B-C), staining scale (NEI density, NEI dot count, Lexitas modified NEI), and kappa statistic type (simple, weighted) for the corneal fluorescein staining images, with the kappa values being presented for the illustrations in (A), and the photographs in (B).

The inter-examiner kappa statistics for each individual region of the cornea is plotted by paired examiner, kappa type, and grading scale for the illustrations in Figure 6A. The inter-examiner kappa statistics for the center, inferior, nasal, and temporal regions of the cornea is plotted by paired examiner, kappa type, and grading scale for the photographs in Figure 6B. The intra-examiner kappa statistics by region, were generally similar to the intra-examiner kappa values by region, with the weighted kappa values indicating a higher level of agreement, and the range of values remaining in the substantial agreement or near perfect agreement categories for most of the regions. The lowest level of inter-examiner agreement was observed for the inferior region with the original NEI (density) staining scale on the illustration assessments (0.03 simple kappa; 0.13 weighted kappa; Figure 6A).

Figure 6 The inter-examiner kappa statistics for each region of the cornea are presented by paired examiners (A-B, A-C, B-C), staining scale (NEI density, NEI dot count, Lexitas modified NEI), and kappa statistic type (simple, weighted) for the corneal fluorescein staining images, with the kappa values being presented for the illustrations in (A), and the photographs in (B).

Discussion

This study was designed to evaluate the reliability and repeatability of the Lexitas modified NEI staining scale when used in the assessment of corneal fluorescein staining recognizing the importance of consistency and accuracy within and between examiners. The evaluation of the inter-examiner and intra-examiner assessments of corneal staining using the Lexitas modified NEI staining scale alongside the original NEI staining scale allowed for a statistical validation of the Lexitas modified staining scale that was developed to facilitate clinical research.¹⁴ The findings of this study indicate that based on the weighted kappa statistics, the Lexitas modified NEI staining scale performed similar to, or had a higher level of agreement for inter-examiner and intra-examiner assessments when compared to the original NEI staining scale, across the illustrations and clinical photographs. The mean intra-examiner and inter-examiner weighted kappa values across all gradable regions for all three grading scales were within the near perfect agreement category (0.8–1.0) or substantial agreement category (0.6–0.8). Evaluation of subsets of the examiner assessments by region of the cornea displayed a lower degree of agreement (both intra-examiner and inter-examiner, Figures 4 and Figure 6, respectively) in specific regions, particularly the inferior and temporal regions for the Lexitas modified NEI scale and original NEI scale. The higher degree of variability of grading observed in certain regions of the cornea was likely due to differences in the perceived area of involvement of the densest area of staining within the region. Differences in perception of the amount of area of the densest staining within a region can be a source of disagreement between examiners and even affect intra-examiner agreement when grading is based on the original NEI staining scale.

Both the simple kappa and weighted kappa statistics were included in this study to allow for an increased understanding of the differences in the intra-examiner and inter-examiner assessments of corneal staining. The values provided by the simple kappa statistics allow for an assessment of the degree of exact agreement in the grades assigned by the examiners either between the 2 validation assessments for the same examiner or for the grades assigned between pairs of examiners, where any difference in grades is determined as disagreement. The weighted kappa values provide additional insight into the degree of agreement by including the magnitude of disagreement in the calculation. This allows grades that were assigned by examiners that were closer together to result in less impact on the kappa statistic. In the context of this study, the average weighted kappa values were generally higher than the average simple kappa values, indicating that when a disagreement in grades assigned by examiners occurred, the magnitude of difference in the grades were less likely to be large differences.

The high level of agreement by the examiners noted in this study is of particular significance for the Lexitas modified NEI staining scale due to the expansion of the scale with respect to the allowance for grading to be performed in half-point increments (0–4 range), for a total of 9 grades possible, rather than an integer-based approach used by both the original NEI density and structured dot count staining scales (0–3) that have 4 possible grades. The expansion of the Lexitas modified NEI scale range to 9 grade options increased the potential for disagreement between examiners reviewing the same images of corneal staining; however, reliability and repeatability was not impacted by the expansion of the grading range, based on the average agreement across all gradable regions for intra-examiner and intra-examiner weighted kappa statistics. This may have been because of the extensive guidance and illustrations associated with the Lexitas modified staining scale. Expansion of a grading scale for corneal staining has the benefit of increasing the overall sensitivity of the scale to define and select a study sample using this clinical characteristic, as well as the potential to increase the sensitivity to observe a treatment effect within the study population. This has important implications for the design and cost of a study because low measurement variability combined with the increased ability to detect differences should result in smaller required sample sizes.

The effect of the expansion of the Lexitas modified NEI staining scale range to 9 grade options rather than 4 grade options is highlighted in looking across the rows within Figure 1. Grade 1 and grade 2 for both the original NEI and structured dot count staining scales have been separated into 2 grade options each for the Lexitas modified NEI staining scale, while Grade 3 for the original NEI scale and dot count staining scales have been expanded into 4 grades in the Lexitas modified NEI staining scale (2.5–4.0). This system increases the range of specificity for identifying patients with mild, moderate, and severe levels of staining. This will not only help with detecting treatment effects in clinical studies, but it could do the same in clinical practice and subsequently allow for more targeted disease management plans.

The Lexitas modified NEI staining scale is also distinguished from the original NEI density-based scale by a requirement to assess all the staining that occurs within each region of the cornea. By factoring in each instance of punctate staining across the region of the cornea, as well as potential areas of confluence or coalescence of staining in more severe instances, a more representational grade can be assigned, as compared to evaluating only the densest occurrence of corneal staining within a region. Therefore, the Lexitas modified NEI staining scale has utility for understanding not only regional damage that for example may occur with incomplete eyelid closure, but it can also provide an overall grade for the cornea, which could be useful as a primary outcome for a regulated clinical trial. The quantification of punctate staining required by the Lexitas modified NEI scale may allow for utility of the scale for automated assessment of corneal staining.

A limitation of this study is that it includes the use of images (illustrations and clinical photographs) of corneal staining rather than the direct observation of staining on patients in the clinic. Although assessment of corneal staining on patients observed through a slit-lamp will likely be the standard practice during a clinical trial research, standardization of assessing the same patient by different clinicians is challenging due to different practice locations and duration of the stain on the eye. The use of high-quality illustrations likewise provided the opportunity to develop a set of staining images that represented the full spectrum of severity for each grading scale. Thus, this study design was required to be able to fairly compare grades between investigators. This study design likewise simulates what might occur in a central reading center, which is also common practice with clinical trials. Nevertheless, a next logical step would be to repeat this study in a clinical sample at the slit-lamp. The restriction of the number of examiners to a total of 3 clinicians is also a potential limitation of the study, yet this was a necessary step in the validation process to begin to understand the properties of the new grading scale. Furthermore, the high degree of agreement observed for both the intra-examiner and inter-examiner assessments suggests that the use of 3 examiners was sufficient to perform the validation process for the Lexitas modified NEI staining scale in comparison to the original NEI density and structured dot count staining scales.

Conclusion

The results of this validation study indicate that the Lexitas modified NEI staining scale is easily used by clinicians experienced with grading corneal fluorescein staining, and the expansion of the grading scale to a 0 to 4 system with half point increments continued to result in a high degree of reliability and repeatability of grading assessments within and across individual examiners. The reliability and repeatability of the Lexitas modified NEI staining scale compared favorably with the original NEI staining scale even with the expansion of the Lexitas scale over the original 0–3 integer-based scoring system used by the original NEI scale. These data are important for designing future clinical trials, and this grading scale also has potential to be incorporated into everyday clinical practice for diagnosing and tracking the progression of diseases such as dry eye disease. Additional investigation into the utility of the Lexitas modified NEI staining scale is warranted to fully understand how this grading scale will perform in a clinical trial or everyday clinical practice.