Impact of Clinician Subjectivity on the Assessment of Dry Eye Disease Prevalence in a UK Public Health Care Patient Population

Introduction

In the past decade since tear osmolarity has become more widely available in the clinic, there have been conflicting reports about the utility of the biomarker. There is broad agreement that hyperosmolarity is a potent cellular stress that negatively impacts corneal and conjunctival epithelial cells. Increasing levels of hyperosmolarity cause an increase in epithelial apoptosis through cytochrome c, capspase-3, JNK and ERK expression,¹ as well as cornification of epithelial cells through increased transglutaminase activity.² Hyperosmolarity is also a potent pro-inflammatory signal, causing increased secretion of IL-1ß, IL-6, IL-8, TNF-A, and MMP-9 from the ocular surface.^3–5 A sophisticated co-culture model of the human ocular surface was able to replicate the core features of dry eye disease including elevated staining, reduced tear film breakup time (TBUT), increased cytokine and TLR-4 expression, increased NF-kB translocation and reduced lubricin expression, solely by exposure to hyperosmolarity caused by an elevated evaporation rate.⁶ Hyperosmolarity is also known to directly impact corneal nerves to interrupt synaptic action potentials, to fundamentally alter nerve sensitivity, and affect morphology in a dose dependent manner.^7–9 These and other factors have lead to the acceptance that tear hyperosmolarity is a core pathogenic mechanism in dry eye disease.¹⁰

Numerous studies have also demonstrated the clinical utility of tear osmolarity. A recent review of 163 articles regarding tear osmolarity, found that 72% of all articles had a positive impression, 21% neutral and 7% reported a negative impression of its clinical utility.¹¹ Importantly, the rate of positivity was independent of industry financial support, with 72% (75/104) of the independent studies rating positive.¹¹ In addition to reports detailing its value in diagnosing DED,^12–14 tracking therapeutic efficacy,^15–20 and grading DED severity,^21–23 tear osmolarity has been found to be important in planning for cataract surgery, as hyperosmolar tears can introduce significant variation in keratometry measurements²⁴ and cause significant unexpected refractive error along with a seven-fold increase in patient dissatisfaction.²⁵ Hyperosmolarity assessment has also found utility in testing asymptomatic diabetics,²⁶ oculoplastics,²⁷ tracking ocular surface outcomes in orthokeratology²⁸ and monitoring the health of the ocular surface following refractive surgery.²⁹

By contrast, a series of studies have proclaimed that tear osmolarity has little to no clinical value.^30–41 Common themes were the lack of correlation between tear osmolarity and other clinical measures of dry eye disease such as corneal fluorescein staining, tear film breakup time (TBUT) and meibomian gland evaluation^30–37 and the test-to-test variability of the marker.^38–41 When criticizing osmolarity, these studies do not independently evaluate the aforementioned signs on their own, with relatively few studies that quantify the test-to-test variation of staining or TBUT – nor do they recognize that the DEWS II consensus on diagnosis suggests each of these tests be performed,⁴² given the difficulty in making a diagnosis of DED. Diagnosis is especially challenging in patients with mild to moderate severity, part of which is the subjectivity in grading these signs, and the inevitable conflicts between signs and symptoms that arise in that population.²¹

It is the purpose of this study to examine the relative correlation and variation of osmolarity against the most commonly used DED signs and symptoms to better understand the critiques of the marker in such a context. These data are particularly relevant for clinical trial design, defining dry eye disease, how to value novel biomarkers in DED, and understanding the dichotomous aspects of tear osmolarity in the literature.

Materials and Methods

A non-randomized prevalence study recruited consecutive patients at four different UK NHS Hospital Ophthalmology clinics. This study was designed with broad inclusion criteria. Subjects 18 years and above presenting to the clinics were invited to participate. Those who agreed and signed the relevant information and consent form were enrolled in the study. Subjects were excluded if they had used eye drops within 2 hours of their appointment so as not to interfere with the dry eye assessments. Each site was asked to enroll up to 250 subjects. Consultants participating in the study adhered to Good Clinical Practice guidelines and followed the tenets of the Declaration of Helsinki. The study was approved by the UK National Research Ethics Service, NRES Committee South East Coast – Surrey.

Following subject demographics, history and ocular surface disease index (OSDI) symptom questionnaire, bilateral tear osmolarity measurements (TearLab Osmolarity System, San Diego USA) were performed prior to other assessments to avoid interrupting tear film homeostasis. Bilateral Oxford corneal fluorescein staining (grades 0–4), manual fluorescein tear film breakup time measured in seconds using a stopwatch and meibomian gland dysfunction on a 0–3 subjective scale representing none (0), mild (1), moderate (2) and severe disease (3), were evaluated. Consultant ophthalmologists that indicated their assessment of dry eye were masked to the tear osmolarity results, but could use any available clinical information in making their diagnosis. Subjects were then exited from the study and continued with their scheduled appointments. TBUT, MGD & corneal fluorescein staining tests required a doctor’s interpretation and were, by our definition, subjective. Osmolarity and OSDI, where the measurement instrument did not change between sites, should not be significantly different if the distribution of patients is similar across sites and are considered objective.

Descriptive statistics for the biomarkers were calculated between sites in an effort to understand the impact of observer subjectivity on each of the dry eye markers. The more severe eye from each patient was used in the analysis of each marker. Diagnostic cutoffs at mild and moderate disease thresholds were assessed for each marker (Table 1).^13,21 Mild fluorescein staining was evaluated ≥ grade 1 and moderate staining ≥ grade 2. Mild TBUT was set at <10 seconds and moderate TBUT ≤7 seconds. The mild osmolarity cutoff was >308 mOsm/L and the moderate osmolarity cutoff was >316 mOsm/L. MGD grading thresholds were ≥ grade 1 and ≥ grade 2. Symptoms as measured by OSDI were considered mild with a score ≥ 13 and moderate ≥ 23. Histograms of the number of mild and moderate signs according to the aforementioned thresholds were calculated across all sites. Distributions of each sign were plotted against OSDI symptoms to understand the predictive value of patient input.

Table 1 Mild, Moderate Cutoffs for Dry Eye Disease Markers

The diagnostic performance of DED signs and symptoms were explored by sequentially omitting one of the biomarkers, then using the remainder of the included markers to define normal from dry eye subjects, similar to the method of Tashbayev et al.³² When incorporated into the inclusion criteria, mild thresholds of MGD ≥ grade 1, OSDI ≥ 13, TBUT < 10 seconds, staining ≥ grade 1, and osmolarity >308 mOsm/L were used. Bonferroni’s method was used to adjust t-test values for multiple comparisons. Pearson’s correlation coefficients were calculated for each marker and against Consultant diagnosis to help elucidate interactions between the markers.

Results

A total of 818 subjects with complete data sets were included in the study, with 465 females aged 67.1 ± 16.7 years, and 353 males aged 65.0 ± 15.9 years. A total of 189 subjects were seen at The Great Western Hospital, 245 subjects at Hillingdon Hospital, 69 subjects at Medway Maritime Hospital, and 315 subjects at St. Peter’s Hospital. A total of 308 subjects were currently taking some form of medicinal eye drop, while 510 reported no data on drops. A total of 378 subjects had some form of surgery, whether cataract (299), laser vision correction (47), or trabeculectomy (19), with the remainder reporting a rare type of surgery below statistical analysis thresholds, including ectropion repair, entropion repair, detached retina, tumor removal, iridotomy, corneal grafts, foreign object removal, ptosis surgery, chalazion, and pupilloplasty. Contact lens use was reported in 43 subjects.

Site-to-site differences in dry eye markers are shown in Table 2. Each site showed a similar mild to moderate patient population. There were statistically significant site-to-site differences in each sign. MGD grading was lower in sites 3 and 4 compared to sites 1 and 2, TBUT was significantly lower at site 2 compared to site 1 and 4, corneal staining was higher at sites 2 and 3 as compared to sites 1 and 4, and osmolarity was higher at site 4 than sites 1 and 2, all p < 0.05.

Table 2 Averages of Each Dry Eye Disease Marker by Site.

Based on the presenting signs and symptoms, with the exclusion of tear osmolarity to which they were masked, consultant ophthalmologists reported a range of diagnosis rates for DED from 42.9% to 62.3%, with a mean of 48.7% across all sites. Positivity rates for each of the DED markers are shown in Table 3. Fluorescein staining gave positivity rates of 41.3% to 84.1% (57.6% averaged across all sites) for the mild threshold ≥ grade 1, and 10.6% to 40.6% (19.8% averaged across all sites) for the moderate threshold of ≥ grade 2. TBUT ranged from 39.1% positive to 61.6% positive for subjects <10 seconds (50.4% of all subjects), while at the moderate threshold of ≤ 7 seconds the TBUT positivity ranged from 23.2% to 49.8% (33.7% average across all sites). For the mild threshold of tear osmolarity >308 mOsm/L, positivity rates ranged from 63.7% to 72.4% (67.4% averaged across all sites), and the moderate threshold of >316 mOsm/L ranged from 35.1% to 56.8% positive (45.7% overall). Mild MGD ≥ grade 1 ranged from 28.9% to 51.3% (40.7% overall), while moderate thresholds of MGD ≥ grade 2 reported lower percentages of positive patients, ranging from 1.4% to 17.5% (10.5% overall). Symptoms, as measured by OSDI ≥ 13 varied between sites from 57.6% to 71.0% (60.8% overall), and at a moderate threshold of OSDI ≥ 23, symptom positivity ranged from 38.8% to 49.3% (41.8% across the sites). At the mild thresholds, the ratio of maximum to minimum positivity between sites showed tear osmolarity to be the least variable (114%), followed by OSDI (123%), TBUT (158%), MGD (178%) and staining (204%). At the moderate thresholds, the ratio of maximum to minimum positivity between sites showed OSDI (127%) to be the least variable, followed by tear osmolarity (162%), TBUT (215%), staining (383%) and MGD (1205%).

Table 3 Positivity Rates for Dry Eye Disease Markers

A histogram of the dry eye measurements is shown in Figure 1, with the percentage of all patients compared to the number of observed mild or moderate assessments. The most common patient profile was a subject with one moderate sign or symptom (37.3%), followed by subjects with two mild signs/symptoms (26.0%) or two moderate signs/symptoms (26.0%). A very small fraction (2.4%) of subjects reported no sign or symptoms at all, and only 20.0% of subjects reported no moderate signs/symptoms, demonstrating that evidence of DED is highly prevalent within the general clinical population.

Figure 1 Histogram of all subjects broken down by number of mild symptoms (black), and number of moderate symptoms (light gray).

Figure 2 shows the distribution of each classical dry eye sign against OSDI as the independent variable. The data shows no clinically meaningful relationship between any of the measurements and symptom severity; with an approximately equal chance of exhibiting either very little staining or strongly elevated staining for both asymptomatic and highly symptomatic subjects. This observation was found to be consistent for TBUT, osmolarity and MGD, establishing that these variables are orthogonal across the broad population, and that symptom intensity has no predictive power as to whether the patient will present with any of the dry eye signs.

Figure 2 Scatterplots of ocular surface disease signs plotted against OSDI symptoms. There is no clinically meaningful relationship between any of the measurements and symptom severity.

To quantify the predictive power between the signs, Table 4 shows the inter-marker correlations across the entire subject population. As with the OSDI score, there were no clinically meaningful associations between any of the markers (average r²=0.05±0.07), except for fluorescein staining and fluorescein TBUT which showed a small but not clinically meaningful correlation of −0.30 (p < 0.001), ostensibly because a single instillation of fluorescein is used to assess both markers consecutively during a single exam and some dependency on retained dye is expected. The consultant ophthalmologist diagnosis correlated most strongly with staining at r = 0.51 and TBUT r = −0.37, followed by MGD r = 0.24 and OSDI r = 0.18 (all p < 0.001). As osmolarity was masked to the physician, there was no correlation to the diagnosis (r = 0.01, p = 0.869).

Table 4 Pearson Correlation Coefficients Between Dry Eye Disease Markers.

Borderline cases in which only one sign was evident with symptoms (OSDI ≥ 13) or cases where at least one sign was evident without symptoms (OSDI < 13) are shown in Table 5. The most common type of patients in the symptomatic subset were subjects with symptoms and hyperosmolarity but no other mild (N = 67, 8.2% of study population) or moderate signs (N = 110, 13.4% of study population). Without the benefit of osmolarity to guide diagnosis, physicians identified a small percentage of these subjects as DED with 11.9% of the mild and 35.5% of the moderate subjects being diagnosed. In the few cases where staining was the only evident mild (N = 17, 2.1% of study population) or moderate sign (N = 23, 2.8% of study population), clinicians elected to diagnose DED at 52.9% and 73.9% respectively, reflecting the well known reliance on staining for DED diagnosis. Subjects with zero to low symptoms (OSDI < 13) frequently exhibited one or more signs of DED, with at least one mild sign (N = 301, 36.8% of study population), at least two mild signs (N = 206, 25.2% of study population), or at least one moderate sign (N = 207, 25.3% of study population) accounting for over 1 in 4 subjects that completed the study. Despite clinical evidence of one or more dry eye signs, these cases were diagnosed at rates of 41.5%, 52.9% and 47.8% respectively, close to a coin flip in terms of chance.

Table 5 Borderline Cases in Which Only One Sign Was Evident with Apparent Symptoms (OSDI ≥ 13) or Signs Were Evident Without Symptoms (OSDI < 13)

Given the lack of clinically meaningful relationship between the markers, the impact of inclusion criteria on the diagnostic performance of the markers is shown in Table 6. None of the measures could usefully differentiate normal and DED patients if that same marker was not part of the inclusion criteria. When TBUT was absent from the inclusion criteria, normal subjects averaged 13.1 ± 9.2 seconds as compared to 11.1 ± 6.7 seconds for the DED subjects (p = 0.295), with dry eye subjects reporting mean values above the mild TBUT threshold. Similarly, when excluded from inclusion criteria, tear film osmolarity averaged 318.4 ± 18.3 and 319.9 ± 21.1 mOsm/L for normal and dry eye subjects, respectively (p = 1.000). When MGD grading was not part of inclusion criteria, normal subjects reported a higher MGD score (grade 0.6 ± 0.7) than DED subjects (grade 0.5 ± 0.7, p = 1.000). Symptoms as measured by OSDI were unable to statistically distinguish normal subjects (19.9 ± 19.4) from DED (23.7 ± 20.5, p = 0.854) if OSDI scores were not part of the inclusion criteria. When excluded, staining showed grade 0.4 ± 0.6 for normal and grade 0.8 ± 0.8 for DED, which, while statistically significant (p = 0.034 without multiplicity correction, p = 0.17 with Bonferroni), both groups were lower than the mild cutoff, and the difference between groups was below the resolution of staining measured in the clinic.

Table 6 Dependence on Inclusion Criteria for Distribution Dry Eye Biomarkers

Discussion

This study evaluated the performance of DED tests within the general population of patients presenting to NHS hospitals. The objective tests of tear osmolarity and OSDI were observed to be more repeatable across sites (Table 3) as compared to the subjective tests of corneal fluorescein staining, TBUT and MGD. This finding supports the hypothesis that the distribution of disease severity was similar across the four sites, and that the variation in interpretation between tests was due primarily to the differing opinion of the consultants. An earlier study examining the longitudinal variability of these same markers also found that osmolarity and OSDI were less variable than their subjective counterparts,²⁰ although that study did not control for site-to-site variation.

The results of this study are particularly relevant for clinical trial design, where sponsors often rely upon subjective grading of corneal staining as a primary endpoint. Corneal staining is subject to variations from the amount of fluorescein instilled, the amount of fluorescein retained due to tear turnover rates, the time of assessment since fluorescein instillation, slit lamp illumination intensity and filter selection.⁴³ The wide variation in site-to-site corneal staining results herein echo other studies that examined staining variability, e.g., those reported in Kourukmas R et al, which found only a marginal intra-rater agreement (Cohens-Kappa = 0.436) for corneal specialists with 6–29 years of experience, with the majority of specialists below K = 0.7 for mirrored images within the same grading session.⁴⁴ Despite such variation, clinicians most closely associated corneal staining with DED diagnosis in this study (Table 4), preferentially relying on staining (r = 0.51) over TBUT (r = −0.37) or symptoms (r = 0.18) when making a diagnosis. Subjects with no other signs of the disease and mild staining were diagnosed at over twice the rate as compared other signs with symptoms (Table 5). Accordingly, rates of DED inclusion to a clinical trial could range from 10.6% to 84.1% depending on the site to which a subject presents and whether the trial protocol uses a grade 1 or grade 2 threshold. Staining remains both the most relied upon⁴² marker and one of the most widely varying and subjective tests for DED.

MGD positivity rates ranged from 1.4% to 51.3% of subjects between sites, depending on whether grade 2 or 1 was used. While a widely cited study is often misquoted to suggest that 86% of all DED subjects have MGD,⁴⁵ the denominator of that figure excluded the substantial number of patients that were deemed to have DED, but could not be classified as either aqueous deficient or evaporative due to the lack of observable meibomian gland abnormalities or low Schirmers results. When all DED subjects were included, Lemp MA et al found that 60.7% (136/224) of the recruited dry eye subjects had some level of observable MGD,⁴⁵ which is closer to the 51.3% found in the current study. The implications of this perspective are profound, since the oft misquoted, co-opted marketing message of 86% of patients having MGD does not include the finer detail that of the entire population of DED patients, more likely somewhere between one-third to one-half have causative MGD, where the disease is sufficiently impactful to initiate a dry eye state.

In this study, 21.6% (177/818) of all subjects presenting to the NHS hospitals exhibited hyperosmolarity and elevated OSDI, with no other clinically observable signs of dry eye disease. An earlier investigation found 29% (65/224) of subjects in a multi-site US study followed this same pattern.⁴⁵ The prevalence of these symptomatic hyperosmolar subjects was over three times that of the short TBUT type of dry eye in this study (56/818, Table 5), where the ocular surface appears completely normal, but the tear film can be compromised and accompany significant symptoms.⁴⁶ Since hyperosmolarity is not observable at a slit lamp, clinicians in this study diagnosed only a fraction of these patients without access to the osmolarity result.

Similar to how short TBUT DED is classified as its own subtype of the disease,⁴⁶ it is becoming clear that the symptomatic hyperosmolar cohort, without other signs, should also be considered as its own classification. In one recent study, in a cataract surgery setting this type of patient (321.0 ± 9.7 mOsm/L, 9.0 ± 1.7 second TBUT, 88% subjects grade 0–1 corneal staining, 71% symptomatic) showed significantly worse uncorrected visual acuity (0.89 ± 0.12 vs 0.76±.14, p < 0.01), significantly worse spherical equivalent (0.60 ± 0.34 vs 0.28 ± 0.25, p < 0.01) and a 10% rate of >1.0 diopter misses from target compared to 0% in a normal group (298.2 ± 7.6 mOsm/L, 87.5% asymptomatic).²⁵ In addition, while 62.5% of the normal population was within 0.5 D of target, only 26.8% of the hyperosmolar group achieved that target.²⁵ Another study of symptomatic, hyperosmolar patients (327.6 ± 2.8 mOsm/L, 34.6 ± 2.1 OSDI, 0.51 ± 0.1 staining, 8.1 ± 0.8 TBUT, 11.8 ± 1.5 Schirmers) found significantly lower corneal stromal reflectivity as measured by OCT in hyperosmolar eyes relative to controls (p < 0.05) and significantly thinner central corneal thickness in the upper quartile of hyperosmolar eyes (539.1 ±7.4 µm than in control eyes (583.1 ±15.0 µm, P = 0.02) or in the lower quartile of hyperosmolar subjects (622.7 ± 5.8 µm, P < 0.0001).⁴⁷ In the absence of other observable signs of dry eye disease, hyperosmolarity is, by itself, predictive of surgical complications and altered corneal phenotypes.

The current study also reinforces the observation that dry eye signs are uncorrelated across the general population.^48,49 As such, comparing osmolarity to a basket of other markers is problematic when there is no predictive information contained within the other measurements. This is known as selection bias. The 2017 TFOS DEWS II Diagnostic Methodology report states: “Selection bias occurs when efficacy of metrics that were used in the selection and differentiation of subjects are directly compared to a novel test that was not used as part of the inclusion criteria… Many biomarkers … provide novel insight into disease pathogenesis, but because this information is unavailable from clinical observation, comparing performance of novel diagnostic metrics against the traditional signs such as staining, TBUT and symptoms will result in an apparently poor performance”.⁴² Selection bias creates a false standard against which new markers are measured, as can be seen in several studies which presented largely predetermined results based on the selection bias inherent to the study design.^{30–33,35–37} Tashbayev et al used a near identical approach to Table 6 of this study, comparing subjects with OSDI ≥ 13, presence of staining, MGD, abnormal TBUT and Schirmer tests to osmolarity.³² Being the only non-included marker, osmolarity did not differentiate the statistically defined groups.³² Similarly, Szalai et al (≥2 of Schirmer <10 mm, TBUT < 10 seconds, fluorescein staining > 4 bright spots)³⁵ and Messmer (≥ 3 of OSDI score >15, staining ≥ 1, TBUT < 7 seconds, Schirmer test < 7 mm, presence of blepharitis/meibomitis)³⁰ excluded osmolarity from a statistical definition of DED and again found that osmolarity did not differentiate those groups as defined. Notably, similar to Figure 1 of this study, Messmer reported that in the “normal” group, only 16 of 200 patients showed no signs or symptoms of DED and in 71 control subjects, up to 2 signs/symptoms of DED were obvious.³⁰ Owing to the poor correlation of dry eye markers, it is a mathematical certainty that osmolarity would not perform well in such a statistical tautology, as do any of the dry eye markers when excluded from the definition of the disease (Table 6). Accordingly, the frequently repeated criticism that osmolarity does not correlate to other markers^30–37 is invalid since none of the markers correlate (Table 4). More specifically, if osmolarity did correlate strongly with staining or TBUT, there would be no additional information gained in its measurement. Criticism based on a lack of correlation offers little insight into the value of tear osmolarity or any other test evaluated in such a manner.

Rather, we suggest that novel biomarkers are best evaluated by their relevance to surgical decision making, ability to predict and track an effective therapeutic response and consistency across different clinical sites of similar demographics. Based on the current findings and existing body of literature, osmolarity remains a unique, objective biomarker that provides information not otherwise available by clinical observation alone and can meaningfully supplement clinical opinion in a variety of ophthalmic disease states.

Conclusion

A large fraction of subjects exhibit high symptoms and hyperosmolarity but no other observable signs of dry eye disease, the majority of whom were undiagnosed by clinical assessment without the benefit of tear osmolarity. Subjective signs of DED varied considerably, whereas objective measurements of OSDI and osmolarity were the most consistent between sites.