Interobserver Reliability of Tear Break-Up Time Examination Using “Smart Eye Camera” in Indonesian Remote Area

Introduction

Dry eye disease (DED) is a disease of the tears and ocular surface with various risk factors.^1,2 DED is the most common eye disease encountered in daily practice with a prevalence ranging from 5% to 50% of the entire world population, numbers keep increasing by screen time, lifestyle factors, environment, and aging.¹ This condition affects patient’s quality of life by disturbing visual ability and has a potential damaging ocular surface. It is also one of the frequent causes of patient visits to the ophthalmologist.^2,3 Clinical examinations that can be performed to establish a diagnosis of DED include TBUT examination as an objective indicator of tear film stability, fluorescein staining of the cornea and conjunctiva.^3,4 Subjective findings are assessed using OSDI (Ocular Surface Index Disease) questionnaire. The diagnosis for DED is sometimes quite difficult regarding inconsistency between the subjective report of symptoms and the clinical findings that lead to underdiagnosing, undertreatment, and affecting its prognosis.⁵

The Asia Dry Eye Society emphasizes the crucial role of TBUT examination and establishes TBUT as the most important objective test for DED patients.⁵ Therefore, to evaluate TBUT along with other ocular damages in the anterior segment caused by DED, a slit-lamp biomicroscope is a highly essential tool. However, conventional slit lamps are large, heavy, expensive, and can only be operated by specialized eye doctors, residents, or trained professionals in hospitals or eye clinics. Hence, these devices are not suitable for use in remote areas.^5,6 Portable slit-lamp is an alternative option as they are easy to carry from one area to another, but expensive and cannot capture images of the anterior segment of the eye.^5,6

Technology in detecting dry eye disease is crucial, especially in remote areas. Remote areas usually have limited access to eye health services and mostly do not have an ophthalmologist. Some remote coastal areas tend to be more often exposed to dry eye risk factors. In addition, from the age range, DED could affect senior citizens who have obstacles of mobilization, where they usually need family assistance to take them to the examination site.^7,8

Limited health care facilities along with COVID-19 pandemic unable clinicians in addressing a standard method for DED diagnosis and rely on patient’s subjective symptoms alone. This method can only be used as screening, not diagnosis, and is often unreliable. The use of alternative tools that can resemble a slit lamp for areas with limited access to eye health services is still under study that has not been completely resolved. This background emerges the importance of SEC invention as a simple, effective, and efficient technology to detect DED. SEC is designed to assess TBUT examination, combined with OSDI as a standard method for DED diagnosis.^3,5 Therefore, it helps in establishing a more precise DED diagnosis, decreasing preventable, unnecessary DED complications, raising better prognosis, and increasing patient’s quality of life.

Smart eye camera (SEC) is a smartphone add-on device that functions similarly to existing portable slit-lamp microscopes, making it easier to use by non-specialized eye healthcare providers and enabling teleconsultation (Figure 1).⁷ It mimics the conventional slit-lamp illumination and is beneficial in detecting signs of DED. The SEC emits blue light at a wavelength of 488 nm when a blue acrylic resin filter is placed over the smartphone’s light source. Any remaining fluorescein on the cornea will appear as a green color. Its capabilities allow it to capture characteristic fluorescein images in DED and it apps enable it to record TBUT examination as video that will stored in “iCloud” (Apple, Inc, iCloud, Cupertino, California, United States of America) mimicking storage. The complement apps of SEC have the ability to record patients’ conditions, provide added value to healthcare practitioners for data collection and support in referral needs.^9,10 This distance between camera to eye is important because the convex lens in front of the camera was designed to be in the best focus by 4 cm. The resolution of the video was 4K, with a frame rate of 30 frames per second. The application of the smart eye camera is demonstrated through repeatability, reproducibility, and validity assessments.^7,9

Figure 1 SEC as smartphone attachment device (A). TBUT examination under slit lamp (B) using SEC (C). A non-ophthalmologist filmed TBUT test using SEC (D).

Research on alternative assistive devices to facilitate eye examinations has been widely conducted. A previous study by Shimizu et al demonstrated that SEC has adequate validity and reliability for diagnosing DED in a clinical setting in Japanese populations.^10,11 However, research has yet to be conducted on DED examination using SEC in remote areas, specifically in Indonesia. The objective of this study aims to determine the interobserver reliability of TBUT measurement in diagnosing DED using SEC among an ophthalmologist and a non-ophthalmologist (resident) in Indonesia remote area as telemedicine solution in making diagnosis of DED.

Materials and Method

This study was conducted with analytical cross-sectional study where data were collected at the time of eye examination. Participants who agreed to join in this study were asked to fill out informed consent, biodata (name, age, gender), the OSDI questionnaire, Schirmer test, and TBUT examination that was conducted using SEC. Observers then interpret TBUT measurement and diagnosis of DED and, and the data of interobserver interpretation were conducted and analyzed.

Study Setting

The study was conducted in Gema Santi Regional Public Hospital, Nusa Penida, Bali, Indonesia. The population reached in this study are all patients with the age of 40–75 years who visited Gema Santi Regional Public Hospital, Nusa Penida, Bali, Indonesia, from September 2nd–to 4th, 2022, and met the inclusion and exclusion criteria using consecutive sampling technique. This study obtained a total of 144 patients and 36 patients excluded due to preexisting diabetes. A total 9 out of 108 patients were excluded due to patient’s cooperation and poor images quality that made videos unable to be interpreted. The sample size in this study was collected based on the correlation hypothesis test formula with the following indicatosr: Zα (5%) =1.96, Zβ (Power 90%) = 0.128, r (minimum estimated correlation) = 0.3. Based on this formula, the minimum number of samples required to be generalized in population are 158. A total number of 198 samples were included in this study (Figure 2).

Figure 2 Research flowchart.

Inclusion criteria are residents aged of 40–75 years, who are willing to take part in the research and sign informed consent. Written Informed Consent was obtained from all participants, so the entire population who met inclusion and exclusion criteria were sampled. The Research Ethic Committee Faculty Medicine of Udayana University approved this study and (Ethical Clearance Dated July 4th, 2022, approval number 1814/UN14.2.2.VII.14/LT/2022) adheres to the tenets of the Declaration of Helsinki. Patients will have option voluntarily to provide informed consent to participate in this study. All parties will make maximal effort to protect patient information and privacy. Exclusion criteria are residents who have history of wearing contact lens, pregnant, history of diabetes mellitus, history of Laser-Assisted in situ Keratomileusis (LASIK) and other eye surgery within 1 month, history of eye trauma, eye infection, allergic to fluorescein and using steroid eye drops or other medications such as antiglaucoma, antipsychotic, diuretics, and poor images on video captured using SEC due to uncooperative patients.

Data collection techniques were carried out with several collection techniques including literature studies and questionnaires. A literature study explored books, journals, and other information relevant to the research. Then, questionnaires were distributed to research samples. The research instruments used several measurements including TBUT examination using fluorescein and assessed with SEC (OUI Inc., SEC-i07, Tokyo, Kanto, Japan). Fluorescein strips (Surgitech Innovation, Fluorescein Sodium Ophthalmic Strips, Karnal, Haryana, India), Schirmer strips (Surgitech Innovation, IO^R Schirmer, Karnal, Haryana, India), and Indonesian version of OSDI questionnaire were used to measure subjective and objective indicators of DED.^5,8 The SEC was placed 4 cm from the corneal apex for the SEC examination. This distance is important because the convex lens in front of the camera was designed to be in the best focus by 4 cm. The resolution of video was 4K, with a frame rate of 30 frames per second.¹¹ The measurement of TBUT based on the video taken by SEC was assessed by two observers (an ophthalmologist and a non-ophthalmologist). For the evaluation, a TBUT shorter than 10 seconds was defined as tear film instability. An OSDI score ≥13 was defined as positive for subjective DED symptoms, according to previous studies.^5,12

The diagnosis of DED was established based on objective and subjective indicators.⁵ Procedure of TBUT assessment with SEC was used a standard procedure between observer 1 and observer 2. Observers then interpret TBUT measurement and diagnosis of dry eye disease based on SEC examinations that recorded on video. All observers did not know each other interpretations. Data were then collected and analyzed.

Statistical Analysis

Patient characteristics were analyzed descriptively, where numerical variables were analyzed by mean and standard deviation, and categorical data by percentage. The SPSS for Windows version 21.0 software was used for statistical analysis.

The Kolmogorov–Smirnov and Levene’s analysis is presented as a non-parametric test with a significance level of 5%.

The suitability test will be analyzed using a numerical suitability test (Bland–Altman) between observers using TBUT measurements in the DED group and those without DED. Based on Bland–Altman test results, the mean difference value with a 95% confidence interval (95% CI) and the value of limit agreement or LoA (lower and upper limits) will be obtained. The mean difference value is the average difference between the two methods. The two examination methods are said to have conformity if the value is close to the zero value (line of equality) and is reliable in the 95% LoA range, which is between the TBUT value of −10 to 10 seconds.

The suitability test analyzed using a categorical variable suitability test (kappa coefficient). This test serves to assess the suitability of the diagnosis of DED between observers. The test used is the kappa reliability test. Consistency between observers is assessed based on the kappa coefficient and person agreement.

Results

Characteristics of Subjects

This study obtained a total of 99 patients and 198 eyes. The sample characteristics of the study were as follows: 51.5% were female and 48.5% were male, with a mean age of 55.22 ± 9.78 years. The mean OSDI score was 45.32, the Schirmer test result for the right eye was 12.83 mm, and the Schirmer test result for the left eye was 15.13 mm (Table 1).

Table 1 Research Sample Characteristics

Interobservers Consistency Agreement

The inter-observer reliability of TBUT data using Bland–Altman analysis shows the average difference information between TBUT measurements by two observers is 0.78 (95% CI=−0.31–1.26; P-value=0.0013; Table 2). This indicates the average amount of difference or variation between measurements.

Table 2 Interobservers Agreement Using Bland–Altman

The DED diagnosis was carried out with the SEC and found a good agreement between observer 1 and 2 kappa coefficient = 0.71, P-value = 0.0013). The inter-observer agreement is an important measure of reliability and consistency in diagnostic judgment. More agreements in the DED and non-DED categories would indicate stronger agreement between observers (Table 3 and Figure 3).

Table 3 Interobservers Agreement on DED Diagnosis

Figure 3 Bland–Altman plot for TBUT assessment between observer 1 and 2.

The reliability of DED diagnosis by comparing OSDI score and TBUT was found to have no correlation (r=−0.037; P-value=0.178; Table 4). This indicates poor agreement between subjective and objective findings in diagnosis of DED supports other reported previous studies and DED consensus.^1,2,5

Table 4 Correlation Between OSDI and TBUT on DED Diagnosis

Discussion

The study aimed to assess the interobserver reliability of TBUT measurement using SEC for diagnosing DED. The results indicated a good level of interobserver consistency in assessing TBUT, suggesting that SEC can be a reliable tool for diagnosing DED. This finding is important as it highlights the potential of using technology to improve diagnostic accuracy and reduce variability between observers.

Two large studies on the prevalence of DED conducted in the United States using the same criteria for DED diagnosis, the Women’s Health Study (WHS; n = 39,000 women, age 49+ years) in 2003 and the Physicians’ Health Study (PHS; n = 25,000 men, age 50+ years) in 2009, were used to make comparisons of DED between genders and between ages.^9,10 Age is cited as one of the risk factors for DED where increasing age decreases tear production. The previous study is consistent with this study in that it was higher in female (51.5%) than male (48.5%) and the mean age was 55.22 ± 9.78 years. A wider age range for future studies may be considered for future studies to get a truer population picture of dry eye not only in the older population.

The difference in Schirmer test results in the right and left eye was counted as a normal event as the diagnosis and severity of DED could differ in both eyes. DED is a multifactorial ocular surface disease that affects not only bilaterally but also could affect unilaterally. One may have been diagnosed with DED, whether the fellow eye could be completely normal.^3,5,9

NIBUT (non-invasive tear break-up time) has recently gained popularity in assessing DED. Wolffsohn et al recommended NIBUT assessment of tear instability.¹³ However, it is still not widely used and needs a high cost, as evidenced by several surveys of optometrists and ophthalmologists.^14–16

In contradiction, TBUT is still widely used by dry eye practitioners,^14–16 including the researchers of the Sjogren’s International Collaborative Clinical Alliance (SICCA)¹⁷ and is recommended as a dry eye syndrome diagnostic test by the Academy of Ophthalmology Preferred Practice Pattern.¹⁸ Bought together, this evidence shows TBUT using sodium fluorescein remains a critical evaluation in dry eye diagnosis and has reasonable diagnostic accuracy. Based on these considerations, our study that uses TBUT is reliable and appropriate due to its simplicity to detect DED in remote areas.

The concentration and volume of fluorescein dye instilled for the break-up time test has been discussed and remains controversial, whether by wetted strip or micropipette. Wu et al have shown that the volume of fluorescent liquid used can significantly affect TBUT results, so a micropipette should be used to control the volume.¹⁹ Johnson et al found differences between TBUT values measured after using fluorescein strips and a micropipette calibrated for 1 μL were not significant (P = 0.95). In addition, no significant differences were found in the repeatability of TBUT recordings between the instillation techniques used in that study (P = 0.18).²⁰ A recent study showed that the maximum TBUT may also be considered to diagnose dry eye. The sensitivity and specificity of the TBUT test are 72% and 61%, respectively.²¹ Another study by Paugh et al found no difference among the methods of Standard Strip, DET and 2.0 μL of 1.0% liquid dye under normal conditions. In the dry eye group, a shorter TBUT compared to liquid dye and standard strip was only found in MGD subjects, but the results of standard strip and liquid dye did not differ.^21,22 Following previous studies, we decided to use fluorescein strips in assessing maximum TBUT examinations as they are sterile and the amount of fluorescein is more accurate for each test, thus reducing bias and achieving standardized measurements.

SEC was used for pre-screening before the ophthalmological examination and post examination to confirm the diagnosis by the attending specialists to the residents. Shimizu et al reported a moderate agreement in the interobserver reliability between the conventional slit-lamp microscope and SEC for TBUT measurement and DED diagnosis in Japanese populations (weighted kappa κ = 0.527). This result implied that SEC is reliable and can mimic conventional slit-lamp as gold standard device.¹¹ This study supports the findings of previous studies that interobserver reliability in assessing TBUT using Bland–Altman was 0.78 (95% CI = 0.31–1.26, P value 0.001) and interobserver reliability in diagnosing DED showed good agreement (weighted kappa = 0.71).

Another study by Zhang et al also reported high agreement for measurement between an undergraduate student and a postgraduate student was 0.864 (95% CI, 0.810–0.904), between an undergraduate student and an ophthalmologist, was 0.922 (95% CI, 0.889–0.945), between a postgraduate student and an Ophthalmologist was 0.949 (95% CI, 0.927–0.964), and the average was 0.911 (95% CI, 0.881–0.934), the mean was 0.911 (95% CI, 0.881–0.934), using the same technology called Lens Attachment for Smartphone.²³

Based on two studies by Shimizu et al, we decided to do further research using SEC as novel inventions in Ophthalmology technology and this is the pilot study in populations-based research using SEC in diagnosis of DED in Indonesia.^7,11 This current study showed good consistency between interobservers (ophthalmologist and non-ophthalmologist) in assessing TBUT examination (mean difference=0.78; 95% CI=−0.26–1.84; p=0.143), and DED detection (kappa coefficient = 0.71). This implies that first, anyone can perform the examination; second, non-ophthalmologists are as reliable as ophthalmologists in interpreting the TBUT examination and dry eye diagnosis based on SEC video examination.

We assume that interobservers’ agreement in our study might be high due to standardized use of fluorescein strips that have previously proven to have good sensitivity and specificity in assessing TBUT, comparisons were only made between two observers, and we excluded any poor images taken with the SEC that may have affected the observer’s interpretation. The SEC study in Japanese conducted among three observers also supports good inter-observer agreement in scoring the TBUT test. Future analysis between more observers is needed to add more value to this study. On the other hand, our findings seem to provide a promising future in diagnosing and managing DED as telemedicine in rural areas.

In our study, reliability of DED diagnosis by comparing OSDI score and TBUT was found to have no correlation (r=−0.037; P-value=0.178). A reason might be because residents live near beach which tends to have more subjective dry eye symptoms. Even though DED is a common eye problem, the diagnosis is not straightforward. The symptoms may also be encountered or overlap in other eye diseases and might not well correlate with the signs. Symptoms can still not occur despite reduction of tear production. One may have minimal complaints to asymptomatic, but their examination might show mild-to-moderate dry eye. Otherwise they might have severe symptoms without any significant complaints.^3,5,11

Inter-observer reliability in examining tear break-up time (TBUT) using smart eye camera in Nusa Penida refers to consistency or agreement between observers involved in measuring TBUT using the technology. Assessment of interobserver reliability involves data analysis that includes the mean difference between observers, confidence intervals, and appropriate statistical methods to measure interobserver consistency.

The inter-observer reliability of the TBUT data shows that the average difference in information between TBUT measurements by two observers is 0.78. This indicates the average amount of difference or variation between measurements. The 95% CI for the average difference is 0.31 to 1.26. This range provides an estimate of the range of possible values for the true mean difference in the population. This suggests that the true mean difference in TBUT measurements between observers is likely to fall within this range with 95% confidence.

Another study reported a poor relationship between symptoms and diagnostic test results in patients with DES by comparing symptoms with the Schirmer test.¹⁹ Subjects may have no symptoms other than eyestrain despite Schirmer test results below 5 mm or even zero. Due to the challenges in diagnosing and assessing DES, clinicians should evaluate eye history, symptoms and signs, and test results together.

Unlu et al performed three tests to confirm the diagnosis. OSDI results correlated well with TBUT results: 35% of the subjects were diagnosed with DES based on OSDI results and 42% of subjects based on TBUT results. The average time taken to perform the OSDI test was 74.11 ± 28.61 seconds and 215.14 ± 65.27 seconds for TBUT.²⁴ The average time spent performing both tests was significantly lower compared to the time taken to perform the Schirmer test. The results of this study suggest that the diagnosis of DED is more reliable and less time-consuming if OSDI and TBUT are performed compared to the Schirmer test. These data support previous research statements that both subjective and objective indicators should be used to improve the accuracy of DED diagnosis.^20,25

The use of mobile apps in DED diagnosis was discussed earlier in a recent research protocol by Nagino et al.²⁴ Their study use DEA01 smartphone apps (InnoJin, Inc., DEA01, Tokyo, Kanto, Japan) to measure subjective symptoms using J-OSDI and maximum blink interval (MBI) as test method for DED screening only.^5,26 The biggest differences and value compared to our study are we followed the standard method by combining TBUT and OSDI for DED diagnosis.^5,26,27 Moreover, SEC is a smartphone attachment device with personalized app that is able to record videos and store it in unlimited storage similar to “iCloud” (Apple, Inc, iCloud, Cupertino, California, United States of America) designed not only for screening but also for more advanced for DED diagnosis.

We suggest to combine test in addressing the diagnosis of DED is essential. In case of need, either indicator (more likely subjective symptoms) can be used only for screening to determine the need for further investigations such as the Schirmer test and TBUT, but not as a single tool in diagnosing DED.^5,26,27

SEC is a breakthrough invention in eye technology, and this is the second study in populations-based study using SEC to diagnose of DED after Japan. The limitations of this study are we only evaluated adult populations in Indonesia, for future research we suggest conducting a study with wider range of age and future research in other rural areas to get more generalized results in the population. Second, this study did not assess tear break-up patterns and meibomian gland function. We face the problem that meibomian gland function still needs to be improved to evaluate using SEC, although no previous study has ever reported evaluating MGD using SEC. We recommend improvements to the SEC tool for the MGD test, which may be beneficial and provide more supporting data in DED diagnosis and future studies. Thirdly, a previous study by Shimizu et al reported SEC validity compared to slit lamps in Japanese populations. However, we were not comparing TBUT examinations’ validity (sensitivity and specificity) of SEC in diagnosing DED with conventional slit lamps as standard examination devices in the Indonesian population. Therefore, we suggest measuring SEC validity in Indonesian populations for future research that might add more value to TBUT assessment for DED diagnosis using SEC.

This innovative technology has several advantages, such as being easy to perform, portable, and inexpensive. Not only for detecting DED but this test is also useful for evaluating complications of DED, such as corneal ulcers and their treatment response or progress. Aside from being a telemedicine solution, we suggest that in the future, this device can be used in the emergency room or in the ICU, where patients cannot undergo slit-lamp examination due to physical limitations or for uncooperative patients. As a portable recordable “slit lamp”, this tool can be assessed by general practitioners, ophthalmologists, or even ophthalmologists, and images can be transferred easily for expert interpretation if needed. The limitation of this tool is that some prototype devices are inherently placed on smartphones, and for now, it is only compatible with iPhone 7 models. So, we suggest refining the design to overcome this limitation.

Asia Dry Eye Society published new consensus in 2017 that stated the modern definitions of DED combine objective findings, subjective symptoms, and mechanistic considerations, among which hyperosmolarity and inflammation play a key role. The combination of symptoms and an unstable tear film (characterized by a short tear film -time) is considered sufficient for a definitive diagnosis of dry eye. This simplified definition implies that individuals experiencing dry eye with a short tear film break-up time should be treated as patients with dry eye disease (DED) rather than merely suspected cases of dry eye.⁵

Additionally, the study compared the diagnostic value of TBUT and OSDI in community screening for DED. However, no correlation was found between the reliability of DED diagnosis using OSDI scores and TBUT (P-value >0.05; Table 4). This suggests that while both TBUT and OSDI are commonly used in diagnosing DED, they may provide different information and should be considered complementary rather than interchangeable.

It is worth noting that the study focused on a specific population, which may limit the generalizability of the findings. Future studies with larger and more diverse samples would be beneficial to validate these results.

Overall, the study provides valuable insights into using SEC to diagnose DED and emphasizes interobserver reliability in clinical assessments. The findings contribute to the growing body of knowledge regarding technology usage in improving DED diagnosis and highlight the need for a comprehensive evaluation that considers multiple diagnostic measures for a more accurate assessment of the condition.

Conclusions

This study showed good agreement between observers in measuring TBUT using a SEC in Indonesia remote area. Statistical analyses such as mean difference, confidence interval, and Bland–Altman method or kappa coefficient were used to measure inter-observer reliability. The results showed that the mean difference value between observers was 0.78. The 95% confidence interval showed a range of values between 0.31 and 1.26. This indicates good consistency between observers in the measurement of TBUT using SEC. In addition, the obtained p-value of 0.0013 shows statistical significance, indicating that the difference between observers in TBUT measurement cannot be explained by chance alone. Thus, it can be concluded that the use of SEC in TBUT screening in Indonesian remote area has good interobserver reliability. This suggests that this technology is reliable in detecting the time of tear break-up and provides consistent results in the diagnosis of dry eye disease, and might be useful as an alternative to telemedicine especially in remote areas.