Analysis of the Readability and Accountability of Online Patient Education Materials Related to Glaucoma Diagnosis and Treatment

Introduction

Glaucoma is the leading cause of irreversible vision loss worldwide.¹ Current estimates indicate that approximately three million Americans are living with glaucoma, with 50% of glaucoma patients unaware that they have the disease.² Glaucoma disease progression can lead to vision loss, reducing a patient’s quality of life.³ Glaucoma disease progression can be slowed with medication therapy.^4–6 However, adherence to medication treatment regimens among glaucoma patients is low.⁷ Although several factors have been identified as likely contributors to poor medical adherence, reduced health literacy among glaucoma patients is a modifiable risk factor that can be targeted in hopes of impacting disease progression.^8,9

After receiving a glaucoma diagnosis, patients often turn to the internet to learn more about their condition.¹⁰ The use of the internet as a source of online health information has expanded drastically in the past decade, with approximately 75% of people in the United States using the internet to search for health-related information.¹¹ However, in order for online patient education materials to effectively convey information, they should be 1) written at a level that is understandable by their intended audience and 2) accurate. Readability and accountability are metrics that can be used to assess whether online patient education materials satisfy the two aforementioned conditions. Readability is a measure of how difficult it is to read a given piece of text. According to the American Medical Association (AMA), online patient education materials should be written at or below a 6th grade reading level.¹² Accountability can be assessed using JAMA benchmarks (scale 0–4), which dictate that a website containing patient education materials should (1) include all authors and their credentials, (2) list references, (3) provide disclosures, and (4) provide date of last update.¹³

Prior studies examining the readability of online patient education materials in many ophthalmology subspecialties including pediatrics, retina, and cornea have suggested that patient education materials are written at an age level that is higher than what is recommended by the AMA.^14–16 Recently, four studies have suggested that online patient education materials that populate after a Google search using the word “glaucoma” are also frequently written at an age-inappropriate level.^17–20 However, after being given a glaucoma diagnosis, patients may elect to search for online information related to their specific type of glaucoma (eg open-angle glaucoma, angle-closure glaucoma) and treatments (eg trabeculoplasty, trabeculectomy) recommended by their ophthalmologist.¹⁰ To the best of our knowledge, there has not been a study to examine the readability and accountability of online patient education materials that populate when searching Google regarding various glaucoma types and glaucoma treatments using search parameters beyond the one-word search “glaucoma”. Furthermore, it is unclear which sources of information (ie national organizations such as the American Academy of Ophthalmology vs Crowdsourced vs Educational Institutions vs Private Practice/Independent User) are populating most frequently after glaucoma-related Google searches, and whether the source of information is associated with greater readability and accountability metrics. As such, the purpose of our study is (1) to evaluate the readability and accountability of online patient education materials that populate after executing a Google search using search terms associated with glaucoma diagnosis and treatments and (2) to evaluate the source of information of common glaucoma online patient education materials, and to determine whether information source is impacting readability and accountability metrics.

Methods

Search Term Selection and Website Inclusion

We conducted an internet search query using Google (Google, Inc., Mountain View, CA) for 10 search terms related to glaucoma diagnosis and 10 search terms related to glaucoma treatment. The internet search query took place in August 2021. We avoided bias from prior search history and geographically targeted search results by performing the Google search on an Incognito browser with location filters, advertisements, and sponsored results disabled. Search terms were selected using the “related queries” feature of the Google Trends tool to discern the most popular search terms associated with both glaucoma diagnosis and treatment.²¹ The 10 search terms associated with glaucoma diagnosis were the following: “glaucoma”, “high eye pressure”, “ocular hypertension”, “open-angle glaucoma”, “angle-closure glaucoma”, “normal-tension glaucoma”, “secondary glaucoma”, “congenital glaucoma”, “pigmentary glaucoma”, and “traumatic glaucoma.” The 10 search terms associated with glaucoma treatment were the following: “glaucoma surgery”, “minimally-invasive glaucoma surgery”, “Istent implantation glaucoma”, “hydrus microstent glaucoma”, “xen gel stent glaucoma”, “trabeculoplasty”, “trabeculectomy”, “glaucoma drainage implant”, “tube-shunt surgery”, and “canaloplasty”.

Prior research reveals that internet users rarely browse past the first page of search results following a Google search.²² Therefore, we evaluated the readability and accountability for the first 10 articles containing patient education materials for each of the 20 search terms studied, for a total of 200 websites analyzed. We excluded peer-reviewed scientific articles, dictionary definitions, and patient forum posts from our analyses in order to limit to websites containing patient education materials.

Readability Analysis

All 200 articles meeting inclusion criteria were evaluated for readability using five validated readability assessments: Flesch Reading Ease (FRE), Gunning Fog Index (GFI), Flesch-Kincaid Grade Level (FKGL), Simple Measure of Gobbledygook (SMOG), and New Dale-Chall (NDC). The FRE scale measures readability by generating a score from 0 to 100 (0–30: very difficult, 30–50: difficult, 50–60: fairly difficult, 60–70: standard, 70–80: fairly easy, 80–90: easy, 90–100: very easy). Each of the four remaining readability scales provide a “grade-level” at which the article was written. For example, a score of 11 indicates that the article was written at an 11th grade reading level. An average grade level for each article was then calculated from the GFI, FKGL, SMOG, and NDC indices.

JAMA Accountability Analysis

All 200 articles which met the inclusion criteria to be considered in our study were evaluated for accountability (score of 0–4) using JAMA benchmarks by SC.¹³ According to JAMA guidelines, a website containing patient education materials should (1) include all authors and their relevant credentials, (2) list references, (3) provide disclosures and (4) provide date of last update.

Source of Information Subgroup Analyses

In order to assess differences in readability and accountability of online patient education materials related to glaucoma diagnosis and treatment based on the information source, articles were placed into one of 4 categories: (1) educational institution including academic medical centers, (2) private practice or independent user, (3) crowdsourced reference (such as Wikipedia), and (4) official patient education materials published by a national organization (such as the American Academy of Ophthalmology [AAO], American Optometric Association [AOA], National Eye Institute [NEI], and American Glaucoma Society [AGS]).

Statistical Analysis

We used one-way ANOVA tests confirmed by Tukey’s HSD post hoc analyses to assess differences in the readability and accessibility of online patient education materials related to both glaucoma diagnosis and glaucoma treatment. Independent sample t-tests were used to assess differences in readability and accountability between articles related to glaucoma diagnosis vs glaucoma treatment, with statistical significance established at p < 0.05.

Results

Readability and Accountability of Patient Education Materials Related to Glaucoma Diagnosis

The average FRE for the 100 glaucoma diagnosis-related articles analyzed was 42.02 (standard deviation 1.08). The average grade level at which articles were written was 11.06 (standard deviation 1.30). Only 13% of articles (13/100) were written at or below a 6th grade reading level according to at least one of the four validated grade-level readability tools. The average JAMA accountability score for all articles analyzed was 1.29 (standard deviation 1.15). Common JAMA accountability metrics included information about date of last update (52% of articles), website authorship (40%), references (37%) and disclosures (2%).

When comparing the readability and accountability of the 10 articles analyzed for each search term, there were no significant differences in average grade level (F_9,90 = 1.299, p = 0.2482) or accountability (F_9,90 = 0.8684, p = 0.5530) (Table 1).

Table 1 Readability and Accountability of Online Patient Education Materials Related to Glaucoma Diagnosis by Search Term and Information Source

Of the 100 articles analyzed related to glaucoma diagnosis, 17 were from crowdsourced websites, 20 were from educational institutions, 47 were from private practices/independent users, and 16 were from national organizations. Significant differences in FRE (F_3,96 = 5.519, p = 0.0015) and average grade level (F_3,96 = 6.226, p < 0.0001) were observed when subdividing articles related to glaucoma diagnosis based on source. With regard to FRE, crowdsourced articles were written at a higher level (FRE = 33.57) than articles from both educational institutions (FRE = 47.03, p < 0.05) and private practice/independent authors (FRE = 45.07, p < 0.05). With regard to average grade level, crowdsourced articles were written at a higher grade level than articles from educational institutions (p < 0.01), private practice/independent authors (p < 0.01), and national organizations (p < 0.05).

Significant differences in accountability were also observed when subdividing articles related to glaucoma diagnosis by source (F_3,96 = 5.239, p = 0.0022). Crowdsourced articles (p < 0.01) and articles from national organizations (p < 0.05) both had higher accountability than articles written by educational institutions.

Readability and Accountability of Patient Education Materials Related to Glaucoma Treatment

The average FRE for the 100 glaucoma treatment-related articles analyzed was 43.86 (standard deviation 1.01). The average grade level at which articles were written was 11.18 (standard deviation 1.54). Of the 100 articles analyzed, only 9% (9/100) were written at or below a 6th grade reading level according to at least one of the four validated grade-level readability tools. The average JAMA accountability score for all articles analyzed was 0.95 (standard deviation 1.15). Common JAMA accountability metrics included information about date of last update (36% of articles), website authorship (33%), and references (29%).

When comparing the readability and accountability of the 10 articles analyzed for each search term related to glaucoma treatment, there were no significant differences in average grade level (F_9,90 = 1.536, p = 0.1473) or accountability (F_9,90 = 1.2489, p = 0.2760) (Table 2).

Table 2 Readability and Accountability of Online Patient Education Materials Related to Glaucoma Treatment by Search Term and Information Source

Of the 100 articles analyzed related to glaucoma treatment, 13 were from crowdsourced websites, 18 were from educational institutions, 64 were from private practices/independent users, and 5 were from national organizations. Significant differences in FRE (F_3,96 = 4.8545, p = 0.0035) and average grade level (F_3,96 = 6.4568, p = 0.0005) were observed when subdividing articles related to glaucoma diagnosis based on source. With regard to FRE, crowdsourced articles were written at a higher level (FRE = 35.46) than articles from both educational institutions (FRE = 48.30, p < 0.01) and private practice/independent authors (FRE = 44.49, p < 0.05). With regard to average grade level, crowdsourced articles (p < 0.01) and articles written by national organizations (p < 0.01) were both written at a higher grade level than articles from educational institutions (Table 2).

Significant differences in accountability were also observed when subdividing articles related to glaucoma treatment by source (F_3,96 = 10.1756, p < 0.0001). Crowdsourced articles (p < 0.01) and articles written by national organizations (p < 0.01) both had higher accountability than articles written by educational institutions and private practice/independent users (Table 2).

Comparing the Readability and Accountability of Articles Related to Glaucoma Diagnosis vs Treatment

There was no significant difference in grade level (p = 0.1149) or FRE (p = 0.2828) when comparing articles related to glaucoma diagnosis vs articles related to glaucoma treatment. However, articles related to glaucoma diagnosis (average accountability score = 1.29) demonstrated higher JAMA accountability than articles related to glaucoma treatment (average accountability score = 0.95, p = 0.0418) (Table 3).

Table 3 Comparing Readability and Accountability of Online Patient Education Materials Related to Glaucoma Diagnosis vs Glaucoma Treatment

Discussion

With patients increasingly turning to the internet to obtain health information, it is imperative that online patient education materials are accurate and understandable.²³ Our findings reveal that online patient education materials related to both glaucoma diagnosis and glaucoma treatment are written, on average, at the 11th grade level, which substantially exceeds AMA recommendations for articles at or below 6th grade reading level.¹² Furthermore, online information related to glaucoma treatment and diagnosis often lacks JAMA accountability metrics intended to improve the quality of information that patients receive online, with only 18% of websites analyzed (35/200) containing either 3 or 4 of the JAMA accountability metrics evaluated. Improving the readability and accountability of online patient education materials related to glaucoma diagnosis and treatment can help to enhance patient understanding of the disease, which may influence adherence to medication treatment regimens and subsequent clinical outcomes observed.

Our findings align with other recent studies that have examined the readability of online patient education materials in several ophthalmology subspecialties. In 2013, a study that evaluated the readability of online patient education materials related to 16 ophthalmology diagnoses also concluded that patient education materials were written at a level that was greater than what the average patient could comprehend—and suggested the use of readability indices in the future when producing patient education materials.²⁴ However, in addition to our findings regarding patient education materials related to glaucoma diagnosis and treatment, recent studies examining the readability of online patient education materials related to common retinal diseases, pediatric diseases, and cataracts have all concluded that patient education materials were written at a level that may not be understandable by the average patient, suggesting that minimal improvements to readability have been made over the past nine years.^14–16

Our results also build on recent literature evaluating the readability of patient online education materials related to glaucoma. Three prior studies published in the past year have analyzed the first 13, 15, and 30 Google search results that arise when typing the word “glaucoma” into the Google search engine.^17–19 Our analysis expands upon the conclusions of these studies by evaluating a total of 200 websites encompassing more detailed glaucoma diagnosis subtypes and available treatments, which may be accessed by patients. When presented with a glaucoma diagnosis, patients are often provided information about the type of glaucoma they have been diagnosed with as well as potential treatment options. Our analyses indicate that when searching for several of the most common types of glaucoma (eg, open-angle glaucoma, angle-closure glaucoma, normal tension glaucoma) and glaucoma treatments (eg, glaucoma surgery, trabeculoplasty, trabeculectomy), patients are often faced with search results that are written at a reading level above that of many patients—presenting a challenge to efforts to improve health awareness and treatment adherence. These results held true when the Google searches involved both colloquial (eg “high eye pressure” and “glaucoma surgery”) and technical (eg “ocular hypertension” and “trabeculoplasty”) terms.

We also sought to expand on prior studies by evaluating the source of the information that patients often encounter when searching for information related to their glaucoma diagnosis and treatment online, and to determine whether some sources produced more readable and accountable online patient education materials. Of the 100 articles analyzed after a Google search related to glaucoma diagnosis, we found that 47% were maintained by private practices/independent users, whereas only 20% were maintained by educational institutions and 16% were from national organizations.

Patients searching for online information about their health are often forced to sift through misinformation that they encounter related to their diagnosis.²⁵ Therefore, connecting patients to resources from established organizations such as the AAO, AOA, NEI, and AGS could conceivably help to ensure that the information that patients encounter is accurate. However, we found that websites from established national organizations providing patient education materials related to glaucoma diagnoses and treatment are rarely encountered after Google search, compared with other sources of information—particularly websites maintained by private practices/independent users which often may not have been vetted for accurate information.

The source of information for online patient education materials related to glaucoma treatment was even further skewed toward private practice/independent users, with the majority (64%) of websites maintained by private practices/independent users and relatively few websites maintained by educational institutions (18%) and national organizations (5%). A concerted effort by national organizations such as the AAO, AOA, NEI, and/or AGS to not only improve the readability and accountability of their online patient education materials but also to invest in search engine optimization strategies could increase the likelihood of their materials being accessed by patients searching for glaucoma-related information online.

Furthermore, regardless of the source of information posted, JAMA accountability metrics were frequently missing from online patient education materials, with accountability scores for websites related to glaucoma diagnosis and treatment averaging 1.29 and 0.95 out of 4 JAMA benchmarks, respectively. It is interesting to note that Crowdsourced patient education materials averaged the highest JAMA benchmarks. This is likely because of the fluid nature of Crowdsourced materials, which allow multiple authors to freely contribute information. As such, it is standard for Crowdsourced materials to provide information on date of last update as well as references used, so that other authors may “fact check” prior authors’ claims. However, in aggregate, online patient education materials rarely provided a list of authors, references or disclosed conflicts of interests. It is possible that online patient education materials from educational institutions or private practices are written by more than one author, potentially preventing such organizations from attributing all material on the website to one person; however, similar to published scientific literature, providing a complete list of authors as well as additional acknowledgements could help to solve this problem and encourage more transparency with regard to authorship online. It is also possible that educational institutions and other private practices who post online patient education materials do not feel the need to provide references because they do not anticipate that patients with limited medical knowledge would be able to interpret primary literature, even if sources are cited. It is possible that this is true, with patients’ average literacy levels at approximately the sixth-grade level; however, it is important that proper references are provided in case patients are seeking more information about their condition.

There are several implications to our findings that online patient education materials are often written above recommended grade levels and lacking JAMA accountability metrics. After receiving a glaucoma diagnosis, patients are often asked to adhere to medication regimens and follow-up appointment schedules, which benefit from patients’ active participation and a basic understanding of potential disease outcomes. Patient education materials not written at readable grade level may contribute to worse clinical outcomes and potentially exacerbate the ethnic disparities that have been reported in glaucoma management.^26–28 Previous research indicates that Black and Hispanic patients have, on average, higher rates of glaucoma diagnosis and lower levels of health literacy.^28–30 As such, poor readability of online patient education materials related to online information for glaucoma diagnosis and treatment may compound existing racial and ethnic health disparities in glaucoma care and outcomes. A previous study aimed at improving glaucoma medication adherence revealed that patients with low health literacy skills benefitted from educational efforts that were tailored to their individual health literacy level.³¹ Future efforts to improve the readability of online patient education materials for glaucoma diagnosis and treatment would benefit from acknowledging variation in health literacy levels. Patient education materials developed with this in mind could better meet the needs of populations at higher risk for more aggressive disease. Individualizing patient education materials rather than adopting a “one size fits all” approach could help to mitigate the racial/ethnic disparities that are observed in glaucoma patient care and outcomes.^28–30

Our analyses of 200 websites related to glaucoma diagnosis and treatment also demonstrate the need for more accessible and accountable patient education materials. Approximately half of patients report searching the Internet for health information prior to visiting their doctor, and more than half use online information after their visit to confirm what they have discussed in the patient encounter.^11,32 Our results indicate that the websites which populate following a Google search for many search terms associated with glaucoma are often written at a level that may not be easily understood by a measurable percentage of the population. Furthermore, patients are more likely to encounter websites maintained by private practices/independent users and are less likely to receive their information from national organizations and educational institutions, which may contain more accurate or standardized information.³³ In the future, ophthalmologists creating patient education materials should consider implementing readability indices before finalizing resources, in order to ensure that education materials are easily comprehensible by patients with varying health literacy levels. A prior study from Williams et al also recommends other aspects of online patient education materials that should be addressed in order to benefit patients, including adequate white space, use of headers, an active voice, and images or illustrations.³⁴ Furthermore, respected national organizations such as the American Academy of Ophthalmology that seek to disseminate accurate health information to patients interested in their eye care may consider greater utilization of alternative distribution methods such as social media to expand their reach, with Google search engine optimization currently favoring websites maintained by private practices/independent users rather than promoting national organization guidelines.

There are limitations to our study. First, while we selected 20 search terms to represent common Google search queries related to glaucoma diagnosis and treatment, our list is not exhaustive, and there are other search terms associated with glaucoma diagnosis and treatment not included in our study. The use of patient-based focused groups to more specifically identify search terms related to glaucoma diagnosis and treatment would help to ensure that the search terms of interest are indicative of what patients are searching for online. We also only assessed written, rather than video, forms of online patient education materials. Next, the websites analyzed in this study were populated after a Google search. Although Google does account for the large majority (more than 90%) of search engine requests in the United States, web searches related to glaucoma diagnosis and treatment on other search engines may not be accounted for in this study.³⁵ In addition, the algorithm that Google uses to populate websites after a search is executed is unknown. In order to avoid bias from prior search history and geographically targeted search results, we executed all Google searches on an Incognito browser with location filters, advertisements, and sponsored results disabled. However, it is possible that Google search algorithms may have impacted the list of websites populated after search execution, if the algorithms account for other features. Furthermore, the generalizability of our findings may be limited by factors such as access to the internet and engagement in social media. Populations particularly vulnerable to low health literacy and needing more accessible education resources may also be more likely to have limited access to internet or social media resources. Next, although we used five validated tools to measure readability for each website analyzed, readability is not a direct measure of understandability. While readability of website content is often used as a gauge for how likely it is that website content is understandable, there may be circumstances when a readable website is not understandable, and vice versa. However, the readability tools utilized in this study have been used in several prior studies measuring readability of online patient education materials in ophthalmology and likely provide insight regarding overall understandability of website content.^14–18,24 Additionally, although we assessed patient education materials for readability and accountability, we did not specifically assess whether the content presented was accurate. However, increased JAMA accountability metrics would likely result in more accurate information presented. Finally, initial search of websites occurred in August 2021. It is possible that website content has been updated between August 2021 and the time of publication.

Conclusions

In summary, despite variation in readability and accountability of online patient education materials related to glaucoma diagnosis and treatment based on information source, materials from all sources are often written above the recommended reading level and lack JAMA accountability metrics. Our findings demonstrate a need for accurate, understandable online information on glaucoma diagnosis and treatment that patients can use to understand their disease and prognosis and to inform their eye care decisions.