Assessing the Accuracy, Quality, and Readability of Patient Accessible Online Resources Regarding Ocular Gene Therapy and Voretigene Neparvovec

Introduction

Gene augmentation therapy is a technique in which genetic material is introduced into a patient’s cells to treat or prevent disease.¹ Gene therapy was introduced in human subjects in the early 1990s, and currently, it is being recognized as a promising treatment option for multiple diseases.² Several trials have been conducted or are ongoing to evaluate the efficacy and safety of gene therapy, including but not limited to hematopoietic stem cell based gene therapy for β-thalassemia major, liver-directed gene therapy for hemophilia B, and T-cell immunotherapy for acute lymphoblastic leukemia.^2,3

Gene therapy has been of particular interest in ophthalmology. It has been studied in years of experimentation to determine its use to treat several ocular conditions including neovascularization, retinitis pigmentosa, Stargardt disease, retinoblastoma, and more.⁴ In 2017, voretigene neparvovec (Luxturna, Spark Therapeutics, Philadelphia, PA, USA) became the third gene therapy to be approved by the Food and Drug Administration (FDA).⁵ Voretigene neparvovec was unanimously approved by the FDA for patients with the biallelic RPE65 gene mutation form of hereditary retinal dystrophy.⁵ In the pivotal study, patients with this inherited retinal dystrophy were treated with subretinal injections of voretigene neparvovec and showed improvement in their functional vision which persisted for at least 4 years.^6,7

With the approval of voretigene and the subsequent press coverage, patients with eye conditions are likely to learn about treatment options on the internet. Millions of Americans search health-related information and seek medical advice online every day.⁸ Previous research shows about 62% of Internet users search the internet for health information.⁸ The content acquired online influences patients’ health decisions.⁹ The content patients encounter would ideally be easy to comprehend. Most patients read at an eighth grade level; however, most sources of medical information are written at a tenth grade level or higher.¹⁰ Low health literacy has been associated with poor medication adherence and adverse outcomes such as uncontrolled chronic disease and increased hospitalizations.¹¹ For these reasons, it is critical that the information patients obtain is accurate, complete, and easy to understand. As such, the goal of this study is to assess the accuracy, quality, and readability of online information regarding ocular applications of this gene therapy.

Methods

Article Selection and Content Analysis

The keywords “eye gene therapy” or “voretigene neparvovec” or “Luxturna” were entered in Google.com and in major medical websites such as the American Academy of Ophthalmology (AAO) website, and relevant articles were selected for analysis from the top search results with exclusion of similar results from the same organization. Preference was given to articles of formal as well as commercial organizations trusted by ophthalmologists and those believed to be regularly accessed by patients such as Wikipedia. The chosen articles included ones from the AAO,¹² American Optometric Association (AOA),¹³ EyeWiki,¹⁴ US Food and Drug Administration (FDA),¹⁵ Spark Therapeutics,¹⁶ MedicineNet,¹⁷ National Eye Institute (NEI),¹⁸ Novartis,¹⁹ WebMD,²⁰ and Wikipedia.²¹ A twenty-five question grading tool was designed by two vitreoretinal surgeons (JS and AEK) to assess the accuracy and quality of patient-relevant information in each article (Table 1). The questions were chosen to match presumed typical questions a patient or patient family member would ask and potential topic discussion when considering treatment with voretigene neparvovec. The evaluation of each website was completed independently by two vitreoretinal surgeons (JS and AEK) and one vitreoretinal surgery fellow (NY). A grading scale from 0–4, with 4 as a maximum, was utilized to evaluate each question as detailed in Table 1. A score of 0 was given if there was no information provided regarding the question. A score of 1 indicated the response was unclear, had inaccurate information, omitted significant information, and was very poorly organized. A score of 2 was for partial answers that somewhat addressed the concept but had gaps in the information and organization. A score of 3 meant the article provided essential elements to answer the question, addressed the most relevant points, and was focus and organized. A maximum score of 4 indicated the answer was accurate and thorough; the article provided a clear answer that was well explained, focused, and organized. Each article was independently graded by each observer, and interobserver reproducibility was assessed with a Spearman correlation. The average score between the three graders was used to compare the quality of the articles.

Table 1 Questionnaire to Assess Online Resources Describing Ocular Gene Therapy with Mean and Total Scores

The articles were further evaluated by two graders (JS and NY) using the DISCERN instrument. DISCERN is widely used to determine the quality of health information provided to patients.²² Each article was independently graded using the 16-question tool. Each question is rated using a 5-point scale ranging from “No” to “Yes.” A score of 1 is given if the answer to the question is “No.” A score of 5 is given if the answer is a definite “Yes.” Scores of 2 through 4 are given if the article meets the criterion of the question partially. The interobserver reproducibility was determined with a Spearman correlation. The average score between the two graders was used to compare the quality of the articles.

Accountability Analysis

The accountability of each article was evaluated using the Journal of the American Medical Association (JAMA) benchmarks.²³ Each article was assessed for the 4 standards: Authorship, attributions, disclosure, and currency. Each article should include the authors and contributors along with their affiliations and relevant credentials to meet the authorship requirements outlined by JAMA. Attributions, or references, should be reported, and disclosures and currency, or date of update, should be specified.

Readability Analysis

The online tool, Readable, was used to analyze readability of the articles.²⁴ Each article was assessed using the Flesch Reading Ease Score (FRE), Flesch Kincaid Grade Level, Gunning Fox Index, Coleman Liau Index, and Simple Measure of Gobbledygook (SMOG) Index. The Flesch Reading Ease Score awards each article a score from 0–100 based on total words, sentences, and syllables in the text. Higher scores translate to better readability. A score between 70 to 80 is comparable to an eighth-grade reading level. The Flesch Kincaid Grade Level, Gunning Fox Index, Coleman Liau Index, and SMOG Index are separate readability formulas which report the reading level required to understand the text. Each index states the US grade level of education reflected by the article. The mean score of these indices corresponds to the numerical US grade level. Therefore, a mean score of 12 indicates a 12th grade reading level while a score above 12 corresponds to a college reading level.

Statistical Analysis

Statistical analysis was completed using IBM SPSS Statistics for Mac, version 25.0, released 2017 (Armonk, NY: IBM Corp). Content analysis using the 25-question assessment and the DISCERN instrument was performed by treating the data as ordinal variables and analyzing with the Kruskal–Wallis test. A post-hoc Dunn-Bonferroni test was used to determine pairwise comparisons. Statistical significance was set at P≤0.05 for the main comparisons and the pairwise comparisons. The readability analysis was conducted using a Kruskal–Wallis test to compare the mean reading grade level for each article. A post-hoc Dunn-Bonferroni test was used to determine pairwise comparisons. A Spearman correlation test was carried out to evaluate correlation between accuracy and readability. Statistical significance was set at P≤0.05 for the main comparisons, Spearman correlation, and pairwise comparisons.

Results

Article Selection and Content Analysis

Ten articles were analyzed for the study. The inter-observer reproducibility was statistically significant between JS and AEK (r=0.66, P=0.038) and between JS and NY (r=0.84, P=0.003) and approached statistical significance between AEK and NY (r=0.62, P=0.058). The average questionnaire score for all the articles was 33.93 (SD 11.21, CI 95% ±6.95) out of 100 possible points. There was a statistically significant difference in the content accuracy and quality between the articles (P=0.017). The top scoring article was EyeWiki with an average score of 51.33 points. MedicineNet was the lowest scoring article with an average score of 16.67 points (Table 2). There was a statistically significant difference in the articles’ scores on each of the 25 questions (P<0.001).

Table 2 Mean and Total Scores for Grading Online Resources Describing Ocular Gene Therapy

For the DISCERN instrument, the inter-observer reproducibility was statistically significant between the two graders (r=0.89, P<0.001). The average score for all of the articles with this grading tool was 45.20 (SD 8.40, CI 95% ±5.21) out of 80 possible points. Results of the instrument also showed a statistically significant difference in the content quality between the articles (P=0.043), with EyeWiki scoring highest with an average score of 60.00 points and MedicineNet scoring lowest with 30.00 points (Table 3). There was a statistically significant correlation between the articles’ scores on the 25-question grading scale and on the DISCERN instrument (r=0.90, P<0.001).

Table 3 Mean and Total Scores for Online Sources Describing Ocular Gene Therapy Graded Using the DISCERN Instrument

Accountability Analysis

None of the articles achieved all 4 JAMA benchmarks. One of the 10 (10%) articles, EyeWiki, achieved 3 of the 4 JAMA benchmarks (Table 4). Currency was the most displayed benchmark followed by attribution and authorship. There was no correlation between the accuracy of the articles and JAMA benchmarks (r=−0.229, P=0.524).

Table 4 JAMA Benchmarks Achieved by Online Sources Regarding Ocular Gene Therapy

Readability Analysis

The mean Flesch Reading Ease Score for all the articles was 39.51 (SD 9.98, CI 95% ±6.19). The mean reading grade for all websites was 12.88 (SD 1.93, CI 95% ±1.19). There was no significant correlation between the FRE score and mean reading grade level (r=−0.340, P=0.336). There was a significant difference between the mean reading grade level of the websites (P=0.001). Wikipedia had the lowest mean reading grade level of 9.63, and the FDA website had the highest mean reading grade level of 16.73. Wikipedia (P=0.005) and the Spark Therapeutics product website (P=0.015) had significantly lower reading levels than the FDA website (Table 5). There was no significant correlation between website accuracy and the mean reading grade (r=−0.444, p=0.199).

Table 5 Readability Analysis of Online Sources Describing Ocular Gene Therapy

Discussion

The results (Table 6) demonstrated that there was significant variation in the quality, accuracy, and readability of the information available to patients about the ocular gene therapy voretigene neparvovec. EyeWiki provided the most complete and accurate information of the articles analyzed and achieved the most benchmarks of all the articles fulfilling 3 of the 4 benchmarks. However, EyeWiki had a mean reading grade level of 12.85, and is difficult to read for a layperson, as it is primarily designed for ocular provider education rather than patient consumption. Still, given that it is freely available and accessible by the public, it was included in the analysis.

Table 6 Summary Table of Grading Indices Used to Assess Online Sources Describing Ocular Gene Therapy

The second-highest scoring resource on both grading tools was the Spark Therapeutics patient informational website regarding voretigene neparvovec; it was the second most readable article as well with a mean reading grade level of 10.40. Nevertheless, it scored only 54.50 out of 80 points on the DISCERN grading tool and below 50 out of 100 points on average on the 25-question assessment. It did not achieve any of the JAMA benchmarks; moreover, despite being the highest scoring resource designed for reading by patients, it is not an ideal source of reference given the potential inherent conflict of interest present in the manufacturing company’s discussion of its own product.²⁵

In this study, the information available from formal organizations such as the FDA, AAO, and NEI was of overall low quality. Ideally, these major organizations would have accurate and complete information that is easy to understand as many patients rely on these institutions for medical knowledge. Prior studies have shown that educated patients have greater trust in federal agencies and community organizations as sources of medical information.²⁶ Overall, patients prefer non-commercial websites especially from established organizations such as the FDA and NEI over other sources.^27,28 The FDA website received the highest content and DISCERN average score of these three resources; however, it had a mean reading grade level of 16.73, the highest level of all the articles. Although the AAO and NEI websites were easier to read with mean reading levels of 11.45 and 13.03, respectively, both resources scored poorly on both scales. Furthermore, the AAO and FDA websites only achieved 1 of the 4 JAMA benchmarks while NEI achieved 2 of the 4 benchmarks demonstrating the overall poor accountability of the articles, although one could argue that the organization as a whole may be taking accountability for the content, which would not be identified by the JAMA benchmark scoring.

This study was limited in several aspects. The 25-question assessment was based on two retina providers' experience and assumptions regarding typical patient questions and was not validated in a patient population. Furthermore, the inter-observer correlation between two pairs of the graders resulted in moderate strength of correlation demonstrating variability in interpretation of the articles or of the grading scale. To further test the online resources being evaluated, the standardized DISCERN instrument was also incorporated, and there was a significant and strong positive correlation between the DISCERN scores and the content assessment scores. Inter-observer reproducibility had greater strength in the DISCERN instrument as well. Patients may use search engines or keywords that vary from the ones used to determine websites included in this study. Some resources included in this study were found beyond the second results page on Google, but patients often only use the first two results pages for their research.²⁸ These resources were still included given their importance as prominent, public organizations (eg NEI, FDA). The resources evaluated were non-peer reviewed sources of varied websites including commercial, government, and non-government organizations. These articles were chosen as they were readily available during search, but patients may prefer to read peer reviewed sources or information from other organizations. Resources were not directly assessed by patients; future studies could further study online resource quality and content by directly evaluating patients’ comprehension.

In conclusion, this study demonstrated that patient accessible online resources regarding the ocular gene therapy voretigene neparvovec are generally incomplete, difficult to read, and of poor quality. The available material varied significantly by source and did not provide adequate information to aid patients in their medical decisions. This indicates an unfilled need to create patient-friendly online content regarding this ocular gene therapy as available research and therapeutic options continue to evolve. Future articles can be improved by assuring accuracy of the information using peer reviewed research papers as reference and by maintaining a reading level sufficient for patients’ understanding.