Assessing the Accuracy, Quality, and Readability of Patient Accessible Online Resources Regarding Ocular Gene Therapy and Voretigene Neparvovec
Received 12 June 2021
Accepted for publication 17 August 2021
Published 17 September 2021 Volume 2021:15 Pages 3849—3857
Checked for plagiarism Yes
Review by Single anonymous peer review
Peer reviewer comments 4
Editor who approved publication: Dr Scott Fraser
Swetha Davuluri,1 Nicolas A Yannuzzi,1 Amy Kloosterboer,1 Ajay E Kuriyan,2 Jayanth Sridhar1
1Department of Ophthalmology, Bascom Palmer Eye Institute, University of Miami Miller School of Medicine, Miami, FL, USA; 2Department of Retina Service, Mid-Atlantic Retina, Wills Eye Hospital, Thomas Jefferson University, Philadelphia, PA, USA
Correspondence: Jayanth Sridhar
Department of Ophthalmology, Bascom Palmer Eye Institute, University of Miami, Miller School of Medicine, 900 NW 17th Street, Miami, FL, 33136, USA
Tel +1 3053266124
Fax +1 3053266417
Email [email protected]
Purpose: To evaluate the accuracy, quality, and readability of online information regarding the Food and Drug Administration (FDA) approved ocular gene therapy voretigene neparvovec (Luxturna, Spark Therapeutics, Philadelphia, PA, USA).
Methods: Ten online resources about voretigene neparvovec were assessed in this cross-sectional study. A novel 25-question assessment was created to evaluate the information most relevant to patients. Each article was assessed by independent graders using the assessment and the DISCERN instrument. An online readability tool, Readable, was used to assess readability. Accountability was evaluated using the Journal of the American Medical Association (JAMA) benchmarks.
Results: The average questionnaire score for all the articles was 33.93 (SD 11.21, CI 95% ± 6.95) out of 100 possible points with significant variation in the content accuracy and quality between the articles (P=0.017). EyeWiki achieved the highest score and MedicineNet the lowest. The mean reading grade for all articles was 12.88 (SD 1.93, CI 95% ± 1.19) with significant variation between articles (P=0.001). Wikipedia was the most readable, and the FDA website was the least. None of the articles achieved all four JAMA benchmarks, and only one of the ten articles, EyeWiki, achieved three of the four JAMA benchmarks.
Conclusion: The information available online regarding this FDA-approved ocular gene therapy is generally of low quality, above the average reading level of the general population, and varies significantly between sources. The articles provide incomplete information that is not entirely accurate or easy to read, and as a result, the material would not support patients adequately in their medical decisions and questions about this new therapeutic option.
Keywords: gene therapy, patient education, vitreoretinal surgery, leber congenital amaurosis, voretigene neparvovec
Gene augmentation therapy is a technique in which genetic material is introduced into a patient’s cells to treat or prevent disease.1 Gene therapy was introduced in human subjects in the early 1990s, and currently, it is being recognized as a promising treatment option for multiple diseases.2 Several trials have been conducted or are ongoing to evaluate the efficacy and safety of gene therapy, including but not limited to hematopoietic stem cell based gene therapy for β-thalassemia major, liver-directed gene therapy for hemophilia B, and T-cell immunotherapy for acute lymphoblastic leukemia.2,3
Gene therapy has been of particular interest in ophthalmology. It has been studied in years of experimentation to determine its use to treat several ocular conditions including neovascularization, retinitis pigmentosa, Stargardt disease, retinoblastoma, and more.4 In 2017, voretigene neparvovec (Luxturna, Spark Therapeutics, Philadelphia, PA, USA) became the third gene therapy to be approved by the Food and Drug Administration (FDA).5 Voretigene neparvovec was unanimously approved by the FDA for patients with the biallelic RPE65 gene mutation form of hereditary retinal dystrophy.5 In the pivotal study, patients with this inherited retinal dystrophy were treated with subretinal injections of voretigene neparvovec and showed improvement in their functional vision which persisted for at least 4 years.6,7
With the approval of voretigene and the subsequent press coverage, patients with eye conditions are likely to learn about treatment options on the internet. Millions of Americans search health-related information and seek medical advice online every day.8 Previous research shows about 62% of Internet users search the internet for health information.8 The content acquired online influences patients’ health decisions.9 The content patients encounter would ideally be easy to comprehend. Most patients read at an eighth grade level; however, most sources of medical information are written at a tenth grade level or higher.10 Low health literacy has been associated with poor medication adherence and adverse outcomes such as uncontrolled chronic disease and increased hospitalizations.11 For these reasons, it is critical that the information patients obtain is accurate, complete, and easy to understand. As such, the goal of this study is to assess the accuracy, quality, and readability of online information regarding ocular applications of this gene therapy.
Article Selection and Content Analysis
The keywords “eye gene therapy” or “voretigene neparvovec” or “Luxturna” were entered in Google.com and in major medical websites such as the American Academy of Ophthalmology (AAO) website, and relevant articles were selected for analysis from the top search results with exclusion of similar results from the same organization. Preference was given to articles of formal as well as commercial organizations trusted by ophthalmologists and those believed to be regularly accessed by patients such as Wikipedia. The chosen articles included ones from the AAO,12 American Optometric Association (AOA),13 EyeWiki,14 US Food and Drug Administration (FDA),15 Spark Therapeutics,16 MedicineNet,17 National Eye Institute (NEI),18 Novartis,19 WebMD,20 and Wikipedia.21 A twenty-five question grading tool was designed by two vitreoretinal surgeons (JS and AEK) to assess the accuracy and quality of patient-relevant information in each article (Table 1). The questions were chosen to match presumed typical questions a patient or patient family member would ask and potential topic discussion when considering treatment with voretigene neparvovec. The evaluation of each website was completed independently by two vitreoretinal surgeons (JS and AEK) and one vitreoretinal surgery fellow (NY). A grading scale from 0–4, with 4 as a maximum, was utilized to evaluate each question as detailed in Table 1. A score of 0 was given if there was no information provided regarding the question. A score of 1 indicated the response was unclear, had inaccurate information, omitted significant information, and was very poorly organized. A score of 2 was for partial answers that somewhat addressed the concept but had gaps in the information and organization. A score of 3 meant the article provided essential elements to answer the question, addressed the most relevant points, and was focus and organized. A maximum score of 4 indicated the answer was accurate and thorough; the article provided a clear answer that was well explained, focused, and organized. Each article was independently graded by each observer, and interobserver reproducibility was assessed with a Spearman correlation. The average score between the three graders was used to compare the quality of the articles.
Table 1 Questionnaire to Assess Online Resources Describing Ocular Gene Therapy with Mean and Total Scores
The articles were further evaluated by two graders (JS and NY) using the DISCERN instrument. DISCERN is widely used to determine the quality of health information provided to patients.22 Each article was independently graded using the 16-question tool. Each question is rated using a 5-point scale ranging from “No” to “Yes.” A score of 1 is given if the answer to the question is “No.” A score of 5 is given if the answer is a definite “Yes.” Scores of 2 through 4 are given if the article meets the criterion of the question partially. The interobserver reproducibility was determined with a Spearman correlation. The average score between the two graders was used to compare the quality of the articles.
The accountability of each article was evaluated using the Journal of the American Medical Association (JAMA) benchmarks.23 Each article was assessed for the 4 standards: Authorship, attributions, disclosure, and currency. Each article should include the authors and contributors along with their affiliations and relevant credentials to meet the authorship requirements outlined by JAMA. Attributions, or references, should be reported, and disclosures and currency, or date of update, should be specified.
The online tool, Readable, was used to analyze readability of the articles.24 Each article was assessed using the Flesch Reading Ease Score (FRE), Flesch Kincaid Grade Level, Gunning Fox Index, Coleman Liau Index, and Simple Measure of Gobbledygook (SMOG) Index. The Flesch Reading Ease Score awards each article a score from 0–100 based on total words, sentences, and syllables in the text. Higher scores translate to better readability. A score between 70 to 80 is comparable to an eighth-grade reading level. The Flesch Kincaid Grade Level, Gunning Fox Index, Coleman Liau Index, and SMOG Index are separate readability formulas which report the reading level required to understand the text. Each index states the US grade level of education reflected by the article. The mean score of these indices corresponds to the numerical US grade level. Therefore, a mean score of 12 indicates a 12th grade reading level while a score above 12 corresponds to a college reading level.
Statistical analysis was completed using IBM SPSS Statistics for Mac, version 25.0, released 2017 (Armonk, NY: IBM Corp). Content analysis using the 25-question assessment and the DISCERN instrument was performed by treating the data as ordinal variables and analyzing with the Kruskal–Wallis test. A post-hoc Dunn-Bonferroni test was used to determine pairwise comparisons. Statistical significance was set at P≤0.05 for the main comparisons and the pairwise comparisons. The readability analysis was conducted using a Kruskal–Wallis test to compare the mean reading grade level for each article. A post-hoc Dunn-Bonferroni test was used to determine pairwise comparisons. A Spearman correlation test was carried out to evaluate correlation between accuracy and readability. Statistical significance was set at P≤0.05 for the main comparisons, Spearman correlation, and pairwise comparisons.
Article Selection and Content Analysis
Ten articles were analyzed for the study. The inter-observer reproducibility was statistically significant between JS and AEK (r=0.66, P=0.038) and between JS and NY (r=0.84, P=0.003) and approached statistical significance between AEK and NY (r=0.62, P=0.058). The average questionnaire score for all the articles was 33.93 (SD 11.21, CI 95% ±6.95) out of 100 possible points. There was a statistically significant difference in the content accuracy and quality between the articles (P=0.017). The top scoring article was EyeWiki with an average score of 51.33 points. MedicineNet was the lowest scoring article with an average score of 16.67 points (Table 2). There was a statistically significant difference in the articles’ scores on each of the 25 questions (P<0.001).
Table 2 Mean and Total Scores for Grading Online Resources Describing Ocular Gene Therapy
For the DISCERN instrument, the inter-observer reproducibility was statistically significant between the two graders (r=0.89, P<0.001). The average score for all of the articles with this grading tool was 45.20 (SD 8.40, CI 95% ±5.21) out of 80 possible points. Results of the instrument also showed a statistically significant difference in the content quality between the articles (P=0.043), with EyeWiki scoring highest with an average score of 60.00 points and MedicineNet scoring lowest with 30.00 points (Table 3). There was a statistically significant correlation between the articles’ scores on the 25-question grading scale and on the DISCERN instrument (r=0.90, P<0.001).
Table 3 Mean and Total Scores for Online Sources Describing Ocular Gene Therapy Graded Using the DISCERN Instrument
None of the articles achieved all 4 JAMA benchmarks. One of the 10 (10%) articles, EyeWiki, achieved 3 of the 4 JAMA benchmarks (Table 4). Currency was the most displayed benchmark followed by attribution and authorship. There was no correlation between the accuracy of the articles and JAMA benchmarks (r=−0.229, P=0.524).
Table 4 JAMA Benchmarks Achieved by Online Sources Regarding Ocular Gene Therapy
The mean Flesch Reading Ease Score for all the articles was 39.51 (SD 9.98, CI 95% ±6.19). The mean reading grade for all websites was 12.88 (SD 1.93, CI 95% ±1.19). There was no significant correlation between the FRE score and mean reading grade level (r=−0.340, P=0.336). There was a significant difference between the mean reading grade level of the websites (P=0.001). Wikipedia had the lowest mean reading grade level of 9.63, and the FDA website had the highest mean reading grade level of 16.73. Wikipedia (P=0.005) and the Spark Therapeutics product website (P=0.015) had significantly lower reading levels than the FDA website (Table 5). There was no significant correlation between website accuracy and the mean reading grade (r=−0.444, p=0.199).
Table 5 Readability Analysis of Online Sources Describing Ocular Gene Therapy
The results (Table 6) demonstrated that there was significant variation in the quality, accuracy, and readability of the information available to patients about the ocular gene therapy voretigene neparvovec. EyeWiki provided the most complete and accurate information of the articles analyzed and achieved the most benchmarks of all the articles fulfilling 3 of the 4 benchmarks. However, EyeWiki had a mean reading grade level of 12.85, and is difficult to read for a layperson, as it is primarily designed for ocular provider education rather than patient consumption. Still, given that it is freely available and accessible by the public, it was included in the analysis.
Table 6 Summary Table of Grading Indices Used to Assess Online Sources Describing Ocular Gene Therapy
The second-highest scoring resource on both grading tools was the Spark Therapeutics patient informational website regarding voretigene neparvovec; it was the second most readable article as well with a mean reading grade level of 10.40. Nevertheless, it scored only 54.50 out of 80 points on the DISCERN grading tool and below 50 out of 100 points on average on the 25-question assessment. It did not achieve any of the JAMA benchmarks; moreover, despite being the highest scoring resource designed for reading by patients, it is not an ideal source of reference given the potential inherent conflict of interest present in the manufacturing company’s discussion of its own product.25
In this study, the information available from formal organizations such as the FDA, AAO, and NEI was of overall low quality. Ideally, these major organizations would have accurate and complete information that is easy to understand as many patients rely on these institutions for medical knowledge. Prior studies have shown that educated patients have greater trust in federal agencies and community organizations as sources of medical information.26 Overall, patients prefer non-commercial websites especially from established organizations such as the FDA and NEI over other sources.27,28 The FDA website received the highest content and DISCERN average score of these three resources; however, it had a mean reading grade level of 16.73, the highest level of all the articles. Although the AAO and NEI websites were easier to read with mean reading levels of 11.45 and 13.03, respectively, both resources scored poorly on both scales. Furthermore, the AAO and FDA websites only achieved 1 of the 4 JAMA benchmarks while NEI achieved 2 of the 4 benchmarks demonstrating the overall poor accountability of the articles, although one could argue that the organization as a whole may be taking accountability for the content, which would not be identified by the JAMA benchmark scoring.
This study was limited in several aspects. The 25-question assessment was based on two retina providers' experience and assumptions regarding typical patient questions and was not validated in a patient population. Furthermore, the inter-observer correlation between two pairs of the graders resulted in moderate strength of correlation demonstrating variability in interpretation of the articles or of the grading scale. To further test the online resources being evaluated, the standardized DISCERN instrument was also incorporated, and there was a significant and strong positive correlation between the DISCERN scores and the content assessment scores. Inter-observer reproducibility had greater strength in the DISCERN instrument as well. Patients may use search engines or keywords that vary from the ones used to determine websites included in this study. Some resources included in this study were found beyond the second results page on Google, but patients often only use the first two results pages for their research.28 These resources were still included given their importance as prominent, public organizations (eg NEI, FDA). The resources evaluated were non-peer reviewed sources of varied websites including commercial, government, and non-government organizations. These articles were chosen as they were readily available during search, but patients may prefer to read peer reviewed sources or information from other organizations. Resources were not directly assessed by patients; future studies could further study online resource quality and content by directly evaluating patients’ comprehension.
In conclusion, this study demonstrated that patient accessible online resources regarding the ocular gene therapy voretigene neparvovec are generally incomplete, difficult to read, and of poor quality. The available material varied significantly by source and did not provide adequate information to aid patients in their medical decisions. This indicates an unfilled need to create patient-friendly online content regarding this ocular gene therapy as available research and therapeutic options continue to evolve. Future articles can be improved by assuring accuracy of the information using peer reviewed research papers as reference and by maintaining a reading level sufficient for patients’ understanding.
This work was supported by the NIH under Center Core Grant P30EY014801, Research to Prevent Blindness Unrestricted Grant. The sponsor or funding organization had no role in the design or conduct of this research.
Dr Nicolas A Yannuzzi is advisor for Alcon, Novartis, Genentech, and Alimera Sciences, outside the submitted work. Dr Jayanth Sridhar reports personal fees from Alcon, Dorc, Genentech, and Regeneron, outside the submitted work. The authors report no other conflicts of interest in this work.
1. What is gene therapy? Genetics Home Reference. Available from: https://ghr.nlm.nih.gov/primer/therapy/genetherapy.
2. Dunbar CE, High KA, Joung JK, Kohn DB, Ozawa K, Sadelain M. Gene therapy comes of age. Science. 2018;359(6372):eaan4672. doi:10.1126/science.aan4672
3. Naldini L. Gene therapy returns to centre stage. Nature. 2015;526:351–360. doi:10.1038/nature15818
4. Liu MM, Tuo J, Chan CC. Gene therapy for ocular diseases. Br J Ophthalmol. 2011;95(5):604–612. doi:10.1136/bjo.2009.174912
5. LaVito A. FDA approves Luxturna gene therapy for rare form of inherited vision loss; 2017. Available from: https://www.cnbc.com/2017/12/19/fda-approves-spark-therapeutics-luxturna-gene-therapy.html.
6. Russell S, Bennett J, Wellman JA, et al. Efficacy and safety of voretigene neparvovec (AAV2-hRPE65v2) in patients with RPE65-mediated inherited retinal dystrophy: a randomised, controlled, open-label, Phase 3 trial. Lancet. 2017;390(10097):849–860. doi:10.1016/S0140-6736(17)31868-8
7. Maguire AM, Russell S, Wellman JA, et al. Efficacy, Safety, and Durability of Voretigene Neparvovec-rzyl in RPE65 Mutation–Associated Inherited Retinal Dystrophy. Ophthalmology. 2019;126(9):1273–1285. doi:10.1016/j.ophtha.2019.06.017
8. Vital decisions: A pew internet health report pew research center. Available from: https://www.pewinternet.org/2002/05/22/vital-decisions-a-pew-internet-health-report/.
9. Chen YY, Li CM, Liang JC, Tsai CC. Health Information Obtained From the Internet and Changes in Medical Decision Making: questionnaire Development and Cross-Sectional Survey. J Med Internet Res. 2018;20(2):e47. doi:10.2196/jmir.9370
10. Safeer RS, Keenan J. Health Literacy: the Gap Between Physicians and Patients. Am Fam Physician. 2005;72(3):463–468.
11. DeWalt DA, Berkman ND, Sheridan S, Lohr KN, Pignone MP. Literacy and Health Outcomes. J Gen Intern Med. 2004;19:1228–1239. doi:10.1111/j.1525-1497.2004.40153.x
12. Gudgel D. Gene Therapy Approved to Treat Rare Cause of Vision Loss; 2017. Available from: https://www.aao.org/eye-health/news/gene-therapy-fda-approved-rare-retina-blindness.
13. Gene therapy successful in treating rare retinal disorder; 2015. Available from: https://www.aoa.org/news/clinical-eye-care/diseases-and-conditions/gene-therapy-successful-in-treating-rare-retinal-disorder?sso=y.
14. Voretigene neparvovec-rzyl (Luxturna). Available from: https://eyewiki.aao.org/Voretigene_neparvovec-rzyl_(Luxturna%E2%84%A2).
15. Fischer A. FDA approves novel gene therapy to treat patients with a rare form of inherited vision loss; 2017. Available from: https://www.fda.gov/news-events/press-announcements/fda-approves-novel-gene-therapy-treat-patients-rare-form-inherited-vision-loss.
17. Roberts S. Therapy Approved for Rare Inherited Vision-Loss Disorders; 2017. Available from: https://www.medicinenet.com/script/main/art.asp?articlekey=209025.
18. NIH-funded Study Points Way Forward for Retinal Disease Gene Therapy. Available from: https://www.nei.nih.gov/about/news-and-events/news/nih-funded-study-points-way-forward-retinal-disease-gene-therapy. 2015.
19. Novartis announces landmark EU approval for one-time gene therapy Luxturna to restore vision in people with rare inherited retinal disease; 2018. Available from: https://www.novartis.com/news/media-releases/novartis-announces-landmark-eu-approval-one-time-gene-therapy-luxturna-restore-vision-people-rare-inherited-retinal-disease.
20. Preidt R. FDA Approves Gene Therapy for Rare Form of Blindness; 2017. Available from: https://www.webmd.com/eye-health/news/20171219/fda-oks-gene-therapy-for-rare-form-of-blindness.
21. Voretigene neparvovec. Available from: https://en.wikipedia.org/wiki/Voretigene_neparvovec.
22. Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health. 1999;53(2):105–111. doi:10.1136/jech.53.2.105
23. Silberg WM, Lundberg GD, Musacchio RA. Assessing, Controlling, and Assuring the Quality of Medical Information on the Internet. JAMA. 1997;277(15):1244. doi:10.1001/jama.1997.03540390074039
25. T’Hoen E. Direct-to-consumer advertising: for better profits or for better health? Am J Health-Syst Pharm. 1998;55(6):594–597. doi:10.1093/ajhp/55.6.594
26. Dutta-Bergman M. Trusted Online Sources of Health Information: differences in Demographics, Health Beliefs, and Health-Information Orientation. J Med Internet Res. 2003;5(3):e21. doi:10.2196/jmir.5.3.e21
27. Rozmovits L, Ziebland S. What do patients with prostate or breast cancer want from an Internet site? A qualitative study of information needs. Patient Educ Couns. 2004;53(1):57–64. doi:10.1016/S0738-3991(03)00116-2
28. Morahan-Martin JM. How Internet Users Find, Evaluate, and Use Online Health Information: a Cross-Cultural Review. Cyberpsychol Behav. 2004;7(5):497–510. doi:10.1089/cpb.2004.7.497
This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.Download Article [PDF]