Accuracy and Quality of YouTube Videos as a Source of Information on Vitiligo

Introduction

Vitiligo is a long-term skin condition characterized by patches of discoloration in the skin, affecting as much as 0.5–2% of the global population.¹ Pathophysiology of the disease involves mainly autoimmune destruction of melanocytes.² Moreover, a strong association between vitiligo and other autoimmune diseases has been well established including Hashimoto thyroiditis, graves disease, pernicious anemia, inflammatory bowel disease and more.^3,4 Symptoms of vitiligo are usually considered harmless and are cosmetic in nature, which may have contributed to the underestimation of the importance of treatment.⁵ Apart from the visual manifestation of the disease, vitiligo also impacts the mental health and health-related quality of life of the patients, particularly those with dark skin.⁵ Studies have demonstrated that vitiligo patients with worse quality of life tend to seek belongingness through support groups for their social, protective effect.^6,7 Social media usage dramatically increased over the previous decades; as much as 86% of adults between the ages of 18–29 years old in the United States use social media.⁸ This increase in usage also altered the way information is circulated; some platforms including YouTube have been used to deliver and share health-related information among groups of people sharing the same medical condition.^9,10 There have been studies which assessed dermatology-related content on different social media platforms, citing misinformation as one of the primary challenges in these set-ups.^11–13 Few studies have documented vitiligo-related content in the social media platform YouTube, which highlighted the same problems of misinformation, but also provided insights on patient perspectives and psychosocial support.^9,14 One of these studies showed that alternative/complementary medicine was discussed in YouTube videos more than traditional medical treatment (44% vs 31%).⁹

One of the many ways by which consumer health information is often evaluated is through a 5-point scale called the Global Quality Scale (GQS). The GQS is usually accompanied with criteria questions to determine the usefulness and quality of the information presented to the reviewer.¹⁵ To further assess the quality of the content, a tool called DISCERN is employed. DISCERN is designed to estimate the quality of information and set standards with regard to the production of high-quality consumer health information.¹⁶ Lastly, to gauge the accuracy (or lack thereof) of the videos, the validated assessment tool Accuracy in Digital-Health Instrument (ANDI) was used.¹⁷ As of writing, there are no studies which integrated DISCERN, GQS, and ANDI in their cross-sectional analysis of the quality and accuracy of YouTube videos as a source of information for vitiligo. Thus, this study aims to evaluate vitiligo-related content in the social media platform YouTube using statistical tools, GQS, ANDI and DISCERN.

Materials and Methods

Screening

The social media platform YouTube was accessed last February 19, 2021, using the search parameter “Vitiligo”. To minimize the probability of bias attributed to search history, the incognito feature was used. The first 30 results by relevance were selected for subsequent analysis, as typical content consumers do not usually go beyond this point.¹⁸ Only videos in the English language were included in the study.¹⁸ Additionally, videos longer than twenty (20) minutes were excluded as they are not representative of the preferences of an average viewer.¹⁹ The uploading channel and the number of likes, dislikes, and comments were recorded for each video. Viewer’s engagement ratio was calculated by dividing the total number of likes, dislikes, and comments by the number of views. Ethical approval was waived by the Research and Studies Department in Jeddah Health Affairs since the study involved data available in the public domain.

Global Quality Scale, DISCERN, and ANDI

Three independent vitiligo experts evaluated the quality of each video using the Global Quality Scale and DISCERN. Additionally, the accuracy of each video was assessed using the validated tool Accuracy in Digital-health Instrument (ANDI). Criteria questions and scales are enumerated in Appendix 1. Inter-rater reliability was measured using Cronbach’s alpha.

Statistical Analysis

Raw data were encoded in Microsoft Excel^TM and exported to SPSS v24.0 (IBM Corp, Armonk, NY) for analysis. Descriptive statistics analyses were performed for both categorical and numerical variables. Mean scores from the different tools for assessment were compared between healthcare and non-healthcare sources using an independent t-test. A p-value ≤0.05 was used as a basis for statistically significant difference for all statistical tests.

Results

Thirty videos were screened according to the inclusion criteria. Videos uploaded by a hospital, or a physician constituted 40%, and those considered as educational channels were around 17% of the total videos. Both were categorized as health-care sources while the rest were regarded as non-healthcare sources. These included (news or health blog) as 23% and (patient perspective) as 20% of the total. The median duration of videos was 217 seconds (min 54 sec, max 1093 sec). The mean number of views, likes and dislikes were 253.7, 3049.4 and 85.4, respectively.

Three independent vitiligo experts rated the quality and accuracy of the YouTube videos using DISCERN, GQS, and ANDI. There was a significant difference between healthcare and non-healthcare sources through the DISCERN assessment tool (Figure 1). The scores for ANDI and GQS were slightly higher for health-care sources but the difference was not statistically significantly. A breakdown of Discern score of all videos showed very poor score in more than half of the videos (57%), poor in 33% and fair in 10%. None of the videos had very good or excellent score (Figure 2).

Figure 1 Vitiligo videos DISCERN, accuracy and global quality mean scores based on health-care vs non-health-care sources.

Figure 2 Breakdown of total DISCERN score of all videos.

The viewer engagement ratio was calculated to be 0.016 for health-care sources and 0.019 for non-healthcare sources, albeit the difference is not statistically significant (p = 0.23). The inter-rater reliability was estimated using the Cronbach’s alpha which showed a good intra-class correlation for DISCERN (0.86), acceptable for ANDI (0.76) and unacceptable for GQS (0.41).

Discussion

Impairment of the quality of life of vitiligo patients has been documented as early as the 1970s.²⁰ It is known that a range of psychological outcomes may arise from this circumstance, ranging from depression to anxiety, as a consequence of low self-esteem and social isolation.²⁰ With the onset of social media platforms like YouTube, patients suffering from the same condition are able to share their perspectives and experiences, and health-care sources can disseminate knowledge about a certain disease to remedy some of these psychological outcomes. Thus, it is important to evaluate the accuracy and overall quality of the content of the information being consumed through YouTube videos. With the paucity of studies on the accuracy, reliability, and quality of vitiligo-related content in YouTube, this study aims to evaluate these contents with the aid of statistics and assessment tools such as the GQS, ANDI and DISCERN.

The majority (57%) of videos came from health-care sources, while non-healthcare sources comprised only 43%. However, the viewer engagement ratio was higher in non-healthcare sources (0.019) than its counterpart (0.016), although the difference is not statistically significant. This is consistent with the findings of a recent study, citing viewer experience and visibility as the areas in need of further improvement for videos produced as health-care sources.¹⁴ Another study hypothesized that the storytelling and anecdotal pattern of delivering information by non-healthcare sources are more emotionally engaging than statistics, numbers, and general descriptions typical of the content from health-care sources.¹⁷

In terms of the content-assessment tools, a statistically significant difference in quality through DISCERN was observed between healthcare and non-healthcare sources (P = 0.009) although the overall quality was still considered to be poor to fair. There was a good overall agreement on that between evaluators (0.86). Moreover, GQS and ANDI also revealed a higher overall quality and accuracy for health-care sources than non-healthcare sources, although the difference was not statistically significant. This is consistent with the findings of a recent study which showed a higher ANDI score for health-care sources compared to non-healthcare sources.¹⁴ Higher GQS scores have also been observed in health-care sources in similar studies conducted on rosacea-related content.¹⁷ Different assessment tools may overlap and may interpret the same properties differently; this implies that DISCERN can potentially capture information missed by GQS, or vice-versa.²¹ Ultimately, this highlights the importance of cross-analyses utilizing multiple assessment tools to gain a holistic evaluation of the contents of the YouTube videos.

Limitations of the study include the sample size, different search results, discordance between raters in ANDI ratings and the possible associations between the engagements and the content. Although this study only sampled 30 videos, previous studies have demonstrated that a minimum of 25–50 videos are adequate in making a distinction of clinically relevant differences.¹⁷ Using the YouTube search tool may generate different search results per search, which is one of the reasons this study employed inclusion/exclusion criteria to accommodate this possibility. Lastly, this study did not evaluate possible associations between comments, likes, dislikes, and views, with the content of the videos, as it is not included in the main objective. However, further studies may be conducted to analyze these associations.

Conclusion

The burden associated with vitiligo symptoms prompts patients to seek information wherever it is available, including social media platforms such as YouTube. This study showed that while most of the videos came from health-care sources, a higher viewer engagement is garnered by non-healthcare sources. Although only statistically significant for Discern, ratings from three independent vitiligo experts were generally higher for DISCERN, ANDI, and GQS scores of hospital-physician produced videos compared to non-healthcare sources. However, the accuracy and quality of these videos is generally low. This highlights the prevalence of health-related misinformation in social media. While very little can be done to prevent non-healthcare sources to produce engaging yet inaccurate videos, future efforts by health-care facilities and experts should consider improving quality and viewer engagement when producing their own information materials. It is also recommended to educate the public on making distinctions between evidence and non-evidence-based health information on the internet.