Misinformation Persists in Complementary Health: Evaluating the Reliability and Quality of YouTube-Based Information on the Use of Acupuncture for Chronic Pain

Introduction

Chronic pain is characterized by persistent pain that lasts for more than 3 months, causing ongoing discomfort and distress. It can arise from various factors, such as physical trauma, surgical interventions, or underlying medical conditions. Annually, about 1 in 5 adults in the United States (US) experience chronic pain.¹ The burden of chronic pain has economic implications costing the US healthcare system over $100 billion annually through reduced quality of life, work productivity, and healthcare costs.² Current treatment options for chronic pain include instituting a multimodal treatment plan consisting of conservative therapy (physical therapy, cognitive-behavioral therapy, and complementary and alternative medicine therapies), analgesics such as opioids, non-steroidal anti-inflammatory drugs, and neuropathic agents, as well as interventional treatment such as injections and neuromodulation.^3–5

Acupuncture is a technique based on traditional Chinese medicine that is commonly used to manage chronic pain syndromes. The exact mechanism of analgesia from acupuncture is still unclear; however, it is believed to be due to release of anti-inflammatory and inhibitory neurotransmitters following the stimulation of acupoints.⁶ Although interest has been growing on the use of acupuncture as a non-pharmacological option for pain management, its effectiveness and clinical utility remain controversial.

A recent study evaluating complementary health approaches (CHAs), including massage, yoga, and acupuncture, showed that 70% of patients with low back pain tried out at least one CHA, while over 40% used at least two. Additionally, the study found that a higher level of education and literacy was associated with increased CHA utilization.^7,8 Social media platforms have recently been used as a resource to learn more about diseases, disease prevention, and various treatments.⁹ To that end, many patients experiencing chronic pain will likely consider using social media platforms, such as YouTube, to obtain more information on acupuncture and other treatment options. Although accessibility of information through social media is an advantage, much of the information presented may be misleading, potentially impacting the patient’s healthcare decisions negatively.¹⁰

YouTube is a popular video-sharing site, with over 122 million daily active users, that is often used to disseminate information regarding healthcare.¹¹ The platform’s widespread availability and accessibility have made it an attractive resource for patients seeking information and support beyond traditional healthcare settings.¹² YouTube’s merits lie in its ability to provide diverse perspectives, personal experiences, and expert opinions, which can help patients make informed decisions and feel more empowered in managing their health.¹³ However, the platform’s open nature and lack of strict regulation also give rise to several demerits, such as the rapid spread of misinformation, unsubstantiated claims, and potentially harmful advice.¹² To combat misinformation on health-related conditions being spread on YouTube, the platform has recently developed a tool for videos that contain certain health-related information. In these videos, an information panel will appear stating the source of the information. This tool was designed to help viewers easily identify eligible and reputable health sources that meet the principles set by the National Academy of Medicine (NAM) and American Public Health Society (APHA). A personal account must apply for eligibility and go through a series of eligibility checks to get an information panel. However, this feature is not available in every country (only available in the US and UK to date), and this feature does not yet apply to all health conditions.

To date, minimal data exists regarding the utility, accuracy, and effectiveness of available information on acupuncture from the YouTube platform. Recent studies have shown that more than 40% of the healthcare seeking population uses social media platforms, such as YouTube, as a source for medical information.^14,15 However, the lack of strict regulations and the open nature of YouTube can lead to the spread of misinformation, causing confusion and potentially harmful decision-making among patients.^14,15 This study aims to bridge this gap and assess the quality and usefulness of information available on YouTube on acupuncture for chronic pain treatment.

The findings of this review hold clinical significance for clinicians, researchers, and patients. It provides clinicians with insights into the information their patients may encounter online, enabling them to offer better guidance and education. Researchers can use this study as a foundation for further investigations into the impact of online health information on patient decision-making and the development of guidelines for creating reliable health-related content on social media platforms. Lastly, patients can benefit by gaining awareness of the varying quality of health information on YouTube and the importance of seeking information from reputable sources.

Materials and Methods

In this study, we performed a cross-sectional analysis of YouTube videos related to the use of acupuncture for managing chronic pain. Our aim was to evaluate the quality and usefulness of the information presented in these videos.

Search Strategy

Several search term combinations were used to query the YouTube platform (www.youtube.com): “acupuncture for chronic pain”, “acupuncture pain relief”, “acupuncture pain”. More than one search term was used to facilitate a sensitive capture of relevant content. The results were filtered based on the search relevance algorithm and then the authors filtered videos by view count from highest to lowest. The Mayo Clinic institutional review board deemed this study as exempt from review as the data involve publicly available data.

Video Selection Process

The top 200 videos by view count were initially selected after the first query. We included any video that was >1 minute duration, contained audio in the English language or contained English subtitles, had >7000 views, and was related to acupuncture. Videos that met any of the following conditions were excluded from analysis:

1. Videos under one minute were excluded (rationale: these videos did not provide sufficient time to cover the necessary content on acupuncture)
2. Videos not in English or without English texts were excluded.
3. Duplicate videos were excluded.
4. Videos without audio were excluded.
5. Videos with less than seven thousand views were excluded (rationale: this threshold was utilized to include the most popular videos).
6. Videos unrelated to acupuncture were excluded.

After reviewing videos for relevance and removing those that met exclusion criteria, a final list of the top 54 videos by view count were selected to be included in the final analysis.

Data Extraction

Two independent reviewers (S.K. and H.T.) extracted and recorded various data points from each video into an electronic database. The collected data included:

1. Uniform resource locator (URL): The unique web address of the video.
2. Date of upload: The date when the video was uploaded to YouTube.
3. View count: The total number of times the video had been viewed.
4. Length of the video in seconds: The duration of the video.
5. Amount of likes: The number of “like” reactions the video received from viewers.
6. Amount of comments: The number of comments left by viewers on the video.

Additionally, the reviewers categorized the production source of each video into one of six classifications: (a) healthcare institutions, which included videos created by hospitals, clinics, or other healthcare organizations; (b) health media, encompassing videos produced by media outlets specializing in health-related content; (c) science media, consisting of videos created by media outlets focusing on scientific topics; (d) news media, comprising videos produced by general news outlets; (e) television shows, including videos originating from television programs; and (f) nonmedical independent user, referring to videos created by individual users not affiliated with any of the above categories.

Usefulness of Video Content

The videos were evaluated by two independent reviewers (S.K. and H.T.) as either useful, misleading, or neither useful nor misleading, using criteria from a previous systematic review identifying evidence-based uses for acupuncture.¹⁶ Any discrepancies were resolved by a third reviewer (R.S.D.). The criteria for a useful video were 1) a correct statement about the definition of acupuncture; 2) an accurate statement of the pain-reducing mechanism of acupuncture; or 3) an accurate statement about the indications, adverse effects, benefits, and potential outcomes of acupuncture. Given that our target audience was the general public, as opposed to a more experienced cohort such as clinicians and researchers, the authors considered it important for videos to highlight the definition of acupuncture to facilitate basic understanding of acupuncture in those with limited knowledge.

The criteria for a video to be deemed misleading were 1) inaccurate definition of acupuncture; 2) at least one inaccurate statement about the pain-reducing mechanism of acupuncture; or 3) at least one inaccurate statement about the indications, adverse effects, benefits, and potential outcomes of acupuncture. A video was classified as neither useful nor misleading if it had 1) no definition of acupuncture; 2) no statement on the pain-reducing mechanism of acupuncture; or 3) no statement on the indications, adverse effects, benefits, and potential outcomes of acupuncture.

Quality of Video Content

The reliability and quality of the video content were determined using the modified DISCERN scale (mDISCERN) and Global Quality Scale (GQS), respectively, which have both been utilized by previous studies.^17,18 These two scoring systems have also been previously used to appraise social media video content.^19–23

The reliability of the information presented in the videos was assessed using the 5-question scoring system called mDISCERN, which was adapted from the longer 15-question DISCERN scale.²⁴ The criteria for the mDISCERN were:

1. Are the aims clear and achieved?
2. Are reliable sources of information used?
3. Is the information presented balanced and unbiased?
4. Are additional sources of information listed for patient reference?
5. Are areas of uncertainty mentioned?

Each domain was scored on a scale of 1 to 5, where each “yes” vote would count as one point and each “no” vote would not award any points. A score greater than 3 indicated high reliability.

The quality of the video content was assessed using the GQS 5-point scoring system.

Score 1: Poor quality, poor flow, and most information is missing, so it is not helpful for patients.

Score 2: Generally poor, with some information given but of limited use to patients

Score 3: Moderate quality and some important information are adequately discussed

Score 4: Good quality, good flow, and most relevant information are covered, making it useful for patients

Score 5: Excellent quality and excellent flow, making it very useful for patients

The videos as a whole were scored between 1 and 5, where a score of 1 or 2 was classified as low quality, a score of 3 was intermediate quality, and a score of 4 or 5 was good quality.^18,25 Any discrepancies to mDISCERN and GQS ratings were resolved by a third reviewer (R.S.D.).

Data Analysis

The characteristics of the YouTube videos, video production source, usefulness metrics, mDISCERN scores, and GQS scores were summarized using their mean values and standard deviation. Cohen’s Kappa was used to determine the inter-rater reliability of the two reviewers’ (S.K. and H.T.) mDISCERN and GQS scores. Values of ≤ 0 were considered to be no agreement amongst the reviewers, 0.01–0.20 were considered slight agreement, 0.21–0.40 were considered fair agreement, 0.41–0.60 were considered moderate agreement, 0.61–0.80 were considered substantial agreement, and 0.81–1.00 were considered almost perfect agreement between the reviewers.²⁶ The agreement percent, expected chance, standard error, and 95% confidence interval were also calculated for the mDISCERN and GQS scores. Pairwise comparisons were conducted between “useful” and “misleading” videos. Continuous variables were compared using a Student’s t-test, whereas binary categorical variables were compared using a chi-square test. A p-value of <0.05 was considered statistically significant. All data were analyzed using GraphPad Prism 9.

Results

A total of 200 videos were considered, and 54 videos met the inclusion criteria (Figure 1). Out of the final list of 54 videos, 57.4% (n = 31) were categorized as useful, while 14.8% (n = 8) were designated misleading, and 27.8% (n = 15) were neither useful nor misleading. The Cohen’s Kappa for the mDISCERN scores was 0.74 (95% CI, 0.58–0.89) with an 83.3% agreement. The Cohen’s Kappa for the GQS scores was 0.44 (95% CI, 0.24–0.62) with a 61.1% agreement.

Figure 1 Flow chart showing systematic video search and selection process.

Video Characteristics

The YouTube video characteristics are summarized in Table 1. There were no significant group differences in the mean duration, total number of days on YouTube, mean number of views, mean viewership/day, mean number of likes, and mean number of comments when comparing the useful and misleading videos.

Table 1 YouTube Video Characteristics and Video Production Sources

When looking at these same characteristics in the videos that had an mDISCERN <3 or mDISCERN ≥3, none of the comparisons were statistically different. Similarly, when these same comparisons were conducted between videos with GQS<3 and GQS ≥3, there were no statistically significant differences. These findings are summarized in Tables 2 and 3.

Table 2 YouTube Video Characteristics and Sources Based on mDISCERN

Table 3 YouTube Video Characteristics and Sources Based on GQS

Useful Videos and Healthcare Institutions

Of the videos deemed useful, 41.8% (n=13) were found to be published by healthcare institutions. However, none of the videos categorized as misleading were published by healthcare institutions (P=0.0249). When comparing the source that published useful videos versus the source that published videos categorized as neither useful nor misleading, we found that only 6.7% (n=1) of videos categorized as neither useful nor misleading were published by healthcare institutions (P=0.0148) (Table 1).

Misleading Videos and Health Media

When assessing the amount of videos produced by health media (unaffiliated with healthcare institutions themselves), we found that 87.5% (n=7) of the misleading videos fell into this category while only 25.8% (n=8) of useful videos fell into this category (P=0.0014) (Table 1). Similarly, a higher percentage of videos classified as neither useful nor misleading were produced by health media (80%, n=12, P= 0.0005).

Video Production Source and mDISCERN Score

When comparing the video production source based on mDISCERN score (Table 2), we found that 11.1% (n=1) of the videos with an mDISCERN score <3 were created by a non-medical independent user, while none of the videos with an mDISCERN ≥3 were created by that source (P=0.0240). No other statistical differences were observed when comparing the other video production sources based on mDISCERN score cutoffs (Table 2).

Video Production Source and GQS Score

When comparing the video production source by GQS score (Table 3), we observed that 14.3% (n=1) of the videos with a GQS <3 were created by a non-medical independent user, while none of the videos with a GQS ≥3 were created by that source (P=0.0089). No other significant differences were observed when comparing the other video production sources based on GQS score cutoffs (Table 3).

mDISCERN and GQS Scores

Videos classified as useful had a higher reliability and quality score as indicated by their mean mDISCERN and GQS score (3.48 ± 0.63 and 3.77 ± 0.67 respectively), compared to videos classified as misleading (2.38 ± 0.52 and 2.5 ± 0.53, respectively; P<0.0001 and P=0.0002, respectively). Similarly, compared to useful videos, videos that were neither useful nor misleading had a mean mDISCERN and GQS of 2.73 ± 0.70 (P=0.0001) and 2.87 ± 0.74 (0.0007), respectively.

Discussion

In this study, we observed that only approximately half of the included videos (57.4%) contained useful information about acupuncture to manage chronic pain. Importantly, about 15% of videos contained misleading information about acupuncture for chronic pain. The useful videos were deemed to have the highest quality with an average GQS score of 3.77 and mDISCERN score of 3.48. In contrast, videos categorized as misleading had, on average, GQS and mDISCERN scores of 2.50 and 2.38, respectively. Almost half of the useful videos were produced by health institutions while videos that were misleading were not produced by health institutions. This suggests that the information presented by healthcare-oriented channels is, on average, more reliable and of higher quality compared to non-healthcare-oriented channels (News Media, Science Media, TV shows, Personal Accounts). Healthcare institutions publishing informational videos without obvious commercial upside were some of the most reliable sources, with all 14 videos created by a healthcare institution having an mDISCERN and GQS score of ≥3. Interestingly, we observed that a video produced by a non-medical independent user had scored a 1 for both the mDISCERN and GQS, suggesting that personal accounts release information that are more likely to contain very poor quality information.

Further research should be conducted about the demographic population of the viewers, which can provide valuable information on who the type of audience viewing these videos. This article also serves as an example of quantifying data gathered from reviewing online social media-based information. This approach has been utilized in prior articles and encourages clinicians to do the same in other areas of medicine.^14,27–29 Our data evaluating acupuncture-based videos suggest a small glimpse of misinformation and with further research evaluating other areas of pain management, we might be able to unearth the full extent of misinformation plaguing social media and the medical community.

Several limitations were encountered in this study. One limitation is that the information gathered from a video on the YouTube platform at a specific point in time (eg, the number of likes, comments, and date since upload) may not accurately represent the video’s information at a later time due to the dynamic nature of YouTube. We attempted to minimize this limitation by having our reviewers record video information on the same day. Secondly, the exclusion of videos with less than 7000 views is a limitation, which may have excluded videos that were relatively popular but did not meet the view count threshold. Additionally, we only analyzed videos in English, which limits the relevance or generalizability of this study to countries outside the United States. Thirdly, our analysis was limited to videos found on the YouTube platform, which prevented the evaluation of acupuncture-related content on other social media platforms that could potentially be of higher quality.

Additional research should be conducted to investigate the demographic characteristics of viewers who watch acupuncture-related videos on YouTube. This information can provide valuable insights into the target audience and help tailor future content to better meet their needs. Moreover, it is essential to expand the research scope to include other social media platforms, such as Facebook, Instagram, and TikTok, as patients may seek health information from multiple sources. By examining the quality of acupuncture-related information across various platforms, researchers can gain a more comprehensive understanding of the information landscape. Future studies should also investigate whether other platforms have implemented features similar to YouTube’s information panel, which indicates the credibility of the health information presented. This research can help identify platforms that prioritize the dissemination of accurate and reliable health information.

Conclusion

Our analysis of the highest viewed acupuncture videos for chronic pain reveals only about half provide useful information, indicating a significant misinformation challenge for viewers. This underscores the urgent need for more high-quality, unbiased videos from healthcare institutions and physicians on complementary health practices like acupuncture.