Overall Reporting Descriptions of Acupuncture for Chronic Pain in Randomized Controlled Trials in English Journals

Introduction

Clinical research can provide a decision-making basis for clinical practice, health policy, and further research, which determines that it needs to provide credible evidence.^1,2 Low-quality clinical research may not only lead to invalid data or distorted results but also be unethical and harm the safety and rights of patients.³ In 2014, the “Increasing Value, Reducing Waste” series in The Lancet provided convincing evidence for waste sources in biomedical research (including patient-oriented clinical research).^4–8 A challenging problem in clinical research is the description quality of the intervention measures.⁹ If there is no complete description of intervention measures, clinicians and researchers will not be able to learn from and replicate the research results, which will lead to the waste of resources to a certain extent. It can be seen that providing detailed intervention measures is the basis for scientific clinical research.¹⁰

To avoid the waste of inadequately producing and reporting research, a series of relevant reporting guidelines for different research types and interventions have been formulated by the committee, team, and so on. The pain-specific supplement to Consolidated Standards for Reporting Trials (CONSORT) was used to provide a list of specific reports for pain clinical trials, which can be used in conjunction with the CONSORT guidelines to optimize RCT reports.¹¹ The acupuncture specific Standards for Reporting Interventions in Controlled Trials of Acupuncture (STRICTA) guidelines were designed to enable researchers to report and display the details of acupuncture interventions more comprehensively.¹² Meanwhile, given the importance of adequate reporting of sham interventions in clinical trials, a reporting guideline-the Template for Intervention Description and Replication (TIDieR)-Placebo was developed by Howick et al in 2020, which is intended to encourage the concise and accurate reporting of the nature and implementation of placebo or sham controls to help explain and use clinical research results.¹³

In this study, we consider acupuncture (body and electric needles) and sham acupuncture interventions used in patients with chronic pain. Chronic pain is a widespread perceptual disease with a continued unpleasant sensory and emotional experience, yielding enormous costs in both health care and socioeconomic burden, and also, very few highly effective treatments.^14,15 Acupuncture is widely used in the treatment of chronic pain. Some studies have conducted a meta-analysis on the effect of acupuncture on relieving chronic pain and improving function.^16,17 Although the number of articles related to acupuncture for pain control has steadily increased during the last two decades,¹⁸ the quality of reporting of randomized controlled trials (RCTs) is not optimal. Therefore, the purpose of our study is to evaluate the overall reporting quality of acupuncture for chronic pain in randomized controlled trials based on the pain-specific CONSORT, revised STRICTA statements, and the TIDieR-Placebo checklist.

Methods

Search Strategy

We conducted a comprehensive search from inception to March 2020 in the following three databases: PubMed, Embase, and Cochrane library. The following search terms were used: (acupuncture OR acupuncture therapy OR body acupuncture OR manual acupuncture OR electroacupuncture OR electro-acupuncture) AND (Chronic Pain OR Chronic Pains OR Pains, Chronic OR Pain, Chronic OR Widespread Chronic Pain OR Chronic Pain, Widespread) AND (Clinical Trials, Randomized OR Trials, Randomized Clinical OR Controlled Clinical Trials, Randomized).

Eligibility Criteria

All RCTs including the word “random” for acupuncture treatment of chronic pain were included. Regardless of whether allocation concealment or blinding were used or not. The following reports were excluded: (1) republished literature; (2) non-chronic pain; (3) case analysis, review, discussions, and other theoretical studies; (4) animal experiments; (5) the full text is not available or non-English literature. Acupuncture was the main treatment method, alone or combined with the control group intervention measures. Manual and electrical needles were included and not including point injection, moxibustion, and cupping were excluded. Acupuncture retention time, frequency, and points were not limited. The control group included sham acupuncture only.

Data Extraction

The included documents were confirmed to be extracted by two independent researchers with the help of the Excel 2010 self-designed document information collection table Each item on the checklists detailed in these reports was answered with a “yes” or“no”. Any missing information was marked “unreported”. Data extraction was conducted by two independent researchers, and if there was any disagreement, it was discussed and finally confirmed by consensus or consultations.

Risk of Bias Assessment

The risk of bias for RCTs was assessed using the Cochrane risk of bias tool, which assessed the risk of bias across five domains: bias in a random process, deviation from established intervention, incomplete outcome data, outcome measurement bias, and selective reporting.¹⁹

Assessment of Reporting Quality

Two independent researchers were trained systematically in studying every item before the evaluation to ensure the proper understanding of each item. They will decide to make a “yes” or “no” judgment according to whether the author reports or not. Disagreements, if any, it was discussed and finally confirmed by the third researcher. Finally, descriptive statistics consisting of frequencies and percentages were used to describe whether studies adhered to the pain-specific CONSORT, STRICTA statement, and the TIDieR-placebo checklist. The CONSORT discussion section items were excluded because we considered them too subjective to evaluate. For the ‘interventions’ item, we used STRICTA and TIDieR-placebo to evaluate the reporting quality instead of pain-specific CONSORT, as STRICTA and TIDieR-placebo have more detailed and specific standards for reporting acupuncture and sham acupuncture interventions. Excel 2010 was used to analyze the data descriptively.

Results

According to the retrieval strategy, 5217 papers on acupuncture for chronic pain were retrieved, after the further screening, 1580 were excluded, 3281 papers were excluded after reading the title and abstract, 282 papers were excluded by full-text reading, and a total of 74 relevant RCTs were finally included. A PRISMA flow diagram was provided in Figure 1.²⁰

Figure 1 The PRISMA flow diagram.Notes: PRISMA figure adapted from Liberati A, Altman D, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Journal of clinical epidemiology. 2009;62(10). Creative Commons.20

Study Characteristics

A total of 10,654 patients were involved in 74 studies, and the number of research centers ranging from 1 to 37. Twenty (27.03%) studies used China as the research location, and another 54 (72.97%) studies used foreign countries as the research location. An overview of study characteristics was provided in Supplementary Material (Shown in Table S1). Over time, the numbers of the included articles were presented in Figure 2. The overall number of studies appeared upswing, with one in 1992 (1.35%), one in 1994 (1.35%), two in 2000 (2.70%), one in 2001 (1.35%), three in 2002 (4.05%), one in 2003 (1.35%), four in 2004 (5.41%), two in 2005 (2.70%), seven in 2006 (9.46%), six in 2007 (8.11%), two in 2008 (2.70%), five in 2009 (6.76%), four in 2010 (5.41%), four in 2011 (5.41%), five in 2012 (6.76%), five in 2013 (6.76%), three in 2014 (4.05%), two in 2015 (2.70%), four in 2016 (5.41%), four in 2017 (5.41%), two in 2018 (2.70%), five in 2019 (6.76%), one in 2020 (1.35%). The types of disease in the included articles were presented in Figure 3. The top of 3 diseases, respectively, were chronic knee pain (20.27%), postoperative pain (13.51%), and migraine (9.46%).

Figure 2 Number of published articles per year (n=74).

Figure 3 The proportion of diseases in literature (n=74).

Risk of Bias Assessment

Risk-of-bias assessments for individual studies were shown in Supplementary Material (Shown in Table S2).

Quality of Reporting

Overall Reporting Quality According to CONSORT-Pain Statement

The quality of the 74 included studies was evaluated based on the CONSORT-pain statement (Table 1). The item of “Statistical methods” remained modest by 52.70% of a positive rate. The “Participant flow” was reported by 70.27%. The “Blinding” was reported by 77.03%. The other items had a substantial degree of agreement by more than 80.00%.

Table 1 Pain Checklist Supplement (n=74)

Overall Reporting Quality According to STRICTA Statement

According to STRICTA criteria, the quality of 74 studies included was evaluated (Table 2). The item of “1c” and “4b” was reported by 0.00% of a positive rate. The item of “4a” was reported by 100.00%, the item of “2c” and “2g” was modest reported by more than 50.00%, the other items had a substantial degree of agreement by more than 80.00%, the content of the items were shown in Table 2.

Table 2 Rating of Overall Quality Using Items from the STRICTA Guideline (n=74)

Overall Reporting Quality According to the TIDieR-Placebo Statement

The number of studies that gave complete descriptions for all TIDieR-placebo items was 0. The reporting quality varied greatly between different items. Compared with acupuncture intervention, the reporting rate of sham control on item 2 (why) and item 4 (what-procedures) was significantly different. The reporting rate of the acupuncture group was more than 90.00%, but that of the sham control was less than 50.00%. The reporting rates of item 3 (what-materials), item 11 (how-well planned), item 12 (how-well actual), and item 13 (measuring the success of blinding) were similar between the two groups. The report rate of the acupuncture group was 56.76%, 10.81%, 6.76%, and 29.73%, and that of sham control was 44.59%, 10.81%, 8.11%, and 29.73%. Figure 4 shows the proportions of studies that met the criteria for each TIDieR-placebo item.

Figure 4 The proportions of articles (n=74) that met the criteria for each TIDieR-placebo item.

Discussion

RCTs, which are considered the most reliable form of scientific evidence, can minimize biases that may compromise the validity of medical research.²¹ The three checklists are evidence-based reporting guides designed to improve research transparency and the reporting quality of interventions in clinical trials of acupuncture. We found that after evaluating the 74 included studies, a considerable number of item descriptions were incomplete or missed. The introduction of these checklists will improve the quality of the report to a certain extent.

Concerning individual item reporting, we found that the lower reported Pain-specific CONSORT items were related to statistical methods, participant flow, and blinding. Statistics plays an essential role in clinical research by providing a framework for making inferences about a population of interest.²² To support the transparency and repeatability of the study and eliminate the impact of improper statistical decision-making on the study conclusion, it is necessary to make a statistical analysis plan in advance.²³ An important issue related to RCTs is the correct use of statistical methods, because some studies have a poor selection of statistical tests.²⁴ Regarding statistics, correct descriptions and explanations are essential. In 2017, a statistical analysis program (SAP) guidance document for RCTs was published in JAMA, which contained a minimum list of items for reporting the details of RCT statistical analysis.²⁵ Therefore, to avoid selective analysis and reporting of clinical studies, it is necessary to provide a clear and comprehensive SAP.

The flow chart includes the number of people screened in the trial, the number of randomization and statistical analysis, and the drop-off situation during the trial. Provide a realistic flow chart that can better reflect the actual clinical situation, statistical analysis results, and the data processing of the dropped cases. In the process of acupuncture treatment of chronic pain, patients often drop out due to poor pain relief or adverse events. If there is no corresponding flow chart, the actual effect of acupuncture is often exaggerated. Compared with the flow chart reports in other fields based on the CONSORT 2010 checklist, in trials on postoperative pain management after total hip and knee arthroplasty, it was found that the reporting rate of participants’ flow chart was about 42%, which was slightly lower than our review results.²⁶ Another review of periodontology journals published in 2011–2016 found that the report rate of participants in the flowchart accounted for about 60%–79%, which was less different from our results.²⁷ Therefore, it is essential to select the appropriate flow chart according to the CONSORT participant flowchart.

A large number of meta-analyses have evaluated the impact of the blind method on the estimation of the intervention effect.²⁸ Through the comparison of blinded and unblinded trials, it has been found that there are large differences between the studies.²⁹ Modern acupuncture-related researches mostly follow the principle of separation of implementer, subject, and evaluator. If blinding is not assessed, we may delude ourselves as to exactly what information we gain from incorporating a placebo comparison.³⁰ At present, there is a lack of unified quantitative blind method quality evaluation method.³¹ Up to now, the evaluation of the blind method completely depends on the information provided by experimental publications. In these publications, the report of the blinding method is very common, but there are few reports on whether the blinding is successful or not.³² This is similar to the results reported according to the Pain-specific CONSORT item 4 and TIDieR-placebo item 13. Although the description of the blind method accounts for 77.03%, the success rate of the blind method only accounts for 29.73%. Clearly, the lack of successful blinding can cause bias on observed estimates of effect. Therefore, it is important to make the evaluation process transparent by the details of the original report.

The highest reported STRICTA items were treatment regimen (eg frequency and duration of treatment), details of needling, and acupuncture rationale, as well as practitioner background. The weakest reported items were related to the depth of insertion (Item 2c) and the setting and context of treatment (Item 4b). Fan et al showed that the effect of acupuncture analgesia is closely related to the safe needling depth, which had been optimized to enhance the analgesic effect.³³ To avoid the occurrence of adverse reactions or complications, it is important to investigate the actual acupuncture depth of each acupoint for a well-designed clinical trial.³⁴ Although the World Health Organization (WHO) has provided standard positions for acupuncture points, it has not given detailed guidance on the depth of acupuncture.³⁵ It is necessary to describe the depth of acupuncture in more detail based on the items of STRICTA, which will provide a solid foundation for providing reasonable acupuncture depth during acupuncture treatment in evidence-based medicine. Similarly, for sham acupuncture, there are different forms of puncture and non-puncture, which leads to some differences in the clinical results by sham acupuncture as a control.³⁶ Therefore, for sham acupuncture, whether acupuncture was punctured and the depth of acupuncture need more detailed and accurate reported. Item 4b includes the training and guidance of acupuncture therapists, as well as the necessary explanation and communication for patients. The evaluation of the clinical efficacy of acupuncture for chronic pain mostly uses the subjective patient-reported, such as pain degree, which is easily affected by expectations and non-specific effects (such as the placebo effect).³⁷ Given the complicated characteristics of acupuncture, the impact of the training of researchers on the treatment results cannot be ignored. First of all, the difference in the educational background of acupuncturists and the time limit of clinical work will bring about differences in their acupuncture skills. All acupuncturists participating in the trial should be trained with sufficient intensity to ensure the integrity of the intervention and treatment.³⁸ Secondly, the doctor-patient communication during the acupuncture process is likely to play an important “placebo” role and affect the active participation of patients, thereby improving the results.³⁹ Therefore, doctor-patient communication should be standardized during the implementation of RCT. In addition to accurately explaining pain assessment to participants, unnecessary communication should be reduced, so that the results of different groups in the study can minimize the impact of expectations and placebo effects. Third, RCT should also fully consider the personnel arrangements for future implementation during the research design, and formulate standardized operating procedures (SOPs) according to different tasks. To ensure the smooth progress of clinical monitoring, preparations of detailed SOP must be made before the formal implementation of the trial. Therefore, in the acupuncture RCTs for chronic pain, it is necessary to pay more attention to the report on the training of researchers and the communication of patients.

We found that the reporting quality of sham acupuncture was poor due to the included studies without providing complete descriptions for all TIDieR-placebo items. Different sham acupunctures might have different effects, which can influence the efficacy of interventions. Our findings suggested that an appropriate description of sham acupuncture is indispensable for confirming the efficacy of acupuncture intervention. Among the 13 checklist items, our findings regarding incomplete reporting of the sham intervention procedure, item 3 and item 4 in the TIDieR-placebo checklist failed to explicitly be reported. If elements such as procedures, provider, and location(s) were not completely reported, the possibilities for clinicians and researchers to properly interpret the adherence rate and clinical effectiveness of the intervention were reduced. Items 11 and 12 of the TIDieR checklist concern intervention adherence.⁹ One of the important indicators to evaluate the quality of RCTs and the prognosis of the disease is the complete research data, and the compliance problem will have a certain impact on the data collection.⁴⁰ In acupuncture trials, it is very common that data missing due to poor compliance, which will affect the reliability of research results. An in-depth interview study was conducted to explore the factors associated with the compliance of acupuncture trials.⁴¹ A previous literature review from China found that only one-third of the acupuncture trials in China did not report the withdrawal or loss of follow-up data.⁴² We found that most of the included studies did not describe in detail how much of the prescribed amount of sham acupuncture was consumed. To improve the compliance of patients and the quality of RCTs in acupuncture, it is necessary to report in detail the compliance problem of sham acupuncture based on the TIDieR-placebo checklist.

Strengths and Limitations

As far as we know, the current systematic review was the first time to adopt quantitative methods based on the three checklists to assess the report quality of acupuncture trials from different perspectives, such as RCT study design, acupuncture intervention characteristics, and sham acupuncture, which was more comprehensive than using a single reporting standard to evaluate. This study is based on a strict and extensive search strategy, considering the contradictory conclusions that often appear in RCTs of acupuncture for the treatment of pain, the population included in this study is patients with chronic pain, which is extensive. Through this study, some current deficiencies in RCTs will be reviewed, such as how to evaluate the success of the blind method, whether the statistical plan is appropriate, etc., which can provide good guidance for future clinical trial research design.

Our study has several limitations. Firstly, Although strict search strategies and inclusion criteria have been formulated in the early stage, there may be a certain degree of missing and bias, which may affect the reliability of our results. Secondly, only RCTs published in English were selected from PubMed, Embase, and Cochrane library, which has narrowed the scope of retrieval to some extent and lacked a more comprehensive and confident reflection about the quality of reports in other languages. However, it is found that most of the trials have been included from Chinese databases such as CNKI, CBM, VIP, Wanfang, etc., which guarantees the extensiveness of the included studies to a certain extent. Thirdly, this study was only a preliminary descriptive study, and there was no strict analysis in methodology statistics. In the future, Cohen’s к-statistic should be used to quantify the consistency among reviewers, and further evaluate the relationship between the overall quality of RCTs and related factors.

Conclusions

In general, based on Pain-specific CONSORT and STRICTA, the overall reporting quality of acupuncture RCTs for chronic pain was moderate. It must be emphasized that the poor quality reporting of items which includes statistical methods, participant flow, setting, and context of treatment, the extent to which treatment was varied. But based on TIDieR-placebo, it is found that the quality of the report of sham control is low, which is related to the late release of the checklist. The descriptions mostly lack information about describing any physical or informational materials used in the sham intervention, sham intervention procedures, and blind method based on the TIDieR-placebo statement. Based on the present findings, future improvement of the descriptions of acupuncture in clinical trials of chronic pain treatment is required.