Assessing the reporting quality in randomized controlled trials of acupuncture for postherpetic neuralgia using the CONSORT statement and STRICTA guidelines

Introduction

Postherpetic neuralgia (PHN) is one of the most common complications of herpes zoster.¹ It is defined as neuropathic pain persisting for 90–120 days after the acute phase of VZV reactivation.² It is estimated that 10–15% of patients who have shingles will experience PHN.³ However, the incidence of PHN could be as high as 83% among over 50 years old patients.⁴ PHN has been considered a health care problem due to its severity, chronicity,⁵ and the resulting poor quality of life.⁶ Tricyclic antidepressants, such as gabapentin and pregabalin, are recommended as first-line treatments for PHN.⁷ However, a high incidence of adverse events such as dizziness, somnolence, fatigue, and ataxia remains refractory.⁵ Most importantly, the effectiveness of pharmacologic therapies remains limited,⁸ with satisfactory analgesia in only 50% of patients.^9,10

Owing to the significant health risks of PHN and the limitations of currently available conventional therapies, complementary and alternative medicine (CAM), due to their significant analgesic effects,¹¹ have received much attention worldwide.

As a widely-used CAM, acupuncture is generally considered to be safe and effective in different chronic pain conditions.¹² Recent years have seen many clinical trials reporting the effects of acupuncture in PHN management.¹³ Among these clinical trials, randomized controlled trials (RCTs) are considered to have the optimal study design.¹⁴ A meta-analysis of these RCTs reported that acupuncture is a safe and effective procedure for the management of PHN.¹³ However, inappropriate study methodology with incomplete reporting is the Achilles’ heel of those RCTs, and significantly affect their reliability and validity.¹⁵

The Consolidated Standards of Reporting Trials (CONSORT) statement, developed in 2001 and revised in 2010,^16,17 provides guidelines for the identification of biased results and improvement of trial reporting. Its purpose is to facilitate the assessment and interpretation of RCTs.¹⁸ While as the Standards for Reporting Interventions in Controlled Trials of Acupuncture (STRICTA) guidelines, drafted in 2001 and revised in 2010,^19,20 serve as an extension of the CONSORT statement. The combination of these two sets of guidelines has been used worldwide to assess the completeness as well as the transparency of RCTs for reporting acupuncture interventions.^21–23

This study not only assesses the extent to which current RCTs examining acupuncture for PHN comply with the CONSORT statement and the STRICTA guidelines but also explores factors associated with reporting quality. The main goals are to reveal the present compliance status of RCTs on the use of acupuncture for PHN with the above guidelines and to provide ideas for facilitating and improving reporting quality of RCTs in that field for better and reliable medical decisions.

Methods

Included studies

All RCTs examining the effect of acupuncture interventions for PHN were assessed for inclusion. Non-randomized, cross-over clinical trials, case-control studies, retrospective studies, animal experiments, case reports, and reviews were excluded.

Participants

All study subjects with a clinical diagnosis of PHN were included regardless of age, sex, occupation, or other demographic factors. PHN was clinically diagnosed according to the observed signs and symptoms (eg, pain persisting for three months or more after herpes zoster infection) or recognized diagnostic standards.^24,25

Interventions and comparisons

Acupuncture is defined as manual or electronic stimulations due to filiform needle penetration of the body, scalp, or auricular acupoints regardless of diameter, length, manufacturer, or material.^26,27 Only studies using acupuncture alone or in combination with other interventions (eg, acupuncture-related treatment, drugs, Chinese herbal medicine, physical therapy) were included. Studies testing non-filiform-needle-penetration (eg, acupoint injection, bleeding with plum-blossom needles) as a primary intervention were excluded. Any additional comparisons in the control groups were considered as acceptable.

Search strategy

The following six databases China National Knowledge Infrastructure (CNKI), Wanfang, Chinese VIP Information (VIP), Cochrane Library, Embase and PubMed, were searched from the date of their inception to December 2018. Combinations of the MeSH terms “neuralgia, postherpetic”, “acupuncture therapy” and text words “postherpetic neuralgia”, “herpes zoster”, “shingles”, “acupuncture”, “electroacupuncture”, “pricking”, “needling”, “meridian”, “acupoint”, “randomized controlled trial”, and “RCTs” were searched. Languages were restricted to Chinese and English.

Data extraction

Duplicate records were initially identified and removed by one of the reviewers using EndNote X8. The abstracts screening was independently performed by both reviewers (Luo and Chen) looking for potentially eligible studies. They subsequently read the full texts for potential inclusion. Disagreements, if any, were resolved through mediation with a third reviewer (Lu). A fourth reviewer (Pei) extracted the relevant information such as eligibility criteria, publication year, number of centers, and sample size.

Evaluation of reporting quality

Overall reporting quality rating

Two reviewers Liu and Zeng used the CONSORT and STRICTA statements to independently assessed the reporting quality of included RCTs. Cohen’s κ-statistic was calculated to evaluate the degree of agreement between the two evaluators. A κ of 0.40 or lower, between 0.40 and 0.60, between 0.60 and 0.80 and from 0.80 to 1.00 were considered as poor, moderate, substantial and “good” or “perfect” agreements respectively. Cohen’s κ-statistical analysis was performed using STATA 13.0.

The overall quality score (OQS) was used for the comprehensive quality evaluation of the included RCTs. Thirty-seven items were scored based on the CONSORT 2010, and seventeen items were scored based on the STRICTA statements. For the calculation of the OQS, one point was scored if the item related information was stated, and zero points if it was either not addressed or unclear.

OQS was used as a dependent variable to explore factors relevant to the reporting quality. Only significant variables (p≤0.10) in the univariate analysis were included in the multivariate regression model (p≤0.05).²⁸

Key methodological items rating

Randomization, allocation concealment, blinding, baseline characteristics, and intent-to-treat (ITT) analysis are the five major methodological items included in the 2010 CONSORT guidelines.²⁸ They are each related to factors that could potentially cause bias and were consequently evaluated in each study. A combined key methodological index score (MIS) was calculated for each study by adding the scores of these five factors. One point score was assigned for each reported item and a score of zero when the subsequent item is not reported or uncertain.

MIS was used as an outcome variable in the regression analysis to search for factors related to methodological quality. We regarded MIS as count data and relied on an ordinal regression model. Linear and ordinal regression analyses were performed using SPSS 21.0.

Results

Reports selection

The initial search identified a total of 3,609 relevant reports among which, 670 were retained after titles and abstracts screening. A total of 137 eligible RCTs were ultimately extracted for analysis (Additional file 2). Search terms for each database are customized and shown in Additional file 1. The general characteristics of the included RCTs are indicated in Additional file 3. Figure 1 outlines the RCTs search and selection process.

Figure 1 Flow chart of the performed search and selection process.

Characteristics of included trials

Thirty-two (23.4%) and 105 (76.6%) RCTs were published in the early periods (1983–2009) and the late periods (2010–2018), respectively. All included articles were written in English (2%) and in Chinese (98%). The sample size ranged from 24 to 211 (median: 68). One hundred and thirty-two (96%) were single-centered trials with 23 (17%) being funded. No trials reported a registration or study protocol (Table 1).

Table 1 General characteristics of included RCTs

Reporting quality evaluation

Overall reporting quality rating using the CONSORT and STRICTA checklists

As presented in Table 2, the analysis based on the CONSORT statement showed a sub-optimal reporting quality for the included RCTs. Among the 37 items of the CONSORT checklist, only ten were sufficiently reported (reported in more than 70% of trials). The remaining fifty-five items were poorly reported (reported by fewer than 5% of trials). Items related to the “structured summary”, “trial design”, “eligibility criteria”, “settings and locations”, “interventions”, “outcomes”, “recruitment and follow-up”, “participants randomly assigned”, “participants included in analysis” and “results for outcomes” sections were well reported while as those in the “title”, “changes to methods”, “changes to outcomes”, “sample size”, “interim analyses”, “type of randomization”, “implementing”, “random allocation sequence”, “blinding”, “additional analyses”, “reasoning for ending”, “absolute and relative effect sizes”, “ancillary analyses”, ‘registration ‘and “full trial protocol” sections were poorly reported. The overall median OQS was 11, ranging from 3 to 29. Item number 5 (Table 2) had a poor degree of agreement, while as 11b had a “moderate” degree of agreement. Items 2a, 2b, 20 and 22 had a “substantial” degree of agreement. The remaining others had a “good” or “perfect” degrees of agreement.

Table 2 Compliance of reporting to the CONSORT checklist

The analysis indicated that the STRICTA checklist items are mostly well reported. Among the 17 items, eight had an excellent report rate (reported in over 70% of trials), and only three items were incompletely reported (reported in fewer than 20% of trials). Items such as “acupuncture style”, “reasoning for treatment”, “names of points”, “needle stimulation”, “needle retention time”, “numbers of treatment session”, “frequency and duration of treatment session”, and “details of other interventions” had positive rates as higher as 70%. Meanwhile, Items under the extent to which treatment was varied, setting and context of treatment, and qualification of acupuncturists sections had below 20% positive rates. Details related to acupuncture interventions were sufficiently described in most RCTs. The overall median OQS was 9, with a minimum of 2 and a maximum of 15. Most of the items had “good” or “perfect” degrees of agreement (Table 3).

Table 3 Reporting compliance to the STRICTA checklist

Factors associated with reporting quality

The univariate analysis results indicated that the numbers of centers, the publishing year and the funding source were associated with a higher total score based on the CONSORT statement. After adjustment, the author’s affiliation, the year, the center, and the funding source were included in the multivariate linear regression analyses. Results showed that RCTs published after 2010 (β coefficient 2.394, 95% confidence interval (CI) 1.168–3.620) and supported by a funding source (β coefficient 4.456, 95% CI: 3.009–5.903) remained independent and were significant predictors of overall reporting quality. As shown in Figure 2, mean scores of studies after 2010 are higher than those of before 2010. The CONSORT statement OQS mean OQS increased by 2.394 for articles published after 2010, and by 4.456 for those supported by a funding source (Table 4).

Figure 2 Distribution of RCT scores with complete reporting of the CONSORT statement items. The blue and red bars represent the before and after 2010 RCT score means according to the CONSORT statement, with 95% CI.

Table 4 Factors associated with OQS based on the CONSORT statement

The univariate analysis showed that only funding source (β coefficient 1.305, 95% CI 0.219–2.391) was associated with an increase in STRICTA guideline related OQS. Furthermore, the STRICTA guideline OQS mean rose by 1.305 for articles with funding sources (Table 5). As shown in Figure 3 and according to the STRICTA guideline based assessments, there are no significant differences between mean scores of studies before 2010 and those after 2010.

Table 5 Factors associated with OQS based on the STRICTA checklist

Figure 3 Distribution of RCT scores with complete reporting of the STRICTA checklist items. The blue and red bars represent the before and after 2010 RCT score means with 95% CI according to the STRICTA checklist.

As for the MIS, the ordinal regression analysis showed that no variables were associated with better methodological quality (P>0.05).

Discussion

To our knowledge, this study is the first to systematically assess the extent of compliance of RCTs on the use of acupuncture for PHN with reporting guidelines.

One hundred and thirty-seven trials were assessed, indicating that a considerable number of essential items are incomplete or omitted. This may substantially impact the internal validity assessment and the applicability of the trial results. These findings are consistent with results of previous studies examining RCTs’ adherence to the CONSORT and STRICTA statements in various fields including acupuncture.^21,22

The results showed that only ten of the items on the CONSORT statement has sufficient data, while fifteen others were incompletely reported. The under-reported items, similar to previous reports,²⁹ are particularly those related to key methodological domains such as the type of randomization, the generation of random allocation sequences, the allocation concealment, and blinding. These items are crucial for bias risk assessment and subsequently, the weighing of effects reliability and adaptability. Indeed, previous studies have indicated an overestimation of treatment effects in trials with inadequate blinding or inadequate randomization concealment.³⁰ Additionally, considering the subjectivity and susceptivity of pain assessment, rigorous designs and better implementation of interventions are necessary for the avoidance of result exaggeration or overestimation.³¹

Consistent with previous studies,^27,31 the assessment also showed that the most under-reported items are those in the following sections: changes to the method, calculation of sample size, losses and exclusions, other analyses, registry, and protocol. Standard adherence requires that any changes made to the initial method or outcome should be demonstrated and explained to guarantee the value and validity of the study results. Moreover, sample size calculation is required for the statistical consideration of differences in therapeutic effects and clinical significance. Even though losses and exclusions might result in deviations from the predetermined number of participants, measures (intention-to-treat, ITT)³² can be taken to lower the occurrence of related bias. Additionally, subgroup or adjusted analyses could help eliminate several confounders by providing supplemental information. However, to avoid misleading or confusing readers, results of subgroup analyses should not be reported as substitutes to the main results. Therefore, knowledge of the pre-established RCT protocols is essential for a clear understanding of the reasons and purposes of any additional group analysis.

The STRICTA checklist assessment indicated a moderate reporting quality with the majority of the items being accounted for at the exception of those related to the following sections: the extent to which treatment varied, the number of needles, response sought, setting and context, acupuncture rationale and practitioner background. As shown by previous studies conducted by Zhuang and You,^28,31 details describing acupoints location, needle retention time, and needle stimulation are crucial and provide valuable references for clinicians. What’s more, items reporting the reasoning behind the choice of treatment or style of acupuncture are also important for the determination of suitability or future clinical implementation of that technique. Unfortunately, as seen in Table 5, items related to contextual factors (items 4b and 5) and practitioner qualifications were severely lacking. This may have a significant impact on the validity and safe implementation of those interventions. In practice, acupuncture is a practitioner-dependent, experience-required, and non-pharmacological intervention. This means inadequate reporting of items or details related to the manipulation and technique is a barrier for future replication. Thus, it is never too late to emphasize a complete adherence to the STRICTA guidelines during design, implementation, and reporting of acupuncture RCTs to guarantee the generalization and the reliability of the obtained results.

In compliance with earlier publications,²⁸ the regression analysis suggested that post-2010 publication years represent an independent predictor of high reporting quality.²⁸ This is indicative of an increase in awareness and adoption of the CONSORT statement since it was drafted and revised. This might be partially attributed to the emphasis made on RCTs designs and implementation, and stricter peer-review processes during the recent nine years. However, no correlation was identified between the year of publication and OQS in the STRICTA guidelines evaluation. One possible explanation might be that authors are accustomed to a flexible style for the description of their study due to the diversity in style, the complexity, and specificity of acupuncture manipulation. Compared to the CONSORT statement, the STRICTA checklists are insufficiently emphasized among researchers and journal editors, rendering their popularization and implementation problematic.

Additionally, in agreement with previous reports,³³ the analysis of both CONSORT and STRICTA guidelines suggested that clinical trials with funding had better study design and higher reporting quality. This is because they have sufficient financial support and competent members.

Strengths and limitations of the current study

To our knowledge, this study is the first to perform a systematic CONSORT statement and STRICTA guidelines adherence assessment of available RCTs on acupuncture for PHN management. It is the result of an extensive search and rigorous assessments based on both checklists and provides further evidence of the association between the two guidelines OQS and the methodological quality of the study. Such findings are important and provide insight into the complexity of methodological issues in clinical trials, especially those related to acupuncture and pain, considering the occurrence of conflicting findings among RCTs. For example, how can bias produced by subjectivities in pain measurement be reduced, and whether an insignificant outcome has any clinical implications?

This study has several limitations such as: first, articles published in languages other than Chinese or English were not included. Acupuncture is a well-known pain management option and might have been subject to research in various languages. Therefore, whether the inclusion of those published reports would have altered the results of assessment and analysis remains unknown. Second, as mentioned by a previous study,²⁹ equal weights were assumed during the calculation of the overall quality score (OQS) even though those checklist items may have varying weights. Third, according to available studies,²² more factors such as the author’s expertise and the publication language should have been considered in the linear regression model. However, as shown in File 3, no author was identified as having statistics-related expertise, and only two articles were published in English. Nevertheless, this study achieved its preliminary objective by successfully evaluating the reporting quality of RCTs on the use of acupuncture for the management of PNH.

Conclusion

Our study revealed suboptimal reporting qualities for RCTs using acupuncture for PHN management. Most trials failed to mention crucial methodological data, registration information, the rationale behind the choice of acupuncture or comparable treatments, the acupuncturist qualification, and the treatment context. However, the findings indicated higher reporting quality in RCTs with funding sources or published after 2010. Therefore, more effort is needed for strict adherence to the CONSORT statement and STRICTA guidelines.