Assessment of Completeness of Reporting in Randomized Controlled Trials of Acupuncture Therapy for Chronic Obstructive Pulmonary Disease

Introduction

Chronic obstructive pulmonary disease (COPD) is a major chronic disease that seriously endangers human health,¹ with high incidence, mortality and prevalence.² Global prevalence is approximately 11.7%.³ By 2030, more than 4.5 million persons are estimated to die from COPD annually,⁴ which has a severely negative effect on lung function, quality of life, and economic burden of patients.^5,6 Acupuncture therapy (AT) for COPD has caused widespread concerns over recent years. Meta-analysis indicated that acupuncture has better efficacy and advantage in COPD, which can delay the decline of lung function and enhance exercise endurance and quality of life.^7,8 However, the methodological quality of included original research were relatively poor, and the reliability of their conclusions remains to be further verified. In randomized controlled trials (RCTs), which are regarded as the most appropriate studies for evaluating the efficacy of interventions,⁹ the quality of methodology and reporting is critical for assessing internal authenticity and external validity. Nevertheless, studies that have systematically evaluated the completeness of reporting in RCTs of AT for COPD are inadequate. Therefore, based on the 2010 revision of the Consolidated Standards of Reporting Trials (CONSORT) statement¹⁰ and Standards for Reporting Interventions in Clinical Trials of Acupuncture (STRICTA) guidelines revised in 2010, which acted as an extension of the CONSORT statement,¹¹ this study aimed to evaluate the completeness of reporting in RCTs of AT for COPD, to establish references for high-quality clinical research in the future.

Materials and Methods

Search Strategy

These eight databases were searched for articles published from their inception to May 8, 2021: PubMed, Embase, Cochrane Library, Web of Science, China National Knowledge Infrastructure (CNKI), Chongqing VIP (CQVIP), WANFANG Data, and China Biology Medicine (CBM). We combined the search pattern of Medical Subject Headings (MeSH) and free terms and formulated detailed search strategies for each database. The search terms were adjusted appropriately according to diverse databases to find all eligible studies. The search terms were related to three keywords: “chronic obstructive pulmonary disease”; “acupuncture therapy”, which included “acupuncture”, “needling”, “moxibustion”, “electroacupuncture”, “abdominal acupuncture”, “floating needle”, “warming needle”, and “auricular acupuncture”; and “randomized controlled trials (RCTs)”.

Inclusion and Exclusion Criteria

The following inclusion criteria were applied. (I) Types of patients: Patients had a definite clinical diagnosis of COPD, with spirometry indicating forced expiratory volume in 1 second (FEV₁)/ forced vital capacity (FVC) <0.70 after bronchodilator inhalation.² (II) Types of interventions: The control group (CG) received placebo acupuncture, conventional therapy, and other non-pharmaceutical therapy (including acupoint application and moxibustion). The experimental group (EG) received AT alone or AT combined with CG. AT was the main therapy, without any restriction of acupuncture genre, method, acupoint selection, and needle retention time. (III) Types of research: RCTs of AT for COPD.

Exclusion criteria included (I) non-invasive EG intervention techniques (for example, acupoint pressing, acupoint embedding, auricular embedding, laser acupuncture, and transcutaneous electrical nerve stimulation), (II) duplicate publications, and (III) conference abstracts and studies with incomplete or inaccessible data.

Literature Selection

Two investigators independently selected literature to determine which articles were eligible. The literature were merged by Endnote X8 software, and duplicated literature were removed. Thereafter, potential eligible studies were identified by scanning titles and abstracts. Finally, after reviewing full-text articles, literature that did not meet the inclusion and exclusion criteria were excluded and moved into a specified folder with various types. Admittedly, a third investigator acted as an arbitrator to resolve disagreements by consensus.

Assessment of Completeness of Reporting

Considering that binary outcomes (17b) in CONSORT do not apply to all trials and STRICTA, as an extension of CONSORT, has more detailed and specific standards that are designed to evaluate the completeness of acupuncture intervention reports. So two items including 17b and details of the interventions intended for each group (5) were excluded. The comprehensive reporting assessment was performed by overall quality score (OQS).¹² OQS is the total score for RCT in reference to the specific description of 35 items of the “2010 CONSORT statement”¹⁰ and 17 items of the “STRICTA guidelines”.¹¹ Each item will be defined “yes” or “no” depending on whether the author explicitly stated or not, counting as “1” or “0” point, respectively.¹² Each article was reviewed by two investigators with similar training. Any inconsistencies in opinion were resolved by consensus.

Data Extraction

We established a standardized form using Microsoft Excel 2019 for data extraction. The form contained basic characteristics of included RCTs, such as title, first author, affiliation, publication year and language, sample size, research center, experimental, comparator, and duration of treatment. The results of data extraction were checked to avoid omissions; a third researcher served as an arbitrator to reach a consensus and resolve any arguments that occurred.

Statistical Analysis

Cohen’s κ-statistic was calculated to assess the consistency between two researchers using the SAS 9.4 software (SAS Institute Inc., Cary, NC, USA). A consistency of 1.0 was defined as “perfect”, 0.8–1.0 as “good”, 0.6–0.8 as “substantial”, 0.4–0.6 as “moderate”, 0.2–0.4 as “low”, and 0.0–0.2 as “poor”.¹³ The scores and percentages assigned according to the CONSORT statement and STRICTA guidelines were recorded by Microsoft Excel 2019. All factors associated with completeness of reporting were determined using SPSS 25.0. Based on previous studies, with OQS as the dependent variable and pre-set characteristics as the independent variables, only variables with statistical significance (P ≤0.10) in the univariate analysis were analysed with multivariate linear regression analysis (P ≤0.05).^12,14

Results

Literature Search and Selection

A total of 1303 literature were retrieved through the specific search strategies of each database; 446 duplicate literature were excluded, 760 literature were excluded after scanning titles and abstracts, and 53 literature were excluded after reviewing full-text articles. Finally, 44 RCTs were included and extracted for further analysis. The selection procedure is detailed in Figure 1.

Figure 1 Flow diagram of study selection.

Characteristics of Included RCTs

A total of 44 RCTs were reported during 2004–2021. Among them, 41 RCTs were conducted in China, 2 in Japan, and 1 in Ireland. The number of publications gradually increased annually, and most literature were published in the last decade. The sample size ranged from 30 to 150 individuals, with a median of 60 individuals. The proportion of single-center trials was 41 (93.18%). 20 articles reported specific funding sources, and only 2 articles mentioned trial registration. Among the 44 RCTs, 37 (84.09%) RCTs were published in Chinese and simply 7 (15.91%) in English. For the first author’s affiliation, 29 (65.91%) of them were hospitals. All RCTs detailed the eligibility criteria for participants, among which 34 RCTs simultaneously described diagnostic, inclusion, and exclusion criteria. The duration of treatment ranged from 3 days to 3 months. Manual acupuncture, electroacupuncture, warm acupuncture, and ear acupuncture were the main types of interventions. The detailed characteristics of included studies are shown in Table 1, Figure 2, and (Table S1).

Table 1 Characteristics of Included RCTs

Figure 2 Number of RCTs published per year from 2004 to 2021.

Completeness of Reporting Based on CONSORT Statement

The CONSORT statement consists of six domains: title and abstract, introduction, methods, results, discussion, and other information. These domains are assessed using 35 items. None of the literature fully complied with the CONSORT statement, and deficiencies were identified in the reporting status of most items. Among the six domains, the introduction domain had the highest average reporting rate of 78.41%, whereas the other information domain had the lowest rate of 16.67%. Among the 35 items of the CONSORT statement, 10 items were fully reported with rates above 70%: structured abstract (1b), background and objectives (2a, 2b), eligibility criteria (4a), settings and locations (4b), random allocation sequence method (8a), statistical methods (12a), recruitment dates (14a), generalisability (21), and interpretation with results (22). In contrast, 11 items were poorly reported at rates less than 5%, of which 8 items are not mentioned changes to methods and outcomes (3b, 6b), interim analyses (7b), additional analyses (12b), reason for trial ending (14b), ancillary analyses (18), harms (19), and protocol (24). The OQS of the 44 RCTs ranged from 3 to 26 with a median of 10. Consistency in the two researchers’ assessments of the CONSORT statement was assessed in this review, and multiple items were considered as “good”, “substantial”, “moderate”, and “perfect” with a Cohen’s kappa coefficient > 0.5. The assessment of completeness of reporting based on the CONSORT statement is shown in Table 2.

Table 2 Completeness of Reporting Based on the CONSORT Statement

Completeness of Reporting Based on STRICTA Guidelines

The STRICTA guidelines involve six sections and amount to 17 items. They mainly assess acupuncture rationale, details of needling, treatment regimen, other components of treatment, practitioner background, and comparator interventions. All literature were not entirely compliant with the STRICTA guidelines. Among the six sections, the average reporting rate of treatment regimen was the highest, reaching 97.73%, whereas practitioner background was the poorest with a rate of 4.55%. 10 items were adequately reported with rates > 70%; they included the style of acupuncture (1a), reason for AT (1b), acupoints names (2b), needle stimulation (2e), needle retention time (2f), number of treatment sessions (3a), frequency and duration of treatment sessions (3b), details of administered interventions (4a), and comparator interventions (6a, 6b). To our satisfaction, merely 3 RCTs were incompletely reported with a rate < 20%; they were number of needles (2a), setting and context of treatment (4b), and practitioner background (5). Considering the 44 RCTs, the OQS ranged from 7 to 14, with a median of 11. The Cohen’s kappa coefficients of almost all items were considered as “good”, “substantial”, and “moderate”, indicating that the evaluation of STRICTA guidelines had a considerable agreement between two researchers. The assessment of completeness of reporting based on STRICTA guidelines is detailed in Table 3.

Table 3 Completeness of Reporting Based on the STRICTA Guidelines

Factors Associated with Completeness of Reporting

Regarding the CONSORT statement, univariate analysis revealed that publication language, trial registration, and funding source were tied with higher OQS. After adjustment, these above three factors were used to construct a multiple linear regression model. The results indicated that there were significant differences in the impact of RCTs with publication language (β coefficient: 6.432, 95% CI 3.202 to 9.663, P <0.001) and funding source (β coefficient: 3.159, 95% CI: 1.045 to 5.273, P =0.004) on the OQS, which remained independent and significant predictors of completeness of reporting. As shown in Figures 3 and 4, the mean scores of most items in trials with publication of English language and funding source were higher than those with Chinese language and without funding source. Specifically, the mean OQS increased by approximately 6.432 for publications with English language and 3.159 for publications with funding source. However, no variables associated with the STRICTA guidelines were identified (P >0.05). The factors associated with completeness of reporting are shown in Tables 4 and 5.

Table 4 Factors Associated with OQS Based on the CONSORT Statement

Table 5 Factors Associated with OQS Based on the STRICTA Guidelines

Figure 3 Distribution of RCT scores with and without funding source based on the CONSORT statement.

Figure 4 Distribution of RCT scores with publication language in English and Chinese  based on the CONSORT statement.

Discussion

The CONSORT statement and STRICTA guidelines were applied to evaluate the completeness of reporting in RCTs of AT for COPD. The completeness of reporting of 44 RCTs on AT for COPD was unsatisfactory; this finding was consistent with those from studies on compliance of acupuncture interventions for diverse systems with these two guidelines.^15–17 A systematic review of the effect of STRICTA and CONSORT on reporting of RCTs on AT revealed that although the publication of the guidelines has improved the completeness of reporting to some extent, completeness of reporting of sample size estimation, generation of random sequence, allocation concealment, blinding and practitioner background remained poor.^18,19 Similarly, the reporting rates in the domains of methods, results, and practitioner background were low, mainly in the aspects of trial design, sample size estimation, generation of random sequence, allocation concealment, blinding, registration, practitioner background and so on. According to the multivariate linear regression analysis, publication language and funding source were independent factors that improve the completeness of reporting in RCTs of AT for COPD.

From the 35 items of the CONSORT statement, items that were not been fully reported were mostly related to the methods domain, mainly manifesting in type of randomization, allocation concealment, and blinding. The generation of accurate random sequence and sufficient allocation concealment can avoid selecting and confounding bias to the greatest extent. Hence, a view prevails among some reports that any study with unclear methodological evaluation was prone to exaggerate the outcome effect value by at least 40%, triggering a greater heterogeneity of results.^20,21 In contrast to allocation concealment, although concealment of random sequence in RCTs is feasible, blinding is barely possible in certain trials,^22,23 just as this study struggled to blind both researchers and participants, stemming from the particularity of implementation of AT. Only participants were blinded, which was consistent with the practice of several studies.²⁴ Probably, this practice may be responsible for insufficient reporting of the methodology section.

In addition, similar to previous studies, this study had insufficient or unreported items in the following parts: sample size estimation, changes to methods and outcomes, interim analyses, additional analyses, harms, registration, and protocol. Scientific and reasonable estimation of sample size recommended by CONSORT can avoid false negative results owing to minimized sample size, and waste of resources induced by excessive sample size.^25,26 However, only three (6.82%) RCTs mentioned, whereas none explained interim analysis. Considering that part of the literature on AT for COPD are exploratory trials with small sample size, which unable to be calculated due to lack of basic data, so the reporting rate of sample size estimation was low. In addition, sample size was mostly calculated by primary outcome to ensure sufficient assurance of the primary outcome analysis, but most RCTs failed to clearly distinguish between primary outcome and secondary outcome. Most RCTs detailed relevant statistical methods. However, none made a description for additional analyses, which could indirectly exert a drastic reduction in the credibility and accuracy of results and conclusions. If intention-to-treat (ITT) analysis is not performed, the direct exclusion of dropouts would disrupt the balance between groups, generate bias, and exaggerate the efficacy of trials.²⁷ Worst of all, despite 15 articles reporting losses and exclusions, only 2 (4.55%) articles adopted ITT analysis. Although AT is a safe treatment, some mild and rare severe adverse events may occur.²⁸ In this review, 3 articles mentioned mild adverse events, such as bruising, pain, dizziness, and erythrasma, which were relieved within a short time. Registration regulates clinical research and reduces the risk of selective outcome reporting and publication bias. Besides, researchers may estimate standardization, accuracy, and completeness of the trial implementation process based on a registered protocol.^29,30 The past few years have witnessed an increase in the number of registrations, but merely 2 (4.55%) articles mentioned, and none reported specific access to full trial protocol. This may reflect the lack of awareness among researchers in the registration of trials on AT for COPD. Consequently, the transparency of clinical trials and limited access to trial details for readers would be adversely affected.

For the assessment of STRICTA guidelines, except for the three items, the overall quality was relatively adequate. In the domain of acupuncture rationale, the reporting of style of acupuncture and reason for AT were sufficient, which were consistent with those of relevant studies.^{12,13,15,31–33} By providing the style and reason for AT, it has great value for the applicability and clinical implementation of acupuncture. A view prevails among some reports that the effective key factors of AT mainly involve acupoint names, needle stimulation, retention time, and needle type.^34,35 Similar to previous reports,^{12,16,17,31,32,36} the report of acupoint names, needle stimulation, and needle retention time was satisfactory. The greater part interpreted needle response dominated by De Qi. Similarly, most literature precisely described the comparator and explained the rationality of setting control, which will be beneficial in reducing bias and increasing reliability of results. As is often recognized, the manipulation, qualification, and professional skill of acupuncturists have a close bearing on the efficacy of acupuncture.³⁷ Worse still, consistent with most studies,^12,15,33,38 the reporting rates of practitioner background and setting and context of treatment were severely deficient. They may have a certain effect on the accuracy and specific implementation of interventions. Additionally, it signifies that insufficient reporting may be a major trouble for future repetition. Therefore, it is strongly suggested that researchers should put high premium on reports in this aspect, so as to improve the efficacy and decrease the occurrence of adverse events.

The completeness of reporting based on CONSORT and STRICTA is almost the same as that in previous literature because of analogously defined reporting evaluation criteria. However, compared with the study of Fernández-Jané et al,³³ significant differences exist in the report of certain items according to the STRICTA guidelines. We preliminarily consider that the reporting standard, according to Fernández-Jané et al,³³ may be inconsistent with that of the previous study, which is divided into fully reported, partially reported, and unreported. Another more important point is that CONSORT and STRICTA guidelines describe the details of checklist, and do not clearly define the basis of judgement for reported and unreported. The binary classification criteria of reported and unreported, defined by most previous studies, were adopted. Fernández-Jané et al³³ pointed out that the actual number of treatments almost never explicitly in all trials, which was quite different from our findings. However, consistent with several studies,^{12,13,15,16,31,32,36} 2 items in the treatment regimen domain were adequately reported. Our assessment standard was that any literature describing the planned number of treatment sessions was considered reported, regardless of the actual number of treatments received by participants. The completeness of reporting of acupoint names or locations also varied from the study of Fernández-Jané et al,³³ mainly due to inconsistent assessment criteria of each study. However, the similarity of assessment results between these two items and most literature^{12,13,15,16,31,32,36} is remarkable, especially the name of acupoint with the highest reporting rate (100%), which is consistent with findings from several literature.^13,32,36

By exploring the factors associated with completeness of reporting, we draw a conclusion that publication language and funding source can be regarded as independent and significant predictors of high OQS based on the CONSORT statement. Consistent with previous reports,^12,14 RCTs with funding source have better trial design and satisfactory reporting. Therefore, studies on acupuncture intervention in COPD should attach importance to funding. In addition, unlike in this study, earlier studies^12,13 have shown that year of publication was significantly correlated with reporting quality. Initially, these two points were considered: included studies were mostly published after 2010, perhaps substantiating the point, and lacked sufficient awareness and standardized training on the reporting standards. Unfortunately, no factor associated with OQS was found in the evaluation of STRICTA guidelines, which coincided with earlier studies.¹³ The differences in the qualification and experience of acupuncturists, manipulation, acupuncture effect on participants, and implementation details are being considered. In addition, the STRICTA guidelines may not have been highly premiumed by investigators and journal editors, remaining some barriers to popularization and implementation.

Study Limitations

There are still some limitations to be clarified. Despite the application of Cohen’s kappa coefficient in testing the agreement in the assessment of completeness of reporting, subjective factors and potential biases were inevitable. Besides, only RCTs published in Chinese and English were included. Those in other languages were omitted, which probably affected the reliability of results. Validated reporting assessment tools for RCTs are currently unavailable, and this critical challenge needs to be addressed urgently. To minimize this problem, two researchers reviewed the details of each item, then pre-specify the evaluation criteria for each item based on the details of each item, and eventually developed a standardized extraction form to unify the evaluation criteria of the two investigators. In addition, only selecting part features for the regression model, more factors should be considered for a more comprehensive and convincing analysis.

Conclusions

Overall, completeness of reporting in RCTs of AT for COPD was low, and needs to be further strengthened. From our findings, consider the following points: (I) Researchers do not standardize the implementation of RCTs, or certain omissions in reporting. (II) The acupuncture operation technology demands the cooperation between investigators and patients, hence, the existing challenge in conducting blinded studies. (III) In terms of AT implementation, elaborate description of qualifications or background of acupuncture therapists and a unified operation manual are lacking. (IV) Relevant journals do not attach enough importance and review to international quality evaluation standards of published RCTs.

Therefore, by recommendation, relevant journals should enforce the CONSORT statement and STRICTA guidelines, standardize reporting items, strictly review, and compile specifications. To enhance the transparency and quality of clinical trial reports, we advocate for vigorous promotion of registration and increase funding source. Moreover, clinical investigators should strengthen the cognization of relevant statements and guidelines, and strictly regulate design and implementation to improve the quality of clinical evidence.