Reporting quality of stepped wedge design randomized trials: a systematic review protocol

Background

Our reliance on the medical literature to inform practice and recommendations put forward in clinical guidelines demands the need for high-quality reporting of randomized trials. The introduction of checklists, such as the CONSORT statement which was introduced in 1996 and revised in 2010, promotes completeness and transparency in randomized controlled trials (RCTs).¹ However, although endorsed by prominent medical journals such as The Lancet and New England Journal of Medicine, authors still fail to give full disclosure on their RCTs through incomplete reporting in the abstracts or in main text of the final reports, which has potential to result in harm to the patients.² Furthermore, many clinicians in developing countries may not have full access to academic journals and the papers within them, and may rely on publically accessible abstracts to assist their medical judgments.³

Selective reporting of outcomes is a problem that not only affects the reliability of the published study but also reviews that include them.⁴ One study found that 65% of harm outcomes in 122 published journal articles were incompletely reported.⁴ In order to have complete, reliable reporting in all trials, consistency between abstracts, trial registration information, and the full published article is paramount. This study seeks to identify the gaps in reporting consistency and evaluate the relationship between complete reporting of the published article and consistency.

A systematic scoping review of reviews on the adherence of reporting guidelines found that 86.0% of the 50 studies reviewed reported suboptimal levels of adherence to reporting guidelines.⁵ The study emphasizes the need for action on the adherence of reporting guidelines. Furthermore, a paper on the reporting quality of alcohol outcome studies found that journals that had adopted the CONSORT statement showed significant improvements in the reporting of key elements of alcohol outcome studies.⁶ However, previous work focused primarily on standard RCTs, and the quality of reporting of stepped wedge design (SWD) trials is largely left unreported.

SWD is a cluster RCT design that sequentially rolls out intervention to all clusters at varying time points.⁷ At the end of a SWD study, all groups will have received intervention. This study design provides strategic advantage by preventing ethical complications that coincide with RCTs where the treatment is considered to be beneficial, leaving one arm of the study to continue receiving suboptimal therapy. This design also allows for unique statistical analysis, as the duration of intervention for each group is variable.⁷ However, being that it is a newer trial design method, it has not received the volume of a real-world experience that other commonly used RCT designs have been exposed to, which creates possibility for not only methodological errors but also reporting inadequacies. In this study, we will systematically review published trials and protocols utilizing SWD, and assess and report our findings on the quality of reporting of these SWD trial reports.

A summary of all SWD trials was previously published in systematic reviews by Beard et al⁸ and Mdege et al,⁹ both of which provide comprehensive reviews of any SWD randomized trials performed since the inception of this novel methodological approach. Each review provides a detailed summary of the trials included, interventions used, reasoning behind the use of SWD, acceptability of the intervention, as well as variations in analysis techniques. While reporting quality is described in both reviews, neither has taken a systematic approach to examining 1) reporting quality using adapted CONSORT guidelines for cluster RCT; 2) reporting quality among protocols and SWD trial abstracts; 3) the areas impacting completeness of reporting; 4) consistencies between final published trials with original protocols or trial registry profiles; and 5) a specific focus on the information provided for sample size estimation. These are the key gaps we aim to fill with our systematic review. Reviews by Beard et al⁸ and Mdege et al⁹ emphasize on the interventions, acceptability of the study design, and current medical fields uptaking the SWD. Neither review has explored the standards of transparency and consistencies in the reporting of such trials.

Objectives

The objectives of this methodological systematic review are the following: 1) to assess the reporting quality of SWD trial reports based on the CONSORT guidelines¹ or CONSORT extension to cluster RCTs;¹⁰ 2) to assess the completeness of reporting of SWD trial abstracts using the CONSORT extension for abstracts; 3) to assess the reporting of sample size details in SWD trial reports or protocols; 4) to assess the completeness of reporting of SWD trial protocols according to SPIRIT guideline;⁴ 5) to assess the consistency between the trial registration information and what is reported in the final SWD trial reports; and 6) to assess the consistency of what is reported in the abstracts and main text of the SWD trial reports. We will also explore factors that are associated with the completeness of reporting. These factors will include: whether or not the journal has endorsed corresponding guideline, funding source for the trial (industry vs nonindustry), high impact factor, multicentered study, number of authors, reporting of allocation concealment, reporting method of sequence generation, sample size, trial quality, type of intervention (pharmacological vs nonpharmacological), year of publication (before or after CONSORT statement publication in 1996), and origin of report (country of origin).¹⁰

Methodology

Eligibility criteria

Literature search will involve all RCTs employing SWD since 1987 (ie, the date of publication of the first SWD trial). We are interested in completed reports or protocols of RCTs utilizing SWD that are written in English and have been published in a peer-reviewed journal.

Search strategy

An extensive search for SWD trials and protocols will be performed by the lead author on the following electronic databases: MEDLINE, Web of Science, CINAHL, EMBASE, and PsycINFO. The specific search strategies tailored to each database, outlined in Table 1, will be used to identify potential trial reports or protocols that meet the eligibility criteria. Cited materials found in participating articles relevant to the systematic review will be reviewed for potential participation as well. A single reviewer will process the articles in the first stage of the review involving searching the chosen databases using the keywords described previously. After the removal of any duplicates from the preliminary search, the titles, abstracts, and full texts generated will then be reviewed by two independent reviewers to determine study eligibility. When differences arise, reviewers will discuss and come to a consensus; if an agreement cannot be reached between both reviewers, a third party will help provide consensus as to the article’s eligibility. The systematic review will be conducted with adherence to the Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) guidelines to ensure adequate reporting quality.¹² As a methodological systematic survey, this study is not registered with PROSPERO. Please refer to Figure 1 for the PRISMA flow diagram used to guide article selection.

Table 1 Search strategy of electronic databases

Figure 1 The Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) flow diagram for article selection. Abbreviation: SWD, stepped wedge design.

Data collection and outcomes

Data extraction forms will be created from the CONSORT statement and extensions (abstract, cluster RCT), SPIRIT guidelines (protocol), as well as a checklist for what we propose for the transparent reporting of sample size details (Supplementary material; Additional File 1). We will pilot test screening and data extraction forms. Discussions will be held with all parties involved (reviewers and methodologists included in this review) to determine whether the description of tasks and items is clear to reviewers carrying out screening and extraction. A kappa statistic calculation will be performed for each stage of screening to determine the agreement between the independent reviewers. Data from each article will be extracted in duplicate and inputted into an Excel spreadsheet that will contain all the extracted data. In specific, we will be extracting the author name, title of article, article type (trial report, protocol), and all applicable criteria pertaining to the CONSORT and SPIRIT guidelines for the corresponding extraction form. The primary outcomes of the data abstraction and analysis will be to assess and quantify the areas of reporting quality in SWD trials, protocols, and abstracts, and make a conclusion as to whether the reporting of these articles, according to CONSORT and SPIRIT guidelines, is of adequate quality. Consistency between trial registration information and abstracts to the final manuscript will also be assessed and reported as a percentage and count (eg, number of articles with inconsistencies between trial registration and final published paper for X variable).

Additional items identified as both important to SWD but not commonly disclosed according a recent review by Davey et al¹³ will be added to our data extraction forms. These items include: detailed cluster/patient retention rate, specification of the time trend utilized in analytical approach, and discussion effect of variance over cluster and time point evaluation. Transparency in the reporting of sample size–related items has been largely left out of previous reviews evaluating SWD. We aim to focus on many of these items including sample size criterion for statistical significance, desired level of statistical significance, desired power, as well as assumptions made about distribution or variability of the outcome. Please refer to Additional File 1 for a complete list of sample size–related items.

We will contact all corresponding authors for articles identified as eligible to ensure we have abstracted data from their studies correctly. We will provide each author an excel file detailing results from the data extraction of their SWD trial.

Statistical analyses

Descriptive statistical calculations (mean, median, standard deviation, range), as well as percentile calculations of the data, will examine and determine the proportion of SWD articles that are reported suboptimally and will reveal areas of reporting in need of quality improvement. Calculations by year will also determine the mean number of checklist items reported using estimate unadjusted and adjusted differences via two-sample t-test and estimation equations. Statistical adjustments will be made accordingly for each trial to account for potential clustering or similarity in articles published in the same journal. Kappa statistic calculations will be conducted to determine the agreement. We will consider kappa statistic values between 0.40 and 0.59 to represent fair agreement, values between 0.60 and 0.74 to represent good agreement, and values of 0.75 or above reflecting excellent agreement.¹⁴ We will also create summary tables for analysis, which will include objective, outcome, hypothesis, and method of analysis columns. All statistical analyses will be conducted using STATA version 13 (StataCorp LP, College Station, TX, USA).¹⁵ Comparisons to similar studies on reporting the quality of articles will provide a perspective on how the reporting quality of studies utilizing SWD compare to other design methods.

Quality assessment

As this is a review of methodological reporting quality, we will not be assessing trial quality, but rather we will specifically look at the reporting quality of the trials. Clarity and completeness will be assessed in conjunction with the checklists, as unclear or incomplete reporting may affect the reporting quality. Risk of biases will be calculated in duplicate using the Cochrane Collaboration’s tool for assessing the risk of bias in randomized trials.¹⁶ Risk of bias items specific to cluster trials will be evaluated, including whether individual studies accounted for 1) clustering during analysis; 2) clusters omitted during analysis due to center dropout, prognostic imbalance between clusters or individuals; or 3) recruitment bias. Any disagreements will be settled with by discussion among team members and in the event the discussion cannot lead to resolution, a third party will be asked to review the disagreement. Categorical scores of “high”, “low”, and “unclear” will be determined for each paper to decide their risk of bias.¹⁴

Discussion

Accurate and complete reporting of clinical trials is important in the interpretation of studies and the advancement of scientific and medical knowledge. This study expects that many aspects of reporting quality will be omitted as seen from other studies, but as more journals and editors begin to enforce guidelines, reporting quality will improve,¹⁷ and we expect to see that in our yearly data. Suboptimal reporting of clinical trials and protocols results in risk of biases and reduces their value for both clinicians and researchers alike.¹⁸ Once completed, the systematic review will offer information to both readers and the authors of SWD reports as to their reporting quality and will identify commonly underreported aspects of SWD trials. This review will offer unique insight and exposure into the design method and promote the transparency of future SWD articles. This insight will allow physicians and researchers over the world to make proper interpretations of SWD studies, and make appropriate medical and academic judgments of the paper.

Acknowledgments

This work was supported in part by the Department of Anesthesia at McMaster University. The funders had no role in the study design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. Dr. Olga Gajic-Veljanoski is supported by Hamilton Health Sciences Foundation.

Author contributions

AT, BBD, JP, OGV, and LT conceived the research question and developed the protocol for this systematic review. AT and BBD completed an initial literature search to inform the background of this review as well as designed the electronic search strategy. Data extraction forms will be developed and pilot tested by AT and BBD. AT, BBD, JP, OGV, and LT developed the statistical analysis plan. BBD, AT, JP, OGV, and LT contributed equally to writing and revision of the manuscript. The final version of the protocol submitted to Clinical Epidemiology has been read and approved by all authors. All authors contributed toward data analysis, drafting and critically revising the paper and agree to be accountable for all aspects of the work.

Disclosure

The authors report no conflicts of interest in this work.

References

1.		Begg C, Cho M, Eastwood S, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA. 1996, 276(8):637–639.
2.		Glass KC. Toward a duty to report clinical trials accurately: the clinical alert and beyond. J Law Med Ethics J Am Soc Law Med Ethics. 1994, 22(4):327–338.
3.		Hopewell S, Clarke M, Moher D, et al. CONSORT for reporting randomised trials in journal and conference abstracts. Lancet. 2008, 371(9609):281–283.
4.		Chan AW, Tetzlaff JM, Altman DG, et al. SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann Intern Med. 2013, 158(3):200–207.
5.		Samaan Z, Mbuagbaw L, Kosa D, et al. A systematic scoping review of adherence to reporting guidelines in health care literature. J Multidiscip Healthc. 2013, 6:169–188.
6.		Ladd BO, McCrady BS, Manuel JK, et al. Improving the quality of reporting alcohol outcome studies: effects of the CONSORT statement. Addict Behav. 2010, 35(7):660–666.
7.		Brown CA, Lilford RJ. The stepped wedge trial design: a systematic review. BMC Med Res Methodol. 2006, 6:54.
8.		Beard E, Lewis JJ, Copas A, et al. Stepped wedge randomised controlled trials: systematic review of studies published between 2010 and 2014. Trials.2015, 16(1):353.
9.		Mdege ND, Man MS, Taylor Nee Brown CA, Torgerson DJ. Systematic review of stepped wedge cluster randomized trials shows that design is particularly used to evaluate interventions during routine implementation.J Clin Epidemiol. 2011, 64(9):936–948.
10.		Campbell MK, Piaggio G, Elbourne DR, Altman DG. Consort 2010 statement: extension to cluster randomised trials. BMJ. 2012, 345:e5661.
11.		Pratoomsoot C, Sruamsiri R, Dilokthornsakul P, Chaiyakunapruk N. Quality of reporting of randomised controlled trials of herbal interventions in ASEAN Plus Six Countries: a systematic review. PLoS One. 2015, 10(1):e108681.
12.		Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009, 339:b2535.
13.		Davey C, Hargreaves J, Thompson JA, et al. Analysis and reporting of stepped wedge randomised controlled trials: synthesis and critical appraisal of published studies, 2010 to 2014. Trials. 2015, 16(1):358.
14.		Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions, The Cochrane Collaboration Version 5.1.0 [updated March 2011]. 2011. Available from www.cochrane-handbook.org.
15.		StataCorp: Stata Statistical Software: Release 12. College Station, TX: StataCorp LP; 2009.
16.		Higgins JP, Altman DG, Gotzsche PC, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011, 343:d5928.
17.		Hopewell S, Ravaud P, Baron G, Boutron I. Effect of editors’ implementation of CONSORT guidelines on the reporting of abstracts in high impact medical journals: interrupted time series analysis. BMJ. 2012, 344:e4178.
18.		Peters JP, Hooft L, Grolman W, Stegeman I. Assessment of the quality of reporting of randomised controlled trials in otorhinolaryngologic literature – adherence to the CONSORT statement. PLoS One. 2015, 10(3):e0122328.