Ignoring Clustering and Nesting in Cluster Randomized Trials Renders Conclusions Unverifiable [Letter]

Dear editor

Siraneh et al¹ conducted a clustered randomized controlled trial (cRCT) to test the effectiveness of additional counseling and social support provided by women identified as “positive deviants” to promote exclusive breastfeeding (EBF) within a community. However, their statistical methods did not account for clustering and nesting effects and thus are not valid.

In the study, randomization occurred at the cluster level (ie, kebeles), and mothers were nested within clusters. Three of 6 clusters were in the treatment group and 3 were in the control. Of the 260 mothers, 130 were enrolled in the treatment group and 130 in the control, according to kebele. According to the published statistical methods section, the analyses ignored clustering and nesting effects. Because this is a hierarchical modeling environment and individuals within a cluster are typically positively correlated, an individual-level analysis that does not address clustering effects will generate underestimated standard errors and unduly narrow confidence intervals.² That is, the results will overstate statistical significance. Treating the individual-level observations (eg, mothers) as independent units of analysis inflates the type 1 error rate.³

One alternative is calculating the mean observation by cluster and analyzing the data at the cluster level. In this case, this approach would reduce the sample size to 6, the same as the number of clusters, resulting in low statistical power. A valid alternative would be to use multi-level hierarchical modeling, which recognizes the hierarchy in the data and accounts for both lower and higher levels as distinct levels simultaneously. Such multi-level modeling estimation will estimate residuals at mother- and kebele-level separately. Statistical packages exist for such modeling.

Moreover, Siraneh et al calculated their sample size based on the randomization of individual participants.¹ In a cRCT, however, sample size and power calculations should account for the number of clusters per condition, average cluster size, and intra-cluster correlation coefficient (ICC).⁴

We requested the deidentified raw data and statistical code from the authors to reproduce their analyses. Even though we pledged to limit our analysis to testing the hypotheses tested in the article, and the Editor-in-Chief deemed our request “appropriate and reasonable”, the authors were unwilling to share their deidentified raw data and statistical code. They said they needed time to analyze the “remaining data” for publication and that the dataset contained identifiers. Thus, we were unable to reanalyze the data using a valid statistical approach accounting for clustering and nesting effects.

Given the analytical methods used, the evidence presented by Siraneh et al¹ neither supports nor refutes whether a positive deviance intervention affects EBF. The analytical methods were incorrect. All authors have an ethical and professional scientific responsibility to correct non-trivial reported errors in published papers.⁵ Furthermore, the Committee on Publication Ethics (COPE), in which this journal is a member, states

Editors should consider retracting a publication if they have clear evidence that findings are unreliable, either as a result of major error (e.g., miscalculation or experimental error), or as a result of fabrication (e.g., of data) or falsification.⁶