Retraction of COVID-19 Pharmacoepidemiology Research Could Have Been Avoided by Effective Use of Reporting Guidelines
Received 25 October 2020
Accepted for publication 2 December 2020
Published 21 December 2020 Volume 2020:12 Pages 1403—1420
Checked for plagiarism Yes
Review by Single anonymous peer review
Peer reviewer comments 4
Editor who approved publication: Professor Henrik Toft Sørensen
Eric I Benchimol,1– 3 David Moher,4,5 Vera Ehrenstein,6 Sinéad M Langan7
1Department of Paediatrics, University of Toronto, Toronto, Canada; 2Child Health Evaluative Sciences, SickKids Research Institute, and Division of Gastroenterology, Hepatology and Nutrition, The Hospital for Sick Children, Toronto, Canada; 3ICES, Toronto, Canada; 4School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada; 5Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada; 6Department of Clinical Epidemiology, Aarhus University, Aarhus, Denmark; 7Faculty of Epidemiology and Population Health, London School of Hygiene and Tropical Medicine, London, UK
Correspondence: Eric I Benchimol
Division of Gastroenterology, Hepatology and Nutrition, The Hospital for Sick Children, 555 University Avenue, Toronto, ON M5G 1X8, Canada
Tel +1 416813-1500
Fax +1 416813-4972
Email [email protected]
Introduction: Two recent high-profile publications (and subsequent retractions) of pharmacoepidemiology studies reporting the effectiveness and risk of hydroxychloroquine in COVID-19 patients received international media attention. Transparent and complete reporting of these studies could have provided peer reviewers and editors with sufficient information to question the methods used and the validity of results. Since these studies used routinely collected health data, the guidelines for the REporting of studies Conducted using Observational Routinely collected health Data (RECORD) should have been applied to ensure complete reporting of the research.
Methods: We evaluated the two retracted articles for completeness of reporting using the RECORD for Pharmacoepidemiology (RECORD-PE) checklist, which includes the checklists for the STengthening the Reporting of OBservational studies in Epidemiology (STROBE) and RECORD. We compared the proportion of STROBE, RECORD and RECORD-PE items adequately reported using Chi-squared statistics.
Results: In the article published by The Lancet, 29 of 34 STROBE items (85.3%) were adequately reported, compared with 3.5 of 13 RECORD items (26.9%) and 9.5 of 15 RECORD-PE items (63.3%)(χ2 = 14.839, P < 0.001). Similarly, the article published in NEJM reported 24 of 34 STROBE items (70.6%), two of 13 RECORD items (15.4%), and 7.5 of 15 RECORD-PE items (50.0%) (χ2 = 11.668, P = 0.003). Important aspects of the methods unique to research using routinely collected health data were not reported, including variables used to identify exposure, outcome and confounders, validation of the coding or algorithms, a description of the underlying database population and the accuracy of data linkage methods.
Discussion: While STROBE items were generally adequately reported, RECORD and RECORD-PE items were not. Reporting guidelines should be effectively implemented in order for transparency and completeness of research manuscripts, allowing for adequate evaluation by editors and peer reviewers.
Keywords: COVID-19, journalology, peer review, pharmacoepidemiology, reporting guidelines, routinely collected health data
Routinely collected health data, such as registries, health administrative data, and electronic health record data, are increasingly being used for pharmacoepidemiology, clinical and health services research. As the COVID-19 pandemic rages in the world, we observed with great concern that two studies were published and subsequently retracted by The Lancet1 and The New England Journal of Medicine.2 These two pharmacoepidemiology studies used routinely collected health data in the form of the Surgical Outcomes Collaborative (Surgisphere), an international registry derived from data automatically extracted from electronic health records, health and financial administrative data, and a point-of-care data entry process for procedures. Both studies made a significant impact in COVID-19 research, public health, and health policy spheres, and received international attention from the media, as evidenced by Altmetric scores of 24,615 and 3829, respectively, at the time of this writing. The conduct of research that is subsequently retracted represents “research waste”, as initially defined in The Lancet.3 Publication of these manuscripts resulted in the temporary suspension of important COVID-19 clinical trials of hydroxychloroquine by the World Health Organization and others. More importantly, the publication and subsequent retraction of research erodes the public trust in science. While publication of these studies may have been symptomatic of the known limitations of the peer review and editorial process, lack of transparent and complete reporting may have also contributed to this failure. Both retracted studies would fall under the purview of the guidelines for the REporting of studies Conducted using Observational Routinely collected Data (RECORD),4 and more specifically the extension of RECORD for Pharmacoepidemiology (RECORD-PE).5 RECORD and RECORD-PE are extensions to the STengthening the Reporting of OBservational studies in Epidemiology (STROBE) guidelines for non-interventional research,6,7 the overall study design for the retracted studies. RECORD and RECORD-PE more directly address observational research using routinely collected health data, expanding on parts of the STROBE checklist to ensure readers, peer-reviewers and editors recognize the strengths and weaknesses of research using health data. We believe that, had these checklists been submitted for assessment by peer reviewers, inherent weaknesses in the manuscripts would have become apparent, potentially resulting in rejection by the journals. Unfortunately, journals and editors still do not routinely ask peer reviewers to use reporting guidelines as part of their assessment.8 In fact, the pressures from both the scientific community and journal publishing industry resulted in the strong drive to accept COVID-19 articles early in the pandemic, which may have compounded the problem of inadequate peer review and the need for retraction of published articles.9
Most research using routinely collected health data is observational in design. While NEJM endorses CONSORT for reporting of clinical trials, STROBE for non-interventional research is not suggested in the instruction for authors.10 The Lancet endorses use of STROBE,11 but endorsement is insufficient to ensure the appropriate use of reporting guidelines by authors and peer reviewers. In addition, STROBE does not fully convey all intricacies in the methodology of research using routinely collected health data. We previously demonstrated that the reporting of research using routinely collected health data was gravely deficient, even in journals that endorsed STROBE.12 Therefore, RECORD and RECORD-PE were created as extensions to STROBE for observational studies involving such data.
We evaluated both articles in question using the RECORD-PE checklist (Tables 1 and 2), and determined the proportion of adequately reported items on the STROBE, RECORD, and RECORD-PE checklists. We reported the proportion of items on each checklist that were adequately reported, allowing for half-points when an item was partially reported. Chi-square statistic was used to determine differences in the proportion of adequately reported items for each checklist.
Table 1 The RECORD Statement for Pharmacoepidemiology (RECORD-PE)5 Checklist of Items, Extended from the STROBE and RECORD Statements, for the Study Mehra et al, The Lancet, 20201. Red Font Indicates Areas That Were Not Adequately Reported
Table 2 The RECORD Statement for Pharmacoepidemiology (RECORD-PE)5 Checklist of Items, Extended from the STROBE and RECORD Statements, for the Study Mehra et al, N Engl J Med, 20202. Red Font Indicates Areas That Were Not Adequately Reported
In the article published by The Lancet, 29 of 34 STROBE items (85.3%) were adequately reported, compared with 3.5 of 13 RECORD items (26.9%), and 9.5 of 15 RECORD-PE items (63.3%) (χ2 = 14.839, P <0.001). Similarly, the article published in NEJM reported 24 of 34 STROBE items (70.6%), two of 13 RECORD items (15.4%), and 7.5 of 15 RECORD-PE items (50.0%)(χ2 = 11.668, P = 0.003). These results indicate that while STROBE items were generally adequately reported, RECORD and RECORD-PE items were not. Important aspects of the methods unique to research using routinely collected health data were not reported, including variables used to identify exposure, outcome and confounders, and validation of the coding or algorithms. RECORD also requires a detailed description of the database population (ie, how data were collected in the population from which the study database is created), and the description and accuracy of linkage of databases. The lack of description of the underlying Surgisphere database should have raised concerns amongst peer reviewers and editors.
Non-transparent and incomplete reporting may have prevented peer reviewers from recognizing significant weaknesses in the research methods, especially if peer reviewers were unfamiliar with research using large health datasets. We found significant differences in the proportion of adequately reported items on the STROBE, RECORD and RECORD-PE checklists, with STROBE items being better reported than the items on the checklists designed for studies using routinely collected health data.
The reason given by three of the authors for the retraction of The Lancet manuscript was the inability of independent auditors to obtain the original data to verify the analyses.1,2 While we understand that legal circumstances may be in place that prevent the data holder from sharing the original data, this runs counter to the movement toward open science and data sharing being adopted by many journals, especially those that endorsed the Wellcome COVID-19 statement on Gold Open Access and reusability,13 indicating underlying support for data availability. The availability of data and programming code allows for the replication of research and verification of results, paramount to the process of scientific discovery.14 The RECORD reporting guidelines include an item (RECORD 22.1) that requests authors “provide information on how to access any supplemental information such as the study protocol, raw data, or programming code”, and the availability of these data was not adequately reported in the two retracted manuscripts. As detailed in the RECORD explanatory document,4 we recognize that some circumstances may restrict the free sharing of raw data, such as privacy laws or data sharing agreements. The RECORD statement is a reporting guideline and does not mandate the open sharing of data. However, RECORD mandates reporting of whether data are available for sharing, and how to access the data, programming code, and study protocol. Had this information been reported in the two manuscripts, the editors and peer reviewers could have made informed decisions about whether to accept the manuscripts for publication.
Reporting guidelines improve the quality of research publications. There is both direct evidence and indirect evidence to support this statement. A randomised controlled trial (RCT) demonstrated a small benefit of including CONSORT and STROBE in the journal peer review process.15 Similarly, a systematic review and meta-analysis of reports assessing the completeness of RCTs reported that journal endorsement of the CONSORT guidelines were associated with improved completeness of reporting.16 Compared to manuscripts based on RCTs, observational research may be more at risk for hidden sources of bias. Therefore, it is essential to encourage transparent and complete reporting of research using routinely collected health data. Journals play a key role in raising awareness amongst authors and peer reviewers about the importance of reporting guidelines to aid with transparent reporting. However, journal editors may be concerned with the rigidity posed by reporting guidelines, and the risk that authors may bypass submission to journals requiring them as part of the submission process.17 Nevertheless, researchers have a generally positive attitude toward reporting guidelines.18 The Equator Network’s catalogue of reporting guidelines by main study type (https://www.equator-network.org/) and interactive walk-through of choice of reporting guideline (https://www.goodreports.org/) has made their use easier for both authors and peer-reviewers.
The design, conduct, and submission of research that is not transparent, open, and adequately reported is a danger to the public trust in science and is wasteful. We have demonstrated that the two retracted studies were inadequately reported, as defined by RECORD and RECORD-PE. We have written this article as an appeal to authors, publishers, editors, and peer reviewers to endorse and effectively implement the correct reporting guidelines in their submission and evaluation of manuscripts. Scientists, journals and the public can no longer accept research that is not transparent and open, especially in the time of a public health emergency such as the COVID-19 pandemic.
Eric Benchimol and Vera Ehrenstein serve on the editorial board of Clinical Epidemiology.
All authors made substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; took part in drafting the article or revising it critically for important intellectual content; agreed to submit to the current journal; gave final approval of the version to be published; and agree to be accountable for all aspects of the work.
SML is supported by a Wellcome Senior Clinical fellowship in Science (205039/Z/16/Z).
EIB and SML are co-Chairs of the RECORD Steering Committee. DM serves on the RECORD Steering Committee, serves on the EQUATOR Network Steering Group, and is Director of the Canadian EQUATOR Centre. SML reports grants from Wellcome Trust, during the conduct of the study. The authors report no other potential conflicts of interest for this work.
1. Mehra MR, Ruschitzka F, Patel AN. Retraction–Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis. Lancet. 2020.
2. Mehra MR, Desai SS, Kuy S, Henry TD, Patel AN. Retraction: cardiovascular disease, drug therapy, and mortality in Covid-19. N Engl J Med. 2020;382(26):2582. doi:10.1056/NEJMoa2007621.
3. Ioannidis JP, Greenland S, Hlatky MA, et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383(9912):166–175. doi:10.1016/S0140-6736(13)62227-8
4. Benchimol EI, Smeeth L, Guttmann A, et al. The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) Statement. PLoS Med. 2015;12(10):e1001885. doi:10.1371/journal.pmed.1001885
5. Langan SM, Schmidt SA, Wing K, et al. The reporting of studies conducted using observational routinely collected health data statement for pharmacoepidemiology (RECORD-PE). BMJ. 2018;363:k3532. doi:10.1136/bmj.k3532
6. Vandenbroucke JP, von Elm E, Altman DG, et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. PLoS Med. 2007;4(10):e297. doi:10.1371/journal.pmed.0040297
7. von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Med. 2007;4(10):e296. doi:10.1371/journal.pmed.0040296
8. Hirst A, Altman DG. Are peer reviewers encouraged to use reporting guidelines? A survey of 116 health research journals. PLoS One. 2012;7(4):e35621. doi:10.1371/journal.pone.0035621
9. Soltani P, Patini R. Retracted COVID-19 articles: a side-effect of the hot race to publication. Scientometrics. 2020:1–4.
10. New Manuscripts. The New England Journal of Medicine; 2020. Available from: https://www.nejm.org/author-center/new-manuscripts.
11. The Lancet - Instructions for Authors. The Lancet; 2020 Updated May. Available from: https://marlin-prod.literatumonline.com/pb-assets/Lancet/authors/tl-info-for-authors.pdf.
12. Hemkens LG, Benchimol EI, Langan SM, et al. The reporting of studies using routinely collected health data was often insufficient. J Clin Epidemiol. 2016;79:104–111. doi:10.1016/j.jclinepi.2016.06.005
13. Publishers make coronavirus (COVID-19) content freely available and reusable. Wellcome; 2020 Updated March 16. Available from: https://wellcome.ac.uk/press-release/publishers-make-coronavirus-covid-19-content-freely-available-and-reusable.
14. Wise J. Data transparency: “Nothing has changed since Tamiflu”. BMJ. 2020;369:m2279. doi:10.1136/bmj.m2279
15. Cobo E, Cortes J, Ribera JM, et al. Effect of using reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: masked randomised trial. BMJ. 2011;343:d6783. doi:10.1136/bmj.d6783
16. Turner L, Shamseer L, Altman DG, et al. Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev. 2012;11:Mr000030.
17. Grindlay DJ, Dean RS, Christopher MM, Brennan ML. A survey of the awareness, knowledge, policies and views of veterinary journal Editors-in-Chief on reporting guidelines for publication of research. BMC Vet Res. 2014;10:10. doi:10.1186/1746-6148-10-10
18. Sharp MK, Bertizzolo L, Rius R, Wager E, Gómez G, Hren D. Using the STROBE statement: survey findings emphasized the role of journals in enforcing reporting guidelines. J Clin Epidemiol. 2019;116:26–35. doi:10.1016/j.jclinepi.2019.07.019
This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.Download Article [PDF]