Back to Journals » Clinical Epidemiology » Volume 6

Quality assessment of observational studies in a drug-safety systematic review, comparison of two tools: the Newcastle–Ottawa Scale and the RTI item bank

Authors Margulis AV, Pladevall M, Riera-Guardia N, Varas-Lorenzo C, Hazell L, Berkman N, Viswanathan M, Perez-Gutthann S

Received 22 April 2014

Accepted for publication 17 June 2014

Published 10 October 2014 Volume 2014:6 Pages 359—368


Checked for plagiarism Yes

Review by Single anonymous peer review

Peer reviewer comments 3

Andrea V Margulis,1 Manel Pladevall,1 Nuria Riera-Guardia,1 Cristina Varas-Lorenzo,1 Lorna Hazell,2,3 Nancy D Berkman,4 Meera Viswanathan,4 Susana Perez-Gutthann,1

1RTI Health Solutions, Barcelona, Spain; 2Drug Safety Research Unit, Southampton, UK; 3Associate Department of the School of Pharmacy and Biomedical Sciences, University of Portsmouth, Portsmouth, UK; 4RTI International, Research Triangle Park, NC, USA

Background: The study objective was to compare the Newcastle–Ottawa Scale (NOS) and the RTI item bank (RTI-IB) and estimate interrater agreement using the RTI-IB within a systematic review on the cardiovascular safety of glucose-lowering drugs.
Methods: We tailored both tools and added four questions to the RTI-IB. Two reviewers assessed the quality of the 44 included studies with both tools, (independently for the RTI-IB) and agreed on which responses conveyed low, unclear, or high risk of bias. For each question in the RTI-IB (n=31), the observed interrater agreement was calculated as the percentage of studies given the same bias assessment by both reviewers; chance-adjusted interrater agreement was estimated with the first-order agreement coefficient (AC1) statistic.
Results: The NOS required less tailoring and was easier to use than the RTI-IB, but the RTI-IB produced a more thorough assessment. The RTI-IB includes most of the domains measured in the NOS. Median observed interrater agreement for the RTI-IB was 75% (25th percentile [p25] =61%; p75 =89%); median AC1 statistic was 0.64 (p25 =0.51; p75 =0.86).
Conclusion: The RTI-IB facilitates a more complete quality assessment than the NOS but is more burdensome. The observed agreement and AC1 statistic in this study were higher than those reported by the RTI-IB's developers.

Keywords: systematic review, meta-analysis, quality assessment, AC1

Creative Commons License This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.

Download Article [PDF]  View Full Text [HTML][Machine readable]