Three Statistical Approaches for Assessment of Intervention Effects: A Primer for Practitioners
Received 12 August 2020
Accepted for publication 11 January 2021
Published 22 February 2021 Volume 2021:14 Pages 757—770
Checked for plagiarism Yes
Review by Single anonymous peer review
Peer reviewer comments 2
Editor who approved publication: Dr Kent Rondeau
Lihua Li,1– 3 Meaghan S Cuerden,4 Bian Liu,1,3,5 Salimah Shariff,6 Arsh K Jain,4,6 Madhu Mazumdar1– 3
1Institute for Healthcare Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY, USA; 2Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai, New York, NY, USA; 3Tisch Cancer Institute, Icahn School of Medicine at Mount Sinai, New York, NY, USA; 4London Health Sciences Centre, London, Ontario, Canada; 5Institute for Translational Epidemiology, Icahn School of Medicine at Mount Sinai, New York, NY, USA; 6Institute for Clinical Evaluative Sciences, Toronto, Ontario, Canada
Correspondence: Madhu Mazumdar
Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai, Mount Sinai Hospital, 1425 Madison Avenue, New York, NY, USA
Tel +1 212-659-1470
Introduction: Statistical methods to assess the impact of an intervention are increasingly used in clinical research settings. However, a comprehensive review of the methods geared toward practitioners is not yet available.
Methods and Materials: We provide a comprehensive review of three methods to assess the impact of an intervention: difference-in-differences (DID), segmented regression of interrupted time series (ITS), and interventional autoregressive integrated moving average (ARIMA). We also compare the methods, and provide illustration of their use through three important healthcare-related applications.
Results: In the first example, the DID estimate of the difference in health insurance coverage rates between expanded states and unexpanded states in the post-Medicaid expansion period compared to the pre-expansion period was 5.93 (95% CI, 3.99 to 7.89) percentage points. In the second example, a comparative segmented regression of ITS analysis showed that the mean imaging order appropriateness score in the emergency department at a tertiary care hospital exceeded that of the inpatient setting with a level change difference of 0.63 (95% CI, 0.53 to 0.73) and a trend change difference of 0.02 (95% CI, 0.01 to 0.03) after the introduction of a clinical decision support tool. In the third example, the results from an interventional ARIMA analysis show that numbers of creatinine clearance tests decreased significantly within months of the start of eGFR reporting, with a magnitude of drop equal to − 0.93 (95% CI, − 1.22 to − 0.64) tests per 100,000 adults and a rate of drop equal to 0.97 (95% CI, 0.95 to 0.99) tests per 100,000 per adults per month.
Discussion: When choosing the appropriate method to model the intervention effect, it is necessary to consider the structure of the data, the study design, availability of an appropriate comparison group, sample size requirements, whether other interventions occur during the study window, and patterns in the data.
Keywords: difference-in-difference, interrupted time series, segmented regression, autoregressive integrated moving average
This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.Download Article [PDF] View Full Text [HTML][Machine readable]