Statistical Analysis of Clinical COVID-19 Data: A Concise Overview of Lessons Learned, Common Errors and How to Avoid Them

Introduction

During the COVID-19 pandemic, the analysis of data from COVID-19 patients is essential to understand the clinical progress, potential treatment effects and expected outcomes. Numerous methodological articles have shown that adequate statistical models are necessary to acknowledge the time-dependent complexity in particular of hospital data. It has also been shown that in several applications, standard models such as Kaplan–Meier survival plots, Cox or logistic regression models can easily lead to biased estimates and wrong conclusions. In this article, we summarize some of the most important pitfalls, provide intuitive clinical examples (mostly related to other types of infections) and refer to methodological articles which explain the mathematical framework. We also describe common errors which can occur in the presence of competing events and time-dependent covariates, or when conditioning on future events.

Methods and Results

Inappropriate Use of Kaplan–Meier Survival Plots

One important outcome of admitted COVID-19 patients are the combined endpoint admission to intensive care, invasive ventilation or hospital mortality. In this case, standard Kaplan–Meier plots are often used for time-dependent events such as admission to intensive care unit, invasive ventilation or hospital mortality, if only hospital data is available.^1–6 However, this implies that patients discharged without the event are censored. This is not a meaningful model assumption, since discharged patients are usually in a better health condition than hospitalized patients. Cumulative incidence functions should be used instead, accounting for discharge as a competing event.

It is also common to use standard Kaplan–Meier plots to display effects of time-dependent covariates such as infectious disease consultation or medical treatment.^7,8 This is misleading and does more harm than good. One way to address this issue is to use landmark models where covariate values are updated at each landmark.⁹

Inappropriate Use of Standard Cox Regression Models

Competing events are often ignored in time-to-event analysis.^8,10-12 In case of events other than death (eg, time-to-extubation) being analysed, death should be considered a competing event. The same is true when death is the event of interest in a time-to-event analysis of in-hospital data: here, discharge alive should be considered a competing event and patients should not be censored at the time of discharge. There are two reasons why ignoring competing events should be avoided. First, it is incomplete¹³ and second, these events are often very informative and the key to understanding your research question.

For example, Wolkewitz et al² do not detect an effect of Oseltamivir on the death hazard of patients hospitalized with confirmed pandemic influenza A/H1N1 2009 infection. However, the discharge hazard is significantly decreased, resulting in a prolonged length of stay for the treated patients. The prolonged length of stay results in an increased risk of death in the hospital. Cause-specific Cox regression for both in-hospital mortality and discharge alive avoids competing risk bias and provides a detailed understanding of potential treatment effects.

Handling time-dependent predictors (eg, time-dependent anti-viral treatment, time to ventilation or time to oxygen saturation below 90%) as time-fixed leads to immortal-time bias.^2,4,14-16 A mathematical formula shows that the hazard ratio for the predictor’s effect on death is always underestimated, meaning that predictors that increase the risk of death may be missed or even appear to be beneficial. In contrast, this bias always overestimates the effect for additional length of hospital stay.

For example, in the primary analysis of Rosenberg et al⁸ on patients hospitalized with COVID-19, the hazard ratio of Hydroxychloroquine + azithromycin is 1.35 (0.67–2.40). The hazard ratio corrected for the immortal time bias is presented in the appendix of the paper by Rosenberg et al (eTable 6).⁸ This sensitivity analysis that properly accounts for the time-dynamics results in a clearly higher hazard ratio: 1.83 (1.02–3.28). The same is applied for Hydroxychloroquine and Azithromycin alone.

This is one of the severe types of bias in epidemiology and very common in the hospital setting. More advanced Cox models can account for time-dependent covariates.¹⁷

The increased mortality in the treatment groups of Rosenberg et al may be explained by time-dependent confounding, which is generally present if treatment is administered over time. Time-dependent propensity scores in the Cox model (known as marginal structural model) adequately take the time-dynamic confounder-treatment feedback into account. For example, Arabi et al discussed the influence of immortal time bias and time-dependent confounding in their excellent retrospective cohort study of patients infected with Middle East Respiratory Syndrome Coronavirus (MERS-CoV).¹⁸ In their study, they find a differential effect of immortal-time bias and time-dependent confounding on the magnitude of the hazard ratio: the hazard ratio of Ribavirin/Recombinant Interferon for 90-day mortality from the marginal structural Cox model was 67% lower than from a time-dependent Cox model adjusted only for baseline confounding (1.52 (1.13–1.06) versus 1.03 (0.73–1.44)).

Inappropriate Use of Standard Logistic Regression

Standard logistic regression is a popular tool for examining associations between risk factors and the event of interest if patient data are available until discharge. However, during a pandemic situation data contain mainly active COVID-19 cases (ie, patients are still in hospital). For instance, Zhou et al¹⁹ discarded infected patients who were still alive on 31 January 2020. The reported numbers are 813 hospitalized and 613 excluded because they were still hospitalized or not confirmed as infected; the numbers for each of the two exclusion reasons are not given. Hence a large proportion of the cohort of interest was excluded, which has the potential to create selection bias which could impact the key risk estimates. In this case, time-to-event models should be used over logistic regression for examining associations between patient-level predictors and the event of interest (eg, death). Time-to-event models allow censoring of patients who are still hospitalised at the last follow-up date. Excluding these patients when using logistic regression introduces selection bias, which can be strong enough to nullify the effect of important predictors. The bias depends on the difference between patients who have long and short lengths of stay. For example, if longer-staying patients are generally older, then the effect of age may be underestimated and the generalisability of the results questionable.

Inappropriate Use of Advanced Regression Models

Sub-distribution hazard models, such as the Fine & Gray model,²⁰ have been developed to evaluate effects of time-fixed covariates on competing events. However, be mindful of time-dependent covariates and competing endpoints.²¹ The risk of finding artefacts is high and results need to be carefully considered before reaching any conclusions about whether covariates truly influence the risk of events.

Inappropriate Conditioning on Future Events

Conditioning on future events, for example, excluding patients who experience events that are not of primary interest,^22–24 violates one of the main principles in time-to-event analysis²⁵ by introducing selection bias. A recent COVID-19 example is the retrospective study by Mehra et al,²⁶ where

patients for whom one of the treatments of interest was initiated more than 48 h after diagnosis or while they were on mechanical ventilation, as well as patients who received remdesivir, were excluded.

In general, it is best to include the whole cohort over time, as excluding patients always has the potential to introduce serious bias. Picture being in a hospital and collecting patient data prospectively. All the patients in your cohort are at risk of the key events, and you have no way of knowing what patients to exclude based on future events.

Discussion

These issues have been known for many years from studies of which factors influence hospital morbidity and mortality. The potential biases in these studies have been explained using mathematical formulae, simulation studies and real-world analyses. Furthermore, most methods to reduce bias are easy to apply and available in standard statistical software. Unfortunately, statistical methods that are better suited to these data have yet to be widely adopted, and too many researchers continue to use simple survival analysis techniques that are not suited to complex hospital data with variables that change over time and multiple outcomes.²⁷ The rush to analyse in-hospital COVID-19 data to inform clinical practice has reduced the time for detailed peer review.²⁸ We are concerned that many current results that examine the in-hospital treatment of COVID-19 patients are untrustworthy. Greater involvement of statisticians with experience in appropriate methods will improve the quality of analysis undertaken and their interpretation before being shared with the wider research community.