The Risk of Selection Bias in a Clinical Multi-Center Cohort Study. Results from the Norwegian Cognitive Impairment After Stroke (Nor-COAST) Study

Background

Vascular disease, usually due to atherosclerosis, is the most prevalent chronic disease in the developed world.¹ Estimates from the Global Burden of Diseases, Injuries, and Risk Factors Study (GBD 2010) has ranked stroke as the second most common cause of death, and the third most common cause of disability-adjusted life-years (DALYs) worldwide.^2,3 Even though the age-standardized incidence of stroke is decreasing in high-income countries, it is still increasing in low- and middle-income countries.^4,5

Stroke is a risk factor for cognitive impairments, and the incidence of dementia is nearly 50 times higher in the year after a major stroke compared to that in the general population.⁶ However, the reported incidences of post-stroke neurocognitive disorder seem to vary a lot.^7–11 This is probably due to lack of consensus on how to define cases^12–14 and also due to great variation in inclusion criteria of the study samples.^7,13,15,16 Even though a rigorous screening protocol and broad inclusion criteria have been implemented, it has been shown that selection bias occurs, as older patients (especially > 80 years old), patients with impaired function prior to the stroke (modified Rankin Scale score > 3), patients suffering from severe stroke, and patients with comorbidities are more likely to be excluded.^15,17,18 These are also factors associated with an increased risk of developing neurocognitive disorder, resulting in an underestimation of the measured rate of post-stroke neurocognitive disorder.^6,8,9

With respect to the external validity, it is of great importance to identify how the baseline characteristics of patients included differs from those not included in a clinical study. By using data from available patient registries, it is possible to answer this question. In addition, increasing the knowledge about the risk factor for being excluded, may increase the awareness of complying with the inclusion criteria in future clinical studies.

The Norwegian Cognitive Impairment After Stroke (Nor-COAST) study was designed to quantify and measure levels of cognitive impairments in a general Norwegian stroke population and to identify biological and clinical markers associated with prognosis for cognitive disorders following incident stroke.¹⁹ Due to potential regional differences in the stroke population, a multicenter study design was applied to improve the external validity. When interpreting the results from a clinical study, it is of great importance to know how the characteristics of those included differ from those not included and whether there are regional differences in baseline characteristics. It will also be of interest to know the risk factors for not being included.

The overall aim of the present study was to investigate whether selection bias has occurred in Nor-COAST. First, we wanted to investigate to what extent important baseline characteristics from patients participating in Nor-COAST differed from those not participating. Secondly, we wanted to describe the distribution of participants versus non-participants within different age-groups. Thirdly, we wanted to explore whether there were regional differences in recruitment of participants, and finally we wanted to assess whether age, stroke severity, sex and stroke subtype were associated with the odds of being included.

Based on the results from previous research, we hypothesized that the participants in Nor-COAST were younger with better pre-stroke function, less comorbidities, and suffered from milder strokes compared to those not participating.

Materials and Methods

Study Design and Setting

This study compared baseline data from the Nor-COAST study with corresponding data from the Norwegian Stroke Registry. The Nor-COAST protocol consisted of an extensive assessment of cognitive, mental, and physical function at baseline, 3-, 18- and 36-month follow-up. In addition, important patient characteristics, like health history and stroke characteristics, were registered at baseline.¹⁹ Baseline variables that were equal in both datasets were used to analyze differences between participants and non-participants. Patients included in Nor-COAST represented the participants while patients admitted to the participating hospitals in the same period registered in the Norwegian Stroke Registry²⁰ and not included in Nor-COAST represented the non-participants.

Nor-COAST recruited patients from five Norwegian hospitals. The participating hospitals were situated in Central Norway (n=2), Western Norway (n=1) and South-East Norway (n=2). Three hospitals were university hospitals treating more than 300 stroke patients per year while two were middle sized hospitals, treating 200–300 stroke patients per year. One hospital was represented by two different wards as patients at the age above and under 60 years were admitted differently. Patients were included during the initial hospital stay with follow-ups at 3, 18, and 36 months.¹⁹

All patients admitted to a Norwegian hospital with the diagnosis of stroke receive evidence-based stroke treatment according to the recommendations during their initial hospital stay.²¹

This study was conducted in accordance with the Declaration of Helsinki. The study was approved by Regional Committees for Medical and Health Research Ethics (REC) North (2015/171/REC north) and the Norwegian Institute of Public Health.

Study Subjects

Patients Included in Nor-COAST (Participants)

Five trained research assistants (four nurses and one physiotherapist) screened patients for inclusion at all hospitals. Due to differences in availability of the research assistants, the screening was done regularly every weekday at three hospitals and every second day at two hospitals.

Patients admitted to one of the stroke units at the participating hospitals were included according to the following inclusion criteria; diagnosis of acute stroke according to the established WHO criteria²² or with findings on magnetic resonance imaging (MRI) compatible with acute infarction or intracerebral hemorrhage, admitted within one week after onset of symptoms, living in the catchment area of the participating hospitals, Scandinavian speaking and aged > 18 years. Patients with expected lifetime less than 3 months were excluded from the study. Eligible patients were included if they were able and willing to sign informed consent. Patients who were not able to give informed consent were also included if their next of kin did not decline participation. This is in keeping with Norwegian consent procedures for patients not able to consent for themselves.

Patients Not Included in Nor-COAST (Non-Participants)

It is mandatory for all patients admitted to hospital with the diagnosis of acute stroke (infarction or hemorrhage) to be registered in the Norwegian Stroke Registry. From 2015 to 2017, the proportion of patients registered in the Norwegian Stroke Registry for the participating hospitals, ranged from 73% to 94%. The inclusion criteria for the non-participants recruited from the Norwegian Stroke Registry were patients admitted to one of the 5 participating hospitals, within the same period of time as for Nor-COAST but not included in the Nor-COAST study.

Baseline Variables

The following variables were available in the dataset from the Norwegian Stroke Registry and the dataset from Nor-COAST and were used to compare baseline characteristics between participants and non-participants; age, sex, housing condition, smoking habits, comorbidities and global function measured by modified Rankin Scale prior to stroke,²³ which were obtained by interview or review of hospital record after admission to hospital. Stroke severity was determined by clinical assessment using the National Institutes of Health Stroke Scale immediately at admission.²⁴ Housing condition prior to the stroke was listed with the following categories; in their own residence (with or without community care), institution/care home (residential care home or nursing home). Side location of symptoms was categorized into right, left, bilateral or no significant side affected. The following comorbidities were registered; previous cerebrovascular diseases categorized into cerebral stroke or transitory ischemic attack, heart attack, atrial fibrillation, and diabetes. Smoking habits were categorized into smokers versus non-smokers. Stroke diagnosis was categorized into cerebral infarction or hemorrhage.

Study Size

No power calculation was made for this study. However, based on power calculations reported in the protocol paper for Nor-COAST, up to 1000 participants or a maximum of a two-year inclusion period was planned.¹⁹ After almost two years of inclusion, a number of 815 participants were reached. Altogether 1690 patients were admitted to the same hospitals during the same period but not included in Nor-COAST, representing the size of non-participants.

Statistical Analysis

All statistical calculations were performed using IBM SPSS Statistics version 24. Baseline characteristics were compared between the participants and the non-participants. For all categorical variables, number and percent of patients have been reported. Pearson Chi-Squared test was used to analyze the differences between the two independent groups for categorical variables. For continuous variables the independent-samples T-test was used to analyze differences between the two groups, as the variables were approximately normally distributed and did not violate the assumptions of the test. P-values < 0.05 were considered statistically significant.

A histogram was used to display the distribution of mild strokes (National Institutes of Health Stroke Scale 0–4) versus moderate/severe strokes (National Institutes of Health Stroke Scale >4) within different age groups for participants and non-participants.

We used univariable and multivariable binary logistic regression analysis to calculate the odds ratio (OR) of being included in Nor-COAST from baseline characteristics. The characteristics of interest were age, sex, modified Rankin Scale prior to stroke, National Institutes of Health Stroke Scale at admission and stroke subtype (hemorrhage versus infarction). In the univariable analysis all variables were unadjusted, while in the multivariable analysis each variable was adjusted for all other variables in the model.

Results

From May 5, 2015, until March 31, 2017, 2505 patients were admitted to the participating hospitals. Of these, 815 patients were finally included as participants in Nor-COAST, while 1690 patients were categorized as non-participants in our analyses. Out of the non-participants, 559 patients did not fulfil the inclusion criteria and were classified as not eligible, while 1131 patients were not included because they declined participation (n=143), had an early discharge (n=92), failed to be screened (n=753) or for other reasons (n=143). Examples of other reasons were severe aphasia, severe cancer, psychiatric diseases or impaired hearing. The main reasons for failing to be screened were lack of daily screening routines at two hospitals and no screening during holidays at all hospitals. The flow of participants is displayed in Figure 1.

Figure 1 Flow-chart showing reasons for not being eligible and not being included in Nor-COAST.

Table 1 shows that 38.3% of the stroke patients admitted to the participating hospitals within Central Norway Health Authority, 26.1% of the stroke patients admitted to the participating hospital in Western Norway Health Authority, and 29.0% of the stroke patients admitted to the participating hospitals in South-East Norway Health Authority were included.

Table 1 Proportion of Patients from the Participating Health Authorities

The baseline characteristics presented in Table 2 show no differences in age and sex (mean (SD) age 73.5 (11.7) versus 74.2 (14.5) years) or sex (44.8% versus 46.9% women), but a greater proportion of the participants in Nor-COAST were diagnosed with mild stroke (541 (69.1%) versus 731 (54.6%) patients, p< 0.001). More participants than non-participants were independent prior to the stroke (704 (86.8%) versus 1286 (77.5%) patients, p < 0.001) and a greater proportion of participants were living in their own residence prior to the stroke (797 (97.8%) versus 1494 (88.8%) patients, p-value < 0.001), while a smaller proportion of the participants had suffered from a previous stroke (155 (19.2) versus 397 (23.6), p=0.013), and atrial fibrillation (142 (17.6) versus 468 (28.1) patients, p<0.001).

Table 2 Characteristics of Participants Versus Non-Participants in Nor-COAST

The three health authorities showed similar results as the whole group except for significantly younger age among the participants and no difference in stroke subtypes in Western Norway Health Authority (Table 3).

Table 3 Characteristics of Participants Versus Non-Participants Within Each Health Authority

Figure 2 shows that the proportion of participants and non-participants with mild stroke decreased as age increased, while the proportion of participants and non-participants with moderate/severe stroke increased as age increased.

Figure 2 The proportion of patients with mild (National Institutes of Health Stroke Scale 0–4) versus moderate/severe (National Institutes of Health Stroke Scale > 4) stroke among participants and non-participants within different age-groups.

The univariable logistic regression analysis showed that low functional level prior to the stroke, stroke severity, and cerebral hemorrhage were independently associated with a lower OR of being included in Nor-COAST. Cerebral hemorrhage did not remain significant in the adjusted analysis (Table 4).

Table 4 Binary Logistic Regression Analysis Showing the Odds Ratio (OR) and 95% Confidence Interval (CI) for Being Included in Nor-COAST According to Baseline Characteristics

Discussion

In contrast to our hypothesis, this study showed no overall difference in age between participants included in Nor-COAST versus those not included. However, the participants had significantly better pre-stroke function and suffered from milder strokes. This finding was also confirmed by the multivariable regression analysis showing increased odds of being included for patients with better pre-stroke function and milder strokes. The participants and non-participants were comparable with respect to sex, smoking habits and the prevalence of diabetes mellitus, previous transitory ischemic attack and previous myocardial infarction.

The overall aim of this study was to assess the risk of selection bias in Nor-COAST from available baseline variables. One limitation was that the Norwegian Stroke Registry does not contain data on cognitive ability or education level prior to stroke, which are important variables in determining the patient’s risk of developing neurocognitive disorder.⁷ We know that people with higher education and better cognitive score are more likely to volunteer for participation in research.^25,26 This might be a potential source for selection bias also in Nor-COAST. Another limitation was slightly different questionnaires being used by the Norwegian Stroke Registry and Nor-COAST with different categories for some of the variables which might have led to some degree of misclassification. However, we regard this as a less likely event.

Our finding of no difference in age between the two groups contrasts with previous research showing that older patients are less likely to be included in clinical studies.^15,17,18 However, the results showed some variability between the hospitals with significantly younger participants compared to non-participants in Western Norway Health Authority. This finding might be explained by different admittance policies and other ongoing competing studies within the participating hospitals.

Altogether, 753 patients failed to be screened during the study period which was the most frequent cause (68%) for not being included. The reason was mainly due to breaks in inclusion during weekends and holidays. Another possible explanation might be slight differences in the screening and recruitment procedures as some hospitals had dedicated nurses who screened patients on a daily basis, while other hospitals only screened the patient lists 2–3 times per week, which increased the risk of failing to include patients with minor symptoms and a short hospital stay. This might also explain the big differences (38% versus 26%) in participating rate among different regions. Even though failing to be screened should be regarded as missing at random, this finding underscores the importance of having enough resources to include patients also during the weekends in clinical studies.

Despite no overall difference in age, the participants in Nor-COAST tended to have a better pre-stroke health condition as measured by modified Rankin Scale, a greater proportion suffered from mild stroke and a greater proportion living in their own residence. All these differences, together with the lower prevalence of atrial fibrillation and previous stroke among the participants, are probably closely related and can partly be explained by the fact that patients with severe illness and short life expectancy (< 3 months), were not eligible for inclusion. Hence, the future results from Nor-COAST will mainly be valid for patients suffering from minor strokes who have been shown to be at less risk of developing cognitive impairments.⁶ Even though the ethical committee had approved to include patients not able to consent for themselves, including patients in a vulnerable situation suffering from very severe stroke may represent an ethical dilemma for the staff. On the other hand, excluding this group of patients from research may also be regarded as unethical. As shown in Figure 2, the oldest age group (≥ 85 years) contained the largest proportion of participants and non-participants with severe stroke. This is also the group who are at the highest risk of neurocognitive disorder and failing to include these patients will most likely contribute to an increased risk of underestimating stroke related cognitive impairments.^8,27,28 This will again make it difficult to improve the diagnostic tools and future treatment for this vulnerable group.

When interpreting the results from the present study, it is important to keep in mind that the control group consisted of patients from the Norwegian Stroke Registry not included in Nor-COAST, meaning that none of these groups should be regarded as the general stroke population. For this purpose, the annual report from the Norwegian Stroke Registry should be used as a standard of reference.²⁰ The proportion of mild strokes among participants and non-participants was 69% versus 55% respectively. According to the annual report, 64.7% of the Norwegian stroke population suffer from a mild stroke, indicating that the participants in Nor-COAST were closer to the general stroke population than the non-participants with respect to stroke severity. This finding was also confirmed by pre-stroke function, where 86.8% of the participants from Nor-COAST were self-reliant (modified Rankin Scale score 0–2) prior to the stroke, compared to 77.5% among the non-participants. Corresponding numbers from the annual report showed that 85.3% of the general stroke population were self-reliant prior to stroke.²⁰ Despite the selection of slightly healthier stroke patients into Nor-COAST, the future results from this dataset should be regarded as representative of the majority of the stroke population which suffers from mild strokes.

The comprehensive and thorough study protocol including a comprehensive questionnaire, several tests measuring both mental, cognitive and physical function in addition to brain imaging and blood samples was a strength of this study.¹⁹ Nevertheless, among the non-participants from the Norwegian Stroke Registry, 21% of National Institutes of Health Stroke Scale admission scores were missing (compared to 4% among the participants), showing that a complete dataset is challenging also in a high-quality registry. There are several potential reasons for the high number of missing National Institutes of Health Stroke Scale values at admission, eg, that patients with vague symptoms may not be diagnosed with stroke right away, and for that reason National Institutes of Health Stroke Scale score is not completed.

The major strength of Nor-COAST was the inclusion of non-participants from the Norwegian Stroke Registry, which is a national medical quality registry for stroke care. Reporting of patients to the Norwegian Stroke Registry is a legal obligation and does not require the patients’ consent. The register fulfils the criteria to the highest level of quality, with a coverage of 86% in 2017.²⁰ Most of the variables in the Norwegian Stroke Registry have substantial to excellent reliability, and serve as a valuable source of data for epidemiological, clinical and healthcare studies.^29,30 Another strength was the high number of participants recruited in a clinical setting at five different Norwegian hospitals, making this a pragmatic clinical study. The multi-center design also strengthens the validity of our results.

Conclusion

Even though the participants included in Nor-COAST seemed to be slightly healthier prior to the stroke and tended to have milder strokes compared to the non-participants, the participants should be regarded as representative of the majority of the stroke population which suffers from mild stroke and who are at less risk of developing neurocognitive disorder. Despite a rigorous screening protocol, this study shows that it is challenging to include the most vulnerable patients and report results that can be generalized to this part of the stroke population. Furthermore, baseline information for non-participants should be available also in future clinical studies.