Validity of Simple Algorithms to Identify Recurrence of Intracerebral Hemorrhage in Two Danish Nationwide Registries

Introduction

Intracerebral hemorrhage (ICH) represents 10–15% of incident strokes in high-income countries, the remaining being primarily due to ischemic stroke.¹ In most cases, ICH is spontaneous, ie, not related to trauma, underlying vascular malformation or tumor. Spontaneous ICH (s-ICH) is largely due to cerebral small vessel disease, predominantly deep perforator arteriopathy (also termed hypertensive arteriopathy) and cerebral amyloid angiopathy.² ICH is the deadliest form of stroke.³ Almost 40% of patients admitted with ICH die within 30 days of admission.^4,5 Additionally, patients surviving ICH face an increased risk of suffering recurrent ICH (re-ICH).^6–8 The risk of re-ICH is highest in the first year after the index ICH,^9,10 but remains increased for years thereafter, particularly in lobar ICH.⁷ To ensure better long-term prognosis and outcome for survivors of ICH, research on the associated risk factors and causes of re-ICH is greatly warranted. However, due to the relative rarity of the disease, large-scale prospective studies of ICH can be difficult and costly to conduct. Large administrative registries and clinical databases may provide attractive alternative data sources for the study of rare diseases such as ICH. In Denmark, two nationwide registries can be used for research on ICH: The Danish Stroke Registry (DSR)¹¹ and the Danish National Patient Registry (DNPR).¹²

We recently reported on the high validity of first-ever ICH-diagnoses in DSR and DNPR.^13,14 However, accurate identification of re-ICH in these registries may pose distinct methodological challenges, compared with first-ever ICH. When tracking a patient with an index ICH event (eg, first-ever ICH) in a registry it can be difficult to determine which, if any, of the patient’s subsequent admissions coded as ICH represent re-bleeds (re-ICH), as opposed to admissions for causes related to the index ICH (eg, transfers between wards, re-admissions, or admissions for complications to the index ICH). The extent of such misclassification likely varies by source, reflecting the purpose for data collection. DNPR is a registry originally established to serve administrative purposes, and is therefore more likely to be prone to the above-mentioned sources of misclassification of re-ICH, compared with DSR, a regularly audited clinical database of patients admitted with acute stroke.¹¹ Conversely, it is conceivable that DNPR offers a higher degree of completeness than DSR, as recently demonstrated for first-ever ICH-diagnoses in these registries.^13,14

The difficulties outlined above could lead to misclassification of the number of ICH events and could ultimately result in overestimation of the burden of re-ICH in the individual patient as well as misclassification of the date of symptom onset (onset date) of re-ICH, thereby affecting risk estimates.¹⁵ Yet to our knowledge, research on the use of registries to identify cases of recurrent stroke is very limited. Only one study¹⁶ from the US has previously reported on the accuracy of registry data for identifying re-ICH (Supplementary Material and Supplementary Figure S1). Therefore, we conducted this study with the purpose to develop and validate algorithms to identify cases of re-ICH using data from DSR and DNPR; and, further, to assess the accuracy of the onset date of re-ICH identified by the algorithms.

Patients and Methods

Setting and Data Sources

Data for this study were based on hospital admissions for residents of the Region of Southern Denmark (RSD) (1.2 million inhabitants); a geographically well-defined area in Denmark. Within RSD there are four dedicated stroke units as well as a neurosurgical department to which all patients suspected of stroke are principally admitted or transferred.

Data were obtained from two nationwide registries: DSR and DNPR. DSR is a clinical database that was established in 2003 to monitor the quality of stroke care.¹¹ It is mandatory for all departments involved in stroke care to report standardized data on all admissions for acute stroke to DSR.^11,17 DNPR is an administrative registry, established in 1977, to which all hospitals in Denmark report standardized data, including diagnosis codes according to the International Classification of Diseases, version 10 (ICD-10), on all hospital contacts (inpatient, outpatient and emergency department (ED)).¹²

In accordance with Danish law regarding register-based research, the study was approved by the Region of Southern Denmark (j.nr. 18/51966) and informed consent was waved.¹⁸

The Gold Standard

In a previous study, we validated cases of first-ever s-ICH among residents of RSD during the period January 1, 2009 through December 31, 2017.¹⁴ We defined s-ICH as ICH not attributable to prior trauma, hemorrhagic transformation of an ischemic stroke, or an alternative explanation (ie, tumor, vascular malformation, or cerebral sinus thrombosis – but not use of antithrombotic drugs).^13,14,19 A total of 2528 patients with verified first-ever s-ICH were included in the inception cohort. Each patient from the inception cohort was followed-up for events of re-ICH by identifying all admissions registered with an ICH-diagnosis (ICD-10 code I61) within DSR and DNPR (inpatient diagnosis, any coding position) from the date of the index ICH through December 31, 2018, which secured at least one year of follow-up. To ensure completeness, we also identified all admissions within DNPR registered with diagnostic codes for a range of other cerebrovascular diseases (ICD-10 codes in parentheses) including subarachnoid hemorrhage (I60.9), subdural hematoma (I62.0 and S06.5), unspecified intracranial hemorrhage (I62.9), acute ischemic stroke (I63 and I66), unspecified stroke (I64), other specified cerebrovascular disease (I67.8), transient ischemic attack (G45.9), and vascular syndrome (G46). Within DSR, we also identified admissions for ischemic stroke and unspecified stroke. For each of the identified events the diagnosis of s-ICH as well as the onset date of re-ICH were validated by manual review of the full medical record and brain scan reports. Validation was performed by study personnel (two medical students and two physicians) supervised by a neurologist and a radiologist, both with special interest in stroke. The recurrence status of ICH and the onset date of re-ICH ascertained by revision of the medical record and brain scan reports were considered the gold standard.

For patients who suffered more than one verified re-ICH during follow-up, we only included the first of these events in the analysis.

To calculate follow-up from index ICH to verified re-ICH (or to censoring due to emigration, death, or end of study period), we linked our data to information retrieved from the Danish Civil Registration System,²⁰ an administrative registry with all Danish citizens’ unique personal identification number (CPR-number), which is continuously updated with information on vital status and place of residence. The CPR-numbers, also recorded in our data, enabled unambiguous linkage between the sources.

Study data were collected and managed using REDCap electronic data capture tools hosted by Open Patient data Explorative Network (OPEN).^21,22 The data can only be accessed via a secured virtual private network (VPN) and only by approved collaborative partners. The data is not publicly available due to the Danish data protection legislation as the data contains information that could compromise the privacy of the research participants.

The Algorithms

We developed two algorithms with the purpose of identifying cases of re-ICH based exclusively on registry data from DSR and DNPR, respectively.

When using the algorithms, all chronologically consecutive hospital contacts were considered as belonging to a single admission episode if the contacts were not separated by a gap (ie, discharge date of a hospital record corresponded to the admission date of the next hospital record). We excluded episodes that included the admission date of the index ICH, as such admissions are directly related to the index ICH (eg, representing transfers between wards) according to our experience from previous studies^13,14 and the validation efforts involved in this study.

In DSR, an algorithm-based re-ICH event (Algo-re-ICH) established exclusively through registry information (ie, not verified through medical records) was defined as the first inpatient contact registered with an ICH-diagnosis following the index ICH. Using data from DSR, the algorithm identified events of interest occurring in the inception cohort during the follow-up period from January 1, 2009 through December 31, 2018. The admission dates corresponding to events identified by the algorithm were defined as the onset date of Algo-re-ICH (Figure 1). We iteratively repeated the above process using a blanking period (BP) of 7, 30, 60, 90, 120, 150, 180 and 360 days, respectively. A BP is a time period immediately after an inception event (here index ICH), during which outcome events (here possible admissions for re-ICH) are ignored (Figure 2).^23,24 The underlying assumption when using a BP is that misclassification of recurrent events is to a certain degree time-dependent, ie, more distant admissions are more likely to represent true recurrent events compared with admissions immediately after the index event.

Figure 1 Flowchart illustrating the development and validation of algorithms for identifying cases of recurrent intracerebral hemorrhage using data from the Danish National Patient Registry and the Danish Stroke Registry. *Multiple consecutive records considered as single admission episode, if no gap between discharge date of one record and admission date of the following record. **Period immediately following the index ICH, during which outcome events are ignored; varying lengths (7–360 days) tested. Abbreviation: ICH, intracerebral hemorrhage.

Figure 2 An example of use of an algorithm with a blanking period to identify recurrent intracerebral hemorrhage based exclusively on registry data. In the example, a blanking period of 30 days is used. The patient had her first-ever ICH on day 0. According to registry data (eg, Danish Stroke Registry), she was subsequently readmitted (ie, had inpatient contacts registered with ICD-10 code I61) 4 times for ICH in the study period. When applying the algorithm, the first two re-admissions (day 4 and day 7) were ignored as they occurred during the 30-day blanking period. The first admission occurring after the blanking period (day 90) was classified as her first recurrent ICH event. As follow-up stopped after the first episode of recurrent ICH, the last admission on day 130 was ignored. Abbreviation: ICH, intracerebral hemorrhage.

We applied the same approach as outlined above with the algorithm using data from DNPR. In DNPR, however, Algo-re-ICH was defined as the first inpatient contact registered with an ICH-diagnosis in primary coding position following the index ICH.

For each registry, we defined the algorithm with the shortest BP that returned a positive predictive value (PPV) of ≥80% to be optimal.

Statistical Analysis

We assessed the validity of the algorithms by calculating the PPV, negative predictive value (NPV), sensitivity, and specificity with the gold standard as reference. We used Wilson score method to estimate 95% confidence intervals (CI) for all proportions.

We also calculated the proportion of onset dates identified by the algorithms that coincided (same date) or occurred within 1, 7, 30, or 90 days of the gold standard onset date, respectively.²⁵

It could be argued that patients who die or emigrate during a given BP should not be included in the analyses as such patients by definition cannot suffer a re-ICH in the follow-up ensuing after the BP in question. Therefore, in a subanalysis, we calculated PPV, NPV, sensitivity and specificity of re-ICH in each registry for the optimal BP and after excluding patients that had died or emigrated during the BP.

All statistical analyses were done using Stata 16.1 (Stata Corp, TX).

Results

The inception cohort comprised 2528 patients, after exclusion of a small number of patients (n = 18) with insufficient information on admissions during follow-up (mostly untraceable medical records). Re-ICH could be verified in 124 patients of the inception cohort leaving 2404 patients not having a re-ICH during follow-up. The gender distribution and median age were similar between patients when stratified by gold standard re-ICH (Table 1). The mean follow-up after index ICH was 2.4 years (median 1.37 years). However, patients that had a re-ICH were followed up for longer than those with no re-ICH (median follow-up, 4.18 vs 1.21 years). This difference was primarily due to the high short-term all-cause mortality of patients with ICH, as demonstrated by the substantial reduction in the difference in follow-up between patients with re-ICH vs patients with no re-ICH in an analysis restricted to those that survived the first 30 days post-ICH (median follow-up: 4.45 years vs 3.75 years).

Table 1 Characteristics of the Study Population (n = 2528) Stratified on the Recurrence Status of Intracerebral Hemorrhage According to the Gold Standarda

We assessed the performance of the algorithms by comparison with the gold standard and evaluated the algorithms’ concordance with the gold standard when using varying lengths of BPs (Tables 2 and 3). When testing the algorithms without a BP, they yielded very low PPVs of 37.5% (95% CI = 31.9–43.6) and 23.0% (95% CI = 19.3–27.1) for DSR and DNPR, respectively. The length of the BP was directly associated with PPV (ie, longer BP resulted in higher PPV). However, application of a BP of more than 60 days had little impact on the PPV in DSR (Table 2), but more so in DNPR (Table 3). Even so, in DNPR, application of the maximum BP of 360 days only improved the PPV by approximately 10% compared with the 60-day BP, and less than 5% compared with the optimal BP (see below). As expected, the length of the BP was inversely associated with sensitivity in both registries with longer BPs incrementally returning lower sensitivities, eg, with the maximum BP of 360 days resulting in sensitivities below 50% for both registries. In both DSR and DNPR, BP-length had little impact on NPV and specificity, which were both high in both registries.

Table 2 Accuracy of the Algorithm Using Data from the Danish Stroke Registry to Identify Cases of Recurrent Intracerebral Hemorrhage When Applying a Blanking Period of Varying Lengtha

Table 3 Accuracy of the Algorithm Using Data from the Danish National Patient Registry to Identify Cases of Recurrent Intracerebral Hemorrhage When Applying a Blanking Period of Varying Lengtha

Overall, the algorithm using data from DSR achieved a higher PPV with a shorter BP when compared with the algorithm using data from DNPR. To illustrate, applying a BP of only 7 days returned a PPV of 75.6% (95% CI = 67.4–82.2). Similar results with the algorithm for DNPR (ie, PPV of 75.4% (95% CI = 66.9–82.3)) were achieved with a BP of 60 days.

We considered the algorithms to be optimal when using the shortest BP possible and still returning a PPV 80%. In DSR, the optimal algorithm (BP 30 days) had a PPV of 89.5% (95% CI = 82.2–94.0), NPV 98.8% (95% CI = 98.2–99.1), sensitivity 75.8% (95% CI = 67.6–82.5), and specificity 99.5% (95% CI = 99.2–99.7). In DNPR, the optimal algorithm (BP 120 days) returned a PPV of 80.6% (95% CI = 71.7–87.2), NPV 98.1% (95% CI = 97.5–98.6), sensitivity 63.7% (95% CI = 55.0–71.6), and specificity 99.2% (95% CI = 98.8–99.5) (Tables 2–4).

Table 4 Concordance of the Recurrence Status of Intracerebral Hemorrhage Identified by the Optimal Algorithms and the Gold Standard

For the optimal algorithms, we evaluated the concordance of the onset date of re-ICH identified by the algorithm and the gold standard. For DSR 79.8% of the onset dates identified by the algorithm matched the gold standard, 88.3% of the onset dates matched within one day of the gold standard, and 91.5% matched within 7 days of the gold standard. For DNPR, 74.7% of the onset dates identified by the algorithm matched the gold standard, 83.5% of the onset dates matched within one day of the gold standard, and 88.6% of the onset dates matched within 7 days of the gold standard. After 30 days, the proportion of onset dates identified by the algorithms matching the gold standard did not change (Table 5).

Table 5 Accuracy of the Optimal Algorithms for Estimating the Onset Date of Intracerebral Hemorrhage Recurrence

Subanalysis

In a subanalysis of validity estimates for the optimal algorithms (DSR: 30-day BP; DNPR: 120-day BP) we excluded patients who died during the BP (none had emigrated). The subanalysis returned higher sensitivity estimates (DSR 91.3% (95% CI = 84.2–95.3); DNPR 81.4% (95% CI = 72.6–87.9)) and slightly lower specificity estimates (DSR 98.9 (95% CI = 98.0–99.4); DNPR 97.9 (95% CI = 96.8–98.7)) than the main analysis (Tables 2 and 3). Predictably, PPV and NPV were unchanged, as calculation of these measures does not depend on data from the BP.

Discussion

For this study, we developed and validated algorithms to identify cases of re-ICH using data from two Danish nationwide registries. The algorithms accurately identified cases of re-ICH as well as the onset date of re-ICH using data from DSR and DNPR, respectively. However, the PPV of the algorithms increased with the length of the applied BP, at the expense of the sensitivity, which decreased. The algorithm using data from DSR yielded a higher PPV with a considerably shorter BP when compared with the algorithm using data from DNPR (ie, BP of 30 days vs 120 days). This difference was to some degree anticipated due to the different purposes of the two registries; whereas DNPR primarily serves as an administrative registry,¹² DSR serves as a clinical database for monitoring stroke care.¹¹ Therefore, events of re-ICH are expected to be coded with greater accuracy in DSR compared with DNPR.

In previous studies,^13,14 we found a high validity of ICH-diagnoses within DSR and DNPR for identifying cases of first-ever ICH. In this study, we found an equally high validity of ICH-diagnoses within DSR and DNPR for identifying cases of re-ICH when using algorithms to account for the methodological difficulties associated with identifying recurrent events. Our study also documents that use of a BP is essential for accurate classification of re-ICH events, as the algorithms where we did not apply a BP returned very low PPVs (<40%); conversely, even a very short BP of 7 days provided marked improvement of PPVs in both registries, particularly DSR. Striking the right balance between PPV and sensitivity will depend on the aim of the individual project. For example, a high PPV is important in studies of prognosis after re-ICH, while in studies of the incidence of re-ICH, high sensitivity is also vital. We note that compared with the main analysis, the subanalysis, returned markedly higher sensitivity estimates for both DSR (75.8% vs 91.3%) and DNPR (63.7% vs 81.4%). However, the higher sensitivity in the subanalysis is not only conditional on ignoring re-ICH events during the BP (as in all main analyses) but also on excluding all patients that died or emigrated during the BP. Based on these results and in order to avoid immortal person-time, we suggest that cohort studies of the risk of re-ICH based on DSR or DNPR that employ a BP should begin follow-up on the date the BP ends and only include cohort members that are eligible at start of follow-up (ie, exclude patients that died or emigrated during the BP). Another consideration is the time frame of interest for a project considering use of DSR or DNPR, as the risk of re-ICH is reported to be highest in the first year after an index ICH.⁹ For projects focusing on, eg, the first year post-ICH, use of DSR would seem more optimal as even a short blanking period provides valid results. Conversely, the markedly longer blanking period necessary to achieve a high PPV for re-ICH in DNPR may be less of a concern in studies interested in the period after the first year post-ICH. We arbitrarily chose to focus on PPV ≥80% for selection of an optimal length of BP in the two algorithms for DSR and DNPR, respectively, but provide values for other BPs that could guide researchers’ choices, depending on their aims.

To our knowledge, this is the first study on the validity of DSR and DNPR for identifying re-ICH. Similarly, no research on identifying re-ICH has been conducted using data from other Scandinavian registries. In general, very little research on the subject is available (Supplementary Material and Supplementary Figure S1). One US-based study¹⁶ using hospital discharge codes to identify cases of re-ICH found a PPV of 53%. However, the study was not specifically designed for identifying re-ICH. As a result, the study did not consider the abovementioned methodological difficulties. In particular, no BP was used, which could, at least in part, explain why the study found a considerably lower PPV compared with our study.

Our study has some strengths. First, we developed algorithms specifically designed for identifying re-ICH thereby being able to account for the abovementioned methodological difficulties associated with using registry data for identifying recurrent events. Second, the algorithms were validated by comparison to a gold standard which was established by reviewing the full medical record and brain scan reports for each case thereby ensuring a high degree of accuracy. Additionally, when establishing the gold standard, we included events registered with an ICH-diagnosis as well as events registered with a wide range of diagnoses for other cerebrovascular diseases thus ensuring completeness of the gold standard. Third, the inception cohort comprised residents of RSD; a geographically well-defined area in Denmark, which previously has been shown to be representative for the Danish population in terms of sociodemographic and health-care characteristics,²⁶ thus ensuring generalizability of the study to the entire Danish population.

Our study has some limitations. First, due to the registry approach used for follow-up, patients with re-ICH that were not admitted to the hospital might have been missed. However, due to the structure of the Danish health-care system where all patients suspected of stroke are referred for acute evaluation at a hospital, we believe that this source of bias only had a minor impact on our results. Additionally, patients dying from re-ICH before hospital admission are not included in our data sources. We considered supplementing our data with information from the Cause-of-Death Registry in an effort to include patients dying from re-ICH before hospitalization. However, the only way to reliably establish ICH as cause of death in such patients would be an autopsy. Since the autopsy rate in Denmark is low,²⁷ it is unlikely that inclusion of autopsy-verified cases of re-ICH would have affected our estimates notably. Supporting this notion is a finding in a previous large Danish study of more than 2800 cases of ICH,¹⁴ where only a single patient’s ICH was established through autopsy (Stine M. Hald, personal communication). Second, some patients might have been missed due to incorrect coding with a non ICH-diagnosis; however, previous studies have found the degree of misclassification of ICH within the Danish registries to be low;^13,28,29 nonetheless, we screened within other cerebrovascular diagnosis codes for ICH, which probably reduced the impact of this potential source of bias. Third, as we did not include outpatient contacts and ED contacts registered with an ICH-diagnosis during the follow-up period for case ascertainment, events recorded exclusively as outpatient/ED contacts might have been missed. However, we know from previous research that this rarely is the case in the Danish setting.¹⁴ Fourth, we excluded events corresponding to 18 patients during case ascertainment; however, as this exclusion was primarily due to medical records being untraceable because of individual hospitals’ storage policies, this source of bias is not systematic and should not affect our estimates of, eg, PPV, although it may have had a minor impact on our estimates of sensitivity. Finally, when collecting data for the gold standard, we did not gather information on inter-rater agreement. However, the study personnel were closely monitored by a senior researcher and were urged to discuss any cases of doubt with the study neurologists and radiologists. Also, in case of doubt, original brain scan images were re-evaluated. We therefore believe that our gold standard is of sufficiently high quality for the purposes of this study.

Conclusion

We developed and validated simple, register-specific algorithms which accurately identified cases of re-ICH and the onset date of re-ICH in DSR and DNPR, respectively. The algorithm using data from DSR more accurately identified events of re-ICH while using a shorter BP compared with the algorithm using data from DNPR.

The results from our study can be used to guide the use of data from DSR and DNPR in further research on the associated risk factors and causes of re-ICH as well as the prognosis of re-ICH.