Development and Validation of an Intracranial Hemorrhage Risk Score in Older Adults with Atrial Fibrillation Treated with Oral Anticoagulant

Introduction

Non-valvular Atrial fibrillation (AF) is highly prevalent in older adults (10% for those >80-years-old),¹ and it is associated with a five-fold increased risk of embolic stroke.² Anticoagulation can reduce this risk by 40–80%, but bleeding is its major complication. Clinical practice guidelines^3,4 recommend that AF patients with an estimated annual stroke risk^5,6 ≥2.2% receive an oral anticoagulant (OAC). Following this recommendation, a large majority of older adults with AF should receive an OAC.^7–9 However, nearly 50% of these patients are not anticoagulated,^10–12 with risk of falls being one of the most commonly cited reasons for withholding anticoagulation, primarily due to the fear of intracranial hemorrhage (ICH) following a fall.^10,13,14

There is limited evidence allowing prediction of ICH risk, particularly due to falls, in older AF patients taking OAC. The widely used and guideline recommended HAS-BLED score generates a risk of all-cause bleeding, where extracranial events predominate.^3,15 Indeed, HAS-BLED was developed on a dataset that included only 9 intracerebral hemorrhage events. As a result, it may not accurately predict risk of ICH and a dedicated ICH prediction tool is needed.¹⁵ ICH events are generally far more likely to be fatal or disabling than extracranial bleeds and there are differences in their underlying mechanism.¹⁶ Here, we report the development and validation of an ICH risk prediction model among adults aged 65 or older with non-valvular AF using a nationally representative sample of Medicare fee-for-service population. We compare our AF ICH risk prediction model’s performance with the HAS-BLED score and the Homer score, a validated fall risk tool based on routinely collected clinical.¹⁷

Methods

Study Design and Population

This retrospective cohort study using Medicare claims data was approved by the Institutional Review Board of the Brigham and Women’s Hospital, Boston, Massachusetts. Data access complied with the data protection and privacy regulations set in the Data Use Agreement with the Centers for Medicare and Medicaid Services. The study cohort consisted of Medicare beneficiaries ≥65 years old who filled a prescription for one of the 5 OACs (i.e., warfarin, dabigatran, rivaroxaban, apixaban, and edoxaban) in 2010–2017 with no OAC exposure in the past 365 days and had ≥1 inpatient or outpatient diagnosis of AF in the preceding 365 days. The date of dispensing was the cohort entry (index) date. Over the baseline period, defined as the preceding 365 days from index date, patients were required to have continuous enrollment in Medicare Part A (inpatient), B (outpatient), and D (prescription coverage) without enrollment in a Medicare Advantage Plan. We excluded those who: 1) had missing age or sex; 2) or had valvular heart disease; and 3) or had a contraindication to DOAC or warfarin therapy, in the past 365 days (Supplementary Materials Figure S1 and Table S1). This study followed the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) reporting guidelines for prediction models (Supplementary Materials TRIPOD Checklist).

Ascertainment of ICH

The primary outcome was any ICH event, including intraparenchymal, sub-arachnoid, subdural hemorrhage (SDH), intraventricular, and epidural bleeds, ascertained from the primary diagnosis of inpatient care setting claims based on a validated algorithm (PPV = 89–97%, Supplementary Materials Table S2).^18,19 In a sensitivity analysis, we also evaluated the model performance predicting intraparenchymal hemorrhage (IPH) and SDH, individually. Follow-up began on the index date and continued for 365 days or until the first of an ICH event, death, disenrollment or end of available data.

Measurement of Candidate Predictors and Descriptors

During the baseline period, we assessed demographic variables (age, sex, race [White, Black, Asian, Hispanic, Other/Unknown]), 34 medications captured by National Drug Codes (NDCs)²⁰ of dispensed prescriptions, 27 comorbidities based on International Classification of Diseases, Ninth and Tenth Revision, Clinical Modification (ICD-9-CM and ICD-10-CM) codes, and 6 measures of healthcare resource utilization as candidate predictors based on potential clinical associations (Supplementary Materials Table S3). We also included 23 predictors of frailty based on Current Procedural Terminology-4 codes and Healthcare Common Procedure Coding System level II codes, such as oxygen delivery systems, walking aids, and wheelchairs, as previously defined in a published claims-based frailty prediction model.²¹ We assessed the CHA₂DS₂-VASc score over the baseline period only as a descriptor of the patient population and not as candidate predictor.^5,6

Alternative Models for Comparison

Based on inpatient and outpatient claims in the baseline period, we assessed a modified HAS-BLED score (hypertension, abnormal renal/liver function, stroke, bleeding history or predisposition, elderly, drugs/alcohol concomitantly) (Supplementary Materials Table S4). Note that labile international normalized ratio (INR) was not included because such information is not available in the Medicare data and is not relevant for patients not previously treated with OACs nor for the direct anticoagulants, generally.^15,22 The Homer score was assessed over the baseline period and comprised of age, sex, 36 dispensed medications captured by NDCs and categorized by the Anatomical Therapeutic Chemical Classification system,²³ and diagnoses from 26 non-overlapping Clinical Classifications Software categories captured by ICD-9-CM and ICD-10-CM codes.¹⁷

Statistical Analysis

We split the study population into training and validation sets by region. Patients from non-Midwest regions (Northeast, South, West, Other/Unknown) were included in the training set and patients from the Midwest region were included in the validation set. No prior data suggest geographical region is associated with ICH risk and the geographical variations in patient demographics and local practice differences can be used to test generalizability of our models. In the training set, we used regularized regression (i.e., elastic net) with ten-fold cross-validation to select informative candidate predictors while optimizing the bias-variance trade-off by shrinkage of model coefficients.^24–26 Then, we enter the selected variables in a Cox proportional hazards regression to predict 1-year risk of ICH, which accounts for both the occurrence of the outcome and time to the event. In the training and validation set, we computed the area under the ROC curve (AUC) to assess discrimination ability and Hosmer-Lemeshow goodness of fit test²⁷ to compare the proportions of predicted and observed ICH cases by decile of predicted ICH risk. In the validation set, we compared the performance of our AF ICH risk prediction model vs. the HAS-BLED bleeding risk score and the Homer score by AUC using Delong’s test²⁸ for 1-year risk of ICH, which accounts for correlation of evaluating AUC in the same population. We also compared performance based on clinical reclassification which allows comparison of the performance of different models in classifying patients into multiple risk groups.²⁹ We used 0.40%, the estimated 1-year risk of ICH in the US population aged 65 years or older with AF not on OAC,⁷ and twice this estimate, 0.80%, to define low, intermediate, and high-risk groups. We examined the classification by HAS-BLED score and calculated the proportion of subjects who were reclassified when the new AF ICH risk model was applied. We also calculated net reclassification improvement (NRI), which measures overall improvement in risk classification among ICH cases and among non-cases, and again allows for comparison of calibration in classifying patient into clinically relevant high, intermediate, and low risk groups.³⁰ We also repeated the reclassification and NRI calculations after recalibrating both scores by fitting univariate logistic regressions with the respective score as the only predictor of ICH in the training set. We also used the Youden Index to select a predicted probability cut-off to identify individuals unlikely to have ICH and calculate the corresponding negative predictive value.³¹ Analyses were conducted in R version 4.1.1 and the Aetion Evidence PlatformⓇ (2023) (including R version 3.4.2).³² All reported p-values were based on two-sided tests with a significant level of 0.05.

Sensitivity Analyses

We conducted iv sensitivity analyses. First, we assessed if adding interaction terms between demographic variables (age [≥75 vs. <75 years], sex, and race [White vs. non-hite]) and all the predictors would improve the model performance by repeating our model development process in the training set. This analysis was used to test whether the original model assumption of no interaction substantially reduced model performance or not. Second, to account for informative censoring and the competing risk of death, we built an inverse probability censoring weighting (IPCW) model by using a regularized Cox regression (elastic net) to select among 92 component factors of claims-based frailty index to predict a composite outcome of censoring due to death, disenrollment, or end of data.²¹ Using the IPCW model, we generated and applied censoring weights to our study cohort to create a pseudo-population in which no one was censored due to death, disenrollment, or end of data. Then, we repeated our ICH model development process in the pseudo-population to predict ICH risk. Third, to assess whether the AF ICH model is robust in predicting ICH subtypes we assessed the AF ICH model’s AUC for 1-year risk of IPH and SDH outcomes individually. Fourth, to address the recent decrease in warfarin initiation, we excluded warfarin users in the training and validation sets and re-assessed the AF ICH model, HAS-BLED score, and Homer score’s AUC for 1-year risk of ICH, IPH, and SDH. Fifth, similarly, we excluded warfarin users in the training and validation sets, repeated the ICH model development process, and evaluated the AUC for a 1-year risk of ICH.

Results

Study Cohort

The study cohort was comprised of 629,804 individuals (mean age 77.5 [SD 7.4], 51.8 female) in the training set and 210,216 individuals (mean age 7.4 [SD 9.0], 53.3 female) in the testing set (Figure 1). The mean CHA₂DS₂-VASc and HAS-BLED score was 4.7 [SD 1.8] and 2.3 [SD 0.8] in the training and 4.7 [SD 1.7] and 2.3 [SD 0.7] in the validation set, respectively (Table 1). There were 3963 ICH events in the training set and 1397 ICH events in the validation set. The corresponding ICH event rate was 7.5 per 1000 person years and 7.9 per 1000 person years, respectively. This included IPH rates of 2.51 and 2.46 per 1000 person years for the 1331 and 434 IPH events in the training and validation sets, respectively, and SDH rates of 3.11 and 3.41 per 1000 person years for the 1648 and 603 SDH events in the training and validation sets, respectively. Over the course of follow-up, 76,729 (9.1%) patients died, 26,599 (3.2%) disenrolled, 146,355 (17.4%) reached the end of available data before 365 days of follow-up, 584,977 (69.6%) survived the 365-day follow-up period without an ICH event, and 5360 (0.6%) experienced an ICH event.

Table 1 Selected Characteristics of the Study Population

Figure 1 Study cohort attrition chart.

Development of the AF ICH Risk Model

Among 93 candidate predictors, regularized Cox regression with ten-fold cross-validation selected the demographic variables (age, sex, race), 5 healthcare resource utilization variables, 7 frailty-related predictors, 16 comorbidities, and 10 dispensed prescription medications (Table 2). The AUC of the AF ICH risk model predicting 1-year risk of ICH was 0.653 95% CI (0.645, 0.661) in the training set and 0.647 95% CI (0.633, 0.661) in the validation set. In the first nine deciles of predicted ICH risk, the difference in the proportion of predicted ICH cases and observed ICH cases was less than 0.17 in the training set and 0.11 in the validation set. In the tenth decile of predicted ICH risk, the difference was 0.56 in the training and 0.49 in the validation set (Figure 2).

Table 2 Prediction Model for the Risk of 1-Year Risk of Intracranial Bleeding

Figure 2 Observed vs. predicted ICH events based on AF ICH risk model. Abbreviations: ICH, intracranial hemorrhage; AF, atrial fibrillation. Note: Chi-squared values based on the Hosmer-Lemeshow test were 190.1 (p<0.001) in the training set and 46.4 (p<0.001) in the validation set.

Comparison of AF ICH Risk Model, HAS-BLED and Homer Scores

In the validation set, the AUC of the AF ICH risk model predicting 1-year risk of ICH was substantially superior to that of the HAS-BLED score (0.647 vs. 0.567, p<0.001) and moderately superior to that of the Homer score (0.647 vs. 0.623, p<0.001) (Figure 3). In the training and validation set, respectively, the Hosmer-Lemeshow goodness of fit test yielded a chi-squared of 190.10 and 46.41 for the AF ICH risk model, <0.01 and 9.18 for the recalibrated HAS-BLED score, and 192.27 and 51.47 for the recalibrated Homer score. Without recalibration, the chi-square values of the Hosmer Lemeshow goodness of fit test of the training and validation set were 7,947,912 and 2,592,514 in the HAS-BLED score and 162,701 and 54,831 in the Homer score. P-values for all Hosmer-Lemeshow goodness of fit tests were all <0.001, except for the recalibrated HAS-BLED score with p-values of 1.00 and 0.33 in the training and validation set, respectively. In the validation set, our AF ICH risk score reclassified 57.8%, 42.5%, and 43.9% for HAS-BLED score and 0.0%, 44.1%, and 19.4% for the Homer score of low, intermediate, and high-risk patients with an NRI of 15.3% (12.0%, 18.8%) and 21.9% (18.4%, 25.1%), respectively (P<0.001) (Table 3). Using our prespecified low-risk group (predicted ICH risk = 0.4%) as the cut-off to prescribe OAC, only 0.28% of the patients developed ICH within one year after the cohort entry. We observed similar findings in the training set (Supplementary Materials Table S5). Using a predicted probability cut-off of 0.566% based on the Youden Index, our model had a negative predictive value of 99.7% in both the training and validation sets. At the cut-off based on the Youden Index, our model yielded a sensitivity of 77.8% and 77.9%, specificity of 44.0% and 43.0%, and positive predictive value of 0.9% and 0.9% in the training and validation sets, respectively.

Table 3 Comparison Between Recalibrated HAS-BLED Score and Homer Score vs. New ICH Model: Reclassification of Predicted 1-Year ICH Risk Categories in the Validation Set

Figure 3 Comparison of AF ICH risk model vs. HAS-BLED score and vs. Homer fall risk model: AUC in predicting 1-year risk of ICH. Abbreviations: AF, atrial fibrillation; ICH, intracranial hemorrhage; AUC, area under receiver operating characteristic curve. Note: (A) (left) displays the model performance in the training set and (B) (right) displays the model performance in the validation set.

Sensitivity Analysis

The AF ICH prediction model that included interaction terms between demographic variables and other predictors did not have higher AUC in the training set (0.651 with interaction vs. 0.653 without interaction; p = 0.074) and in the validation set (0.644 with interaction vs. 0.647 without interaction; p = 0.232) (Supplementary Materials Figures S2 and S3). Considering the competing risk of death, the AF ICH IPCW prediction model yielded similar results with an AUC of 0.653 in the training set of and 0.647 in the validation set (Supplementary Materials Tables S6, S7 and Figures S4, S5). Our model also had similar AUCs in predicting IPH and SDH, individually, in the training set (0.631 and 0.665, respectively) and in the validation set (0.625 and 0.664, respectively) (Supplementary Materials Figures S6–S9). In sensitivity analyses excluding warfarin users, our model remained superior to HAS-BLED and Homer Score in re-evaluation of AUC predicting ICH with an AUC of 0.651 and 0.653 in the training and validation set, respectively (Supplementary Materials Table S8 and Figures S10, S11). Re-development of the AF ICH model in a training set excluding warfarin users improved AUC performance in the training set from 0.651 to 0.658, but did not improve AUC performance in the validation set (0.653 vs. 0.642) when compared to the original AF ICH model performance in this subset of DOAC users (Supplementary Materials Tables S8 and S9).

Discussion

ICH is the most severe adverse effect of OAC.¹⁶ A well-calibrated claims-based score predicting ICH risk while on OAC can assist the OAC prescribing decision in the predominantly older population of patients with AF, who typically have multiple comorbidities and are taking multiple medications.^4,34 Such a claims-based score can also balance ICH risk in comparative effectiveness studies of OACs. We developed and validated an ICH risk prediction score in a large, representative population of older adults with AF receiving OAC. Our model substantially outperformed the widely used HAS-BLED score, which targets the whole range of bleeding events, most of which are extracranial and generally less severe than ICH events. Our model also markedly outperformed the Homer fall risk score, where fall risk may be used as a proxy for ICH risk.

Anticoagulation is commonly withheld in AF patients with a history of recurrent falls or those at high fall risk due to the risk of ICH following a potential fall, but this prescribing decision is often subjective and not based on evidence. One simulation-based decision analysis that strongly argued against withholding OAC due to risk of falls has been widely cited for nearly 2 decades,³⁵ yet it has hardly influenced underutilization of OAC for AF. A limitation is that the input data for the simulation were mainly drawn from the selected trial populations^36,37 where frail patients with recurrent falls were severely under-represented. To aid shared decision-making based on evidence from routine care, researchers can use our AF ICH score to assess the predicted risk of ICH on OAC treatment.

To the best of our knowledge, our model is the first dedicated model that predicts ICH risk in older adults with AF. It is not surprising that our model dedicated to predicting ICH risk based on a very large national sample outformed the HAS-BLED score developed for all-cause bleeding risk in a much smaller dataset with few ICH events.¹⁵ Our study also demonstrated that the Homer score¹⁷ that was developed using a machine-learning approach predicting fall risk performed suboptimally when predicting ICH risk, although it did perform better than the HAS-BLED score. With the wide availability of electronic health records (EHR) and institutional clinical informatics support, clinicians are no longer limited to manual calculation of clinical risk scores based on few variables. We used a machine-learning approach that yields an interpretable model with coefficients associating each covariate with ICH risk (Table 2), so the quality of the covariate measurement can be evaluated. Our score based on 41 factors can be readily calculated from routinely collected data by a program embedded in an EHR. There are increasing uses of automatically calculated risk scores in EHR in routine care to aid clinical practice.^38,39 Physicians can potentially use this AF ICH risk score to inform the prescribing decision of OAC. Our score can also be readily computed based on administrative claims data and used as a risk stratification tool or as a care-quality metric.³⁹

Another potential application of our model in the research context is to improve confounding adjustment and subgroup analyses. The risk of ICH is an important confounder and potential effect modifier for comparative effectiveness and safety analyses of OAC.^40–42 However, because the low incidence of ICH, its risk often cannot be estimated in small cohorts with statistical precision. Our model can be helpful in quantifying the propensity for ICH risk, with comparable performance in predicting the IPH and SDH subtypes. The predicted risk of ICH can be used as an adjustment tool to balance the background risk of ICH across the treatment arms and for risk stratification when assessing subgroup effects of OACs.

Based on our model, the predicted and observed risks of ICH align well in the low-risk patients, but the model can overestimate ICH risk in high-risk patients. Because clinicians tend to be overly conservative when withholding OAC for high risk of falling,^10–12 we believe our model is most useful in identifying the low-risk group for safe prescribing of OAC. Using our prespecified low-risk group cut-off (predicted value = 0.4%) to prescribe OAC, we observed that 0.28% of the patients developed ICH within one year after the cohort entry in the validation set. The estimated 1-year risk of ICH in the US population aged 65 years or older with AF not on OAC was 0.40%.⁷ Since the estimated risk of ICH is not higher than that in AF patients not on OAC, we recommend OAC not to be withheld in the low-risk group based on our model. OAC decisions in the higher ICH risk need to be individualized, depending on the estimated risk of ischemic stroke and patient preference. For a rare outcome like ICH, it is expected that using the model to identify those with high likelihood of having ICH will inevitably select a subgroup in which a substantial number of patient will not have this rare event (i.e., positive predictive value may be low). Instead, our model demonstrated its utility to identify individuals unlikely to have ICH with a negative predictive value of 99.7% in the training and validation sets using the Youden Index based cut-off of 0.566%. Therefore, we recommend using our model to identify those with a very low likelihood of having ICH and consider prescribing OAC despite a high perceived fall risk.

Our study has limitations. First, our database did not have laboratory results, gait speed, balance measures, cognitive function, and vital signs, and we cannot assess their associations with the risk of ICH and of falls. However, we did use components of a validated frailty index to partially account for the effect of these features. Next, the predictors identified in our study should not be interpreted as having a causal effect on the ICH or falls. Lastly, our study cohort consists of older adults with AF, of which 76,729 (9.1%) patients died within one year, and competing risk due to death may be of concern. Yet, it is reassuring that accounting for competing risks due to death using IPCW yielded similar results.

In conclusion, we developed and validated a claims-based score predicting 1-year ICH risk among older adults with AF initiating OAC. This score performed substantially better than the standard HAS-BLED score and the Homer fall risk score. This score can assist the OAC decision in individual AF patients and can also balance ICH risk in comparative effectiveness studies of OACs.