A Novel Chronic Kidney Disease Phenotyping Algorithm Using Combined Electronic Health Record and Claims Data

Introduction

Chronic kidney disease (CKD) is a serious public health challenge that affects approximately 32 million individuals in the United States (US),¹ with over 6% of individuals having moderate-to-severe CKD (stages 3–5) between 2015 and 2018. It is estimated that the Medicare spending on beneficiaries with CKD exceeded $87 billion, representing 23% of total Medicare fee-for-service (FFS) expenditures in 2019.¹ CKD is associated with poor quality of life, high morbidity, and increased mortality.^2–4 Additionally, it is commonly assessed as a key factor for risk stratification and confounding adjustment in comparative safety and effectiveness research.^5,6

In routine clinical practice, a patient with CKD is identified by measuring their serum creatinine value based on which the glomerular filtration rate can be estimated. While much of the comparative safety and effectiveness research has relied on administrative claims data,⁷ information on laboratory results is not available in claims, and CKD is very unreliably coded as a diagnosis.^8,9 CKD stages based on only International Classification of Diseases (ICD) codes were found to have low sensitivity (3–30%)^9,10 with widely variable positive predictive values (PPVs).^3,9,11,12 Because patients with advanced CKD are often excluded from clinical trials, evidence from real-world data such as claims data is particularly important for informing clinical decisions in routine care. While electronic health records (EHRs) have been increasingly used to generate real-world evidence, missing data due to care provided by provider networks outside of the reach of the research team has been a great threat to study validity.¹³ Yet, relying on claims data may lead to misclassification of CKD in clinical studies based on data lacking detailed laboratory results. Therefore, we aimed to build prediction models for identifying a CKD phenotype determined by laboratory results in EHR using information available in claims data.

Methods

Data Source

We used data from the Research Patient Data Repository (RPDR)¹⁴ linked to Medicare FFS Parts A (inpatient coverage), B (outpatient coverage), and D (prescription benefits) claims data from 2007 to 2017. The RPDR includes longitudinal EHR data from 2 networks in the Boston metropolitan area. The first network (EHR system 1) consists of 1 tertiary hospital, 2 community hospitals, and 19 primary care centers. The second network (EHR system 2) includes 1 tertiary hospital, 1 community hospital, and 18 primary care centers. EHR system 1 was used for training and system 2 for validating the EHR-continuity prediction model. This dataset includes information on body mass index (BMI), blood pressure, smoking status, and laboratory and radiology test results. The Medicare claims data were linked with the EHR dataset by the unique Medicare beneficiary number, date of birth, and sex, with a success linkage rate of 98.7%.¹⁵ Medicare is a US federal health insurance program that currently covers approximately 50 million Americans by providing medical and prescription drug coverage to individuals aged 65 years and older and to younger individuals with disabilities. The Medicare FFS claims database contains longitudinal, individual-level data on healthcare utilization, inpatient and outpatient diagnoses, diagnostic tests and procedures, and pharmacy filled prescriptions. These data are commonly used in real‐world drug effectiveness and safety studies.^15–17 This study was approved by the Institutional Review Board of the Brigham and Women’s Hospital, Boston, Massachusetts.

Study Population

Using the EHR-Medicare linked database, we identified a cohort with a valid serum creatinine laboratory result (measurement date = cohort entry date [CED]) and at least 90 days of continuous Medicare enrollment before and after the CED. Operationally, a valid serum creatinine measurement means a laboratory value with a unit-specific Logical Observation Identifiers Names and Code (LOIC) that corresponds to serum creatinine, excluding those with implausible values (Table S1). The range of implausible values was verified by board-certified physicians on the team (JMP and KJN).

Patients with end-stage kidney disease (ESKD), who were on dialysis, who were younger than 65 years of age, or had missing information on age or sex were excluded.

Cohort Characteristics

Baseline characteristics were assessed during the baseline assessment period (BAP), which was defined as 90 days before and after CED. We considered 134 potential variables based on clinical expertise and a comprehensive literature review.^18–20 These included demographic characteristics (eg, age and sex), CKD-related and other comorbidities (eg, acute kidney injury, diabetes, anemia, etc.), medication use variables, a combined comorbidity score,²¹ a claims-based frailty index (CFI) validated against clinical measures of frailty,^22–25 and healthcare utilization variables (ie, number of outpatient office visits, number of emergency room visits, number and duration of hospitalizations). A full list of included variables is provided in the Supplementary Appendix (Table S2).

Outcome

The study outcomes were continuous estimated glomerular filtration rate (eGFR), and eGFR in categories (<60, <45, and <30 mL/min/1.73m²). This was calculated based on measured serum creatinine using the Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) equation:

where:

S_cr is serum creatinine in mg/dL,

κ is 0.7 for females and 0.9 for males,

α is −0.329 for females and −0.411 for males,

Min indicates the minimum of S_cr/κ or 1, and

Max indicates the maximum of S_cr/κ or 1.

Statistical Analysis

To describe the baseline characteristics of patients included in the study, we used means and standard deviations (SDs) for continuous variables and proportions for categorical variables. The models were developed using data from the training set. We first applied the least absolute shrinkage and selection operator (LASSO) regression with Bayesian information criteria to select predictors from the 134 preselected variables. To get our primary model, we then fit a linear regression model with the continuous eGFR as the dependent variable and the selected predictor variables as independent variables. Because those with eGFR <30 represent an infrequent and severe phenotype, the predictors of this particular phenotype and their coefficients may be different from that for the continuous eGFR. Therefore, in a secondary analysis, we used LASSO regression with Bayesian information criteria to select predictors for the binary outcome of eGFR <30 and fit a logistic regression with the LASSO-selected variables predicting eGFR <30. Model performance characteristics were assessed by calculating the area under the receiver operating curves (AUC) for the predicted probability of eGFR in the training and validation sets. We used the coefficient of determination, R², to assess our primary model predicting continuous eGFR. Additionally, model calibration was assessed using goodness-of-fit to compare the observed and predicted events of CKD, which were tested by Hosmer–Lemeshow tests. We calculated overall accuracy, defined as the number of accurate predictions divided by the number of total predictions, PPVs, defined as the probability of being a true case, given positive prediction by the model, and sensitivity, defined as the probability of being identified as a case of specific eGFR by the model for a true case for that specific level. The optimal cut-off was chosen to maximize overall correctness while keeping a PPV of at least 80% if achievable (choose the optimal PPV if a PPV of 80% is not possible).²⁶ All analyses were conducted in SAS version 9.4 (SAS Institute Inc., Cary, NC, USA).

Results

Patient Inclusion and Characteristics

Out of a total of 718,341 patients in the linked EHR-Medicare dataset, 478,699 (66.6%) had a valid serum creatinine laboratory result. After applying the inclusion/exclusion criteria, 174,220 patients were included in the analysis: 117,476 (67.4%) in the training set and 56,744 (32.6%) in the validation set (Figure 1).

Figure 1 Flowchart of study population derivation. Abbreviations: HER, electronic health records; ESKD, end-stage kidney disease; BAP, baseline assessment period.

Overall, patients included in the training and validation sets had comparable characteristics: the mean age was 74.8 (73.2) years in the training (validation) set, 58.2% (59.6%) were female in the training (validation) set, the majority of patients were white in both sets, mean eGFR was 70.9 (72.5) mL/min/1.73m² in the training (validation) set, and both sets had similar distributions of comorbidities, medication use, and health care utilization (Table 1).

Table 1 Patient Characteristics in the Training and Validation Sets (N = 174,220)

Model to Predict Continuous eGFR Values

Our primary model predicting continuous eGFR included a total of 113 factors with an adjusted R² of 0.354. The top-ranked predictors being history of CKD, age, portal vein thrombosis, liver transplantation, acute kidney injury, hyperkalemia, and use of loop diuretics (Table S2). The mean measured and predicted eGFR values were similar within each predicted eGFR decile in the training vs validation set (Figure 2). For example, the mean measured eGFR in the first decile was 58.3 mL/min/1.73m² (±17.7 SD) in the training set and 59.4 mL/min/1.73m² (±18.0 SD) in the validation set. A similar trend was observed for the SD.

Figure 2 Mean measured eGFR by predicted eGFR decile in the training vs validation sets. Error bars represent standard deviations.

Performance of the Model Predicting eGFR <60 and <45 mL/min/1.73m²

The AUC was similar across the three categorical levels of eGFR score when comparing the validation set to the training set (Figure 3). In the validation set, the AUC was 0.81 and 0.88 when predicting eGFR <60 and <45 mL/min/1.73m², respectively. Based on the optimal cut-off, the overall accuracy was 0.81 (95% confidence interval [CI]: 0.81, 0.81) and 0.90 (0.90, 0.91) when predicting eGFR <60 and eGFR <45 mL/min/1.73m², respectively. The corresponding PPVs were 0.80 (0.79, 0.81) and 0.79 (0.75, 0.84) [Table 2]. Based on the Hosmer-Lemeshow Goodness-of-Fit test, the model tends to overestimate the risks of the high-risk patients, which is more pronounced when predicting eGFR <45 than <60 mL/min/1.73m² (Tables S3 and S4).

Table 2 Performance of the Model Predicting eGFR <60 and <45 mL/min/1.73m2 Categories

Figure 3 Area under the receiver operating characteristic (AUROC) curves of the performance of the eGFR prediction tool in the training and validation set.

Performance of the Model Predicting eGFR <30 mL/min/1.73m²

The primary model predicting continuous eGFR has an AUC of 0.92 in the validation set when predicting eGFR <30 mL/min/1.73m². The secondary model developed to predict the binary outcome of eGFR <30 mL/min/1.73m² included a total of 53 factors. The top-ranked predictors included history of CKD, acute kidney injury, age, use of loop diuretics, hyperkalemia, abdominal ultrasound testing, and liver dysfunction (Table S5). The AUC of this secondary model was 0.93 when predicting binary eGFR <30 mL/min/1.73m² (Table 3). The overall accuracy of the model was similar for the primary and secondary models (0.97, 95% CI: 0.97, 0.98). Compared to the primary model, the secondary model yielded a higher PPV (0.56 vs 0.38) at the cost of lower sensitivity (0.003 vs 0.05, Table 3). Based on the Hosmer-Lemeshow Goodness-of-Fit test, the model tends to overestimate the risks of the high-risk patients (Tables S6 and S7).

Table 3 Performance of the Prediction Model for eGFR <30 mL/min/1.73m2

Discussion

In this study, we developed and validated a model based on information in claims data to identify a CKD phenotype. Based on data from 174,220 Medicare beneficiaries, our model was able to predict a CKD phenotype with good discrimination capability. The AUC for predicting eGFR of <60, <45, and <30 mL/min/1.73m² was 0.81, 0.88, and 0.92, respectively. While the PPVs for the models identifying eGFR of <60 and <45 mL/min/1.73m² were satisfactory (ranging between 0.79 and 0.80), the PPV for identifying eGFR <30 mL/min/1.73m² was suboptimal.

Given the burden of CKD, it is critical to accurately identify patients with CKD for safety and effectiveness studies. In a systematic review of the validity of administrative database coding for CKD, sensitivity and PPVs were substantially variable in 19 studies.¹² Particularly, sensitivity values had a median of 0.41 with a range from 0.03 to 0.88, while PPVs had a median of 0.78 with a range from 0.29 to 1.00. However, the majority of the studies included in the systematic review had relatively small sample sizes, different study populations (eg, some included patients with ESKD), variant algorithms for CKD definition, and different reference standards (eg, medical chart review and laboratory values). Additionally, in a large study of a US commercial insurance claims database, the PPV of a claims-based (ICD-10) algorithm to identify adults with CKD stages 3–5 was 0.86.⁷ The results from our models, especially PPVs, well align with the upper bounds reported in previous studies.

Since patients with advanced CKD are often excluded from clinical trials,²⁷ it is important to evaluate drug safety and effectiveness in patients with advanced CKD in the population using routinely collected information in databases, such as insurance claims data. Development and validation of a claims-based CKD phenotype prediction model can facilitate identification of patients with CKD because laboratory results (eg, serum creatinine) are typically not available and CKD is unreliably coded as a diagnosis in the claims data.^8,9 Currently, there is no diagnosis-code-based algorithm that can identify those with moderate-to-severe CKD (ie, eGFR <45 [stage 3b and worse] and eGFR <30 [stage 4 and worse]). Our model therefore can advance researchers’ ability to investigate the subgroup effects of the clinical questions at hand among patients of higher vulnerability due to advanced renal dysfunction.

The model developed in this study was able to predict with sufficient accuracy a CKD phenotype using routinely collected data. Our cut-off was chosen to optimize PPV at the cost of reduced sensitivity because claims data usually offer a large cohort containing sufficient patients with CKD. Using a cut-off with high PPV can ensure those identified as having CKD have a high probability of having true CKD in which a valid causal inference regarding CKD can be drawn.⁹ The predicted CKD can also be used as a proxy factor for confounding adjustment. Because claims data typically do not have laboratory results, renal function defined by laboratory results would otherwise be an important unmeasured confounder in the analysis. Another possible application of our model is to use the predicted CKD as the clinical end points in a comparative safety analysis. This application will favor high PPV, since relative risk estimates are unbiased if outcomes are assessed with 100% specificity, even if sensitivity is lower.²⁸

Practically, our algorithm can be used as an imputation model in settings where eGFR is critical but not available. This requires the missing at random assumption where all predictors of missingness are observed when imputing eGFR. There are two main reasons that may enhance our confidence in predicting the missingness (ie, the data missing mechanism is missing at random). First, serum creatinine is widely ordered as a routine laboratory test, making it a simple and practical marker to estimating GFR. Second, the risk factors that often prompt clinicians to order serum creatinine test are captured in the claims data, including medical diagnosis (eg, CKD, acute kidney injury, and diabetes), medical procedures (eg, cardiac catheterization), and use of certain medications (eg, diuretics, diabetes medications, and angiotensin II receptor blockers [ARBs]).

The study has several limitations. First, the study population was limited to patients aged 65 years or older with Medicare coverage; thus, we were not able to generalize the results to patients younger than 65 years. Because CKD is highly prevalent in older adults and Medicare claims data are often used in comparative effectiveness research,^29–31 we argue that our models are useful as they are not intended to generalize to the younger populations. Second, the application of our model to other populations relies on the assumption that the relationship between covariates and renal dysfunction is similar across care delivery systems. We used two academic EHR systems in Massachusetts as the training and validation datasets. These two systems both consist of care facility of all care continuum, including primary care center, community hospitals, and tertiary referral centers. Because both care delivery networks are general hospital based rather than specialty-based centers (eg, cancer centers), it is more likely that this assumption can be met, which is supported by the fact that we did not observe reduction in model performance in the validation set. Third, our model relies on serum creatinine as the gold-standard for kidney function, which may underestimate severity of kidney disease in patients with reduced muscle mass.³²

Conclusion

In conclusion, based on a cohort of Medicare beneficiaries aged 65 years or older, we developed and validated a phenotyping algorithm to identify patients with CKD stages 3–5 based on information in claims data. Our models can be used as a proxy factor for confounding adjustment, as a proxy clinical end point during follow-up, and to enhance assessment of subgroup effects by these CKD phenotypes in clinical research using large insurance claims data in the absence of laboratory testing results.