Testing a Cloud-Based Model for Active Surveillance of Medical Devices with Analyses of Coronary Stent Safety Using the Data Extraction and Longitudinal Trend Analysis (DELTA) System

Introduction

Timely identification of medical device safety concerns, particularly early safety signal identification, has been a challenge using current approaches, eg, post-approval studies that are slow to enroll and often not completed and “passive” surveillance through adverse event reporting. The National Evaluation System for health Technology Coordinating Center (NESTcc) sought to strengthen the system of post-market medical device safety and effectiveness evaluation using real-world data (RWD).¹ The NESTcc explored a cloud-based infrastructure for the primary purpose of active surveillance using RWD from electronic health records (EHRs) and other sources.² The Data Extraction and Longitudinal Trend Analysis (DELTA) system was chosen as an active surveillance tool for identifying safety signals in NESTcc’s analytics environment. DELTA is an open-source software package that supports a variety of statistical surveillance methodologies that can be configured to ingest new data and perform sequential analyses on predefined schedules. [See Supplemental Content.]

DELTA accesses databases containing patient characteristics, treatment exposures, and outcomes, and then utilizes a variety of statistical methodologies, including risk-adjusted prospective surveillance analyses, survival studies, sequential analyses, and extensive propensity matching algorithms, to analyze the data. The application provides a single graphical user interface to configure the analyses which are executed via scripts written in the R programming language.

The full spectrum of DELTA analytic methods was initially tested and validated within the NEST cloud environment using complex synthetic datasets and historical clinical data from a single healthcare system. The validation analysis using complex synthetic datasets demonstrated comprehensive accuracy of the DELTA cloud implementation as compared to results generated on a single, local, server (Supplemental Content Tables S1 and S2). Separately, the DELTA cloud implementation was used to evaluate the safety of drug-eluting stents as compared with bare-metal stents, using previously analyzed data, and accurately replicated the published analysis (Supplemental Content Table S3 and Figure S1).

The purpose of the current study was to further demonstrate the feasibility of using DELTA for cloud-based active surveillance by analyzing the relative clinical performance of two commonly used drug eluting coronary stents [zotarolimus-eluting stents (ZES) and everolimus-eluting stents (EES)] using RWD from two health systems as a prototype for future cloud-based active surveillance efforts. Importantly, the selection of ZES and EES as devices to evaluate through this validation exercise was based on frequency of device usage and structured data availability, given the harmonization of data element definitions between Mercy and Lahey through their common use of the National Cardiovascular Data Registry’s (NCDR^®) CathPCI Registry^®.

Methods

Data Sources

Data were sourced from both the Mercy Building Unique Device Identifiers (UDIs) Into Longitudinal Data for Medical Device Evaluation (BUILD) database and the Lahey Health Observational Database Environment (LHODE). BUILD is an FDA and industry-funded initiative in which three large health systems created UDI enriched research databases (UDIRs) and formed a distributed data network.³ The only BUILD data used in the current study were from the Mercy UDIR, which has been described in detail elsewhere⁴ and which was extensively redesigned to conform to the BUILD common data model (CDM). The BUILD CDM is comprised of elements from the CDM developed for the FDA-sponsored Sentinel program that include longitudinal electronic health record (EHR) and payer data⁵ as well as cardiology-specific tables that closely followed CathPCI data content.⁶ In addition, the CDM included tables of publicly accessible device data from the FDA Global UDI Database (GUDID) available from AccessGUDID at the National Library of Medicine⁷ and from the “augmented UDI” (AUDI) table containing nine stent-specific attributes selected by an expert panel of cardiologists during the Mercy demonstration project.⁸ In addition to the device data, Mercy UDIR data consist of all discrete fields within the EHR including billing, data from Mercy’s cardiac cath lab clinical systems, and data from systems used to report to the CathPCI Registry^®. Likewise, the Lahey Health submitted data also used all EHR-derived data, including billing information, as well as the CathPCI abstracted data submissions.

The Lahey Health Comparative Effectiveness Research Institute (CERI) maintains a data warehouse that is dynamically updated with structured demographic, clinical, laboratory, and billed claims data for all patient encounters in any of the Lahey Health System’s facilities, extracted from the Lahey electronic health data system. Additionally, data elements from Lahey’s CathPCI submission database are linked to CERI data warehouse to provide additional procedural information, patient characteristics, and device-specific information related to the use of coronary stents. The Lahey CERI data warehouse is also linked to the Social Security Death Master File for ascertainment of vital status at 1-year post-procedure.

Study Design

All patients receiving ZES or EES within the Mercy Health BUILD database and Lahey Health system between July 1, 2015 and December 31, 2017 were included in the study. Patients whose stents lacked attributes in AUDI or AccessGUDID associated with their UDIs, ie, drug-eluting status; had invalid medical records; or had multiple stents with mixed drug attributes (both ZES and EES) were excluded from the analysis. Data for this analysis were ingested into the NEST cloud, which is a cloud-based computing environment developed by NESTcc to support active surveillance with RWD. The DELTA 3.7 system was used to perform comparative safety analyses integrating data from both healthcare sources.

The primary outcome was freedom from death at one year. Secondary endpoints included freedom from major adverse cardiovascular events (MACE) at 1 year. MACE is defined as a composite of death, stroke, acute myocardial infarction (AMI), repeat PCI after the index procedure, or the need for coronary artery bypass grafting (CABG). Each component of the composite endpoint was also analyzed separately as secondary endpoints. Prior randomized trials had demonstrated no significant differences in the performance of the two stent types, and there was no suggestion from observational studies or other information sources to suggest that there was a safety concern with either device.^9–16

Covariates and outcomes were defined using primarily billing codes from a combination of data sources, and a patient was identified as having a covariate or outcome of interest if any one of the available data sources indicated such.

Lists of International Classification of Diseases 9th and 10th Revision Clinical Modification (ICD-9-CM and ICD-10-CM, respectively) diagnosis codes and Current Procedural Terminology (CPT) codes, termed “value-sets”, were created to identify covariates and outcomes of interest (see Tables S1 and S2). Each institution then queried their data warehouse to determine if the ICD or CPT codes in the value-sets were added to a patient’s medical record in the routine course of care.

Both Mercy and Lahey submit data to the NCDR^® CathPCI Registry^® and store the submissions in research databases to supplement RWD collected from electronic health records. In addition to ICD codes, all covariates and some outcomes were partially identified using the original data submitted to NCDR^®. The specific CathPCI Registry^® data element sequence numbers referenced for the purposes of this study are provided in Tables S1 and S2.

Mortality status was sourced from the EHR at both institutions. The level of missing mortality in the analyzed data was judged to be insignificant in the BUILD study based on similar mortality to that in clinical trials.^3,9,11 The CERI data warehouse additionally contains death information sourced from the Social Security Death Master File to identify deaths not reported to Lahey nor captured through updates to a patient’s medical record.

Statistical Methodology

A propensity score model was constructed using known predictors of post-drug eluting stent (DES) mortality including coronary artery disease (CAD) presentation (categorical), left main (LM) coronary artery stenosis (continuous), source of admission (categorical), history of heart failure (categorical), smoking (categorical), dyslipidemia (categorical), hypertension (categorical), and dialysis (categorical). A complete list of covariates is included in Table 1.

Table 1 Covariate Balance Before and After Weighting

Inverse probability treatment weighting (IPTW) was performed by assigning a weight to each case, derived as the inverse of the probability of receiving the respective treatment, which shrinks the influence of those patients receiving the device of interest proportional to their degree of dissimilarity to those patients receiving the alternative device. Thus, patients who are most similar, as measured by the propensity score, to the patients receiving the alternate device have a greater influence on the results, as compared to those patients who are less similar. Weights were trimmed at the 5% and 95% levels to reduce the influence of extreme outlier propensity values. Baseline characteristics after propensity score weighting are described, and standardized differences were calculated as the difference in means or proportions divided by the standard error (SE), with an absolute value <0.10 indicating adequate balance in the distribution of the covariate. For IPTW, the measure of interest was the adjusted odds ratio (OR_adj) for the exposure covariate, namely: treatment with EES, for each outcome: mortality and MACE at 12 months. The result was considered significant for increased risk of an adverse event if the OR_adj was greater than 1.0, with a 95% CI excluding 1.0.

As a further sensitivity analysis, the data were re-analyzed using propensity score matching survival methods¹⁷ in addition to the IPTW approach. In the propensity-matched analysis, those patients treated with the device of interest (EES) were identified and matched in a 1:1 manner using a greedy matching algorithm with patients treated with the comparator stent (ZES) and using a caliper width equal to 0.2 of the standard deviation of the logit of the propensity score. A matching radius of 90 days (ie, control cases performed up to 90 days prior to, or after, the procedure of interest) was used for this analysis. We used the Kaplan–Meier estimator¹⁹ to estimate the survival curves for freedom from death and MACE events in each of the matched treated and control groups. Time to event was defined as the number of months from the date of treatment with EES in the treated matched group and ZES in the control matched group. We used the Log Rank test²⁰ to compare the survival curves between matched groups for both events. A significance level of <0.05 was used to reject the null hypothesis that there is no difference between the curves.

All analyses were performed within DELTA version 3.7 using R (version 4.0)¹⁸ based packages for propensity weighting, matching, and survival analyses.

Missing covariate data were handled using univariate imputation rules, assuming absence of a condition for dichotomous variables, and using the gender-specific median value for continuous variables. Any covariates with more than 5% missing values were excluded from the propensity score model.

This study involves a secondary analysis of Mercy’s BUILD data. BUILD was granted exempt status by the Mercy IRB (ID number 865972-3). The project was submitted to the Lahey IRB for review and was determined not to be human subjects research (ID number 1692573-1).

Results

A total of 4681 patients (Mercy 2637, Lahey 1769) underwent PCI with EES or ZES implantation during the observation period. After applying exclusions, there remained 4336 patients (Mercy 2581, Lahey 1755). A total of 56 Mercy patients were excluded either for invalid records or because they had a second procedure within 14 days of the index procedure. Within the Lahey patients, 14 were excluded as they received both ZES and EES devices during the index procedure. Of the included patients, 3334 patients (76.9%) received EES (Mercy 2356, Lahey 978) and 1002 (23.1%) received ZES (Mercy 225, Lahey 777).

Propensity Weighting

Baseline characteristics of significance and their distribution in patients receiving EES and ZES including age, gender, CAD presentation [stable angina, unstable angina, non-ST elevation myocardial infarction (NSTEMI), or STEMI], most severe percent stenosis in the LM coronary artery, source of admission, history of myocardial infarction, heart failure, diabetes mellitus, smoking history, dyslipidemia, hypertension, and current dialysis are shown in Table 1. The treated populations were similar between the two organizations, and the overall covariate distributions are typical of contemporary interventional cardiology practice, with a mean age of approximately 66 years, 70% male and 34% with a history of diabetes at the time of their index procedure. The results of propensity score weighting are also shown in Table 1.

Prior to weighting, there were significant imbalances between the stent treatment groups in the proportion of patients with “prior myocardial infarction” and “emergency procedures”. After IPTW, residual standardized differences were significantly improved, with only 2 of 15 covariates having residual imbalance, defined as a standardized mean difference of >0.1. These two covariates, which are both associated with higher risk of mortality and MACE, were imbalanced in opposite directions, with a higher proportion of ZES treated patients having a prior history of myocardial infarction, and a larger proportion of EES treated patients undergoing the stent procedure as an emergency.

Mortality

One-year mortality was not significantly different in patients receiving EES compared to those receiving ZES [adjusted odds ratio = 0.97 (95% CI 0.81–1.17); p = 0.780].

MACE

The one-year incidence of MACE was not significantly different in patients receiving EES compared to those receiving ZES [adjusted odds ratio = 1.04 (95% CI 0.92–1.16; p = 0.551)].

Propensity Matching

Table 2 shows the results of the propensity score matching analysis.

Table 2 Propensity Match Covariate Balance Table

Matches to patients receiving EES were found for 992 of the 1002 patients receiving EES, with absolute standardized differences ranging from 0.002 to 0.077 for the 15 covariates (well less than the pre-specified 0.10 criterion). Unmatched EES patients were more likely than ZES patients to be female and to have had an emergency procedure.

Mortality

One-year survival curves for matched patients receiving only EES (“treated”) and those receiving only ZES (“control”) along with 95% confidence bands (CB) are shown in Figure 1. The CB overlap at all time points, indicating there is no significant difference in one-year mortality in patients receiving EES (547 of 992 of EES patients remained available for analysis, after censoring, at the end of 12 months) compared to ZES (546 of 992 patients). The Log-Rank statistic was 0.3348 (p = 0.563) indicating no difference in survival curves.

Figure 1 Survival Summary Report: Freedom from Death at 12-months - EES vs ZES.

MACE

One-year MACE-free survival curves for matched patients receiving only EES (“treated”) and those receiving only ZES (“control”) along with 95% confidence bands (CB) are shown in Figure 2. The CB overlap at all time points, indicating that there is no significant difference in one-year MACE in patients receiving EES (484 of 992 EES patients were free from MACE and remained available for analysis at the end of 12 months) compared to ZES (510 of 992 patients). The Log-Rank statistic was 0.0534 (p = 0.817) indicating no difference in survival curves.

Figure 2 Survival Summary Report: Freedom from Major Adverse Cardiac Events (MACE) at 12-months - EES vs ZES.

Conclusion and Discussion

In this retrospective, observational, analysis, we found low rates of adverse events and no difference in the one-year safety profiles of the predominant drug-eluting stents available in the United States during the study period. The current study demonstrates the feasibility of leveraging cloud-based active surveillance of multi-health system real-world data sources in the evaluation of comparative safety of commonly used cardiovascular devices. Reassuringly, no meaningful differences in the 1-year rates of death, MACE or the individual components of MACE between ZES and EES were identified using either the primary IPTW analysis or using the propensity-matched survival methods. These results are consistent with those of randomized controlled trials^9–16 comparing the two stent types as well as the findings of Mercy’s prior analysis.³

These findings also demonstrate the potential value of applying active surveillance tools, such as DELTA, to real-world EHR data from multiple healthcare systems within a cloud computing environment. Routinely applying active surveillance tools to EHR data may provide physicians, patients, regulators, and manufacturers with near real-time assessment of medical device performance. EHR-based surveillance, though challenged by data quality and completeness issues, offers unique advantages in terms of granularity as well as timeliness, relative to other commonly used real-world data sources, such as claims data and registry data. The results of the current demonstration support the continued development of cloud-based, multiple health system medical device surveillance systems as part of enhanced post-market medical device evaluation efforts.

Limitations

As with all observational studies, the risk of residual confounding may have influenced the results of this analysis. Propensity scoring and matching methods were used to mitigate this possibility, though minor residual imbalances in prior myocardial infarction and emergency remained that may have impacted the results. Separately, this analysis was performed on retrospective data simulating a prospective study, thereby introducing an additional potential bias related to more complete case-level information being available at the time the analysis was performed than would have likely been available if the data had been analyzed with accruing data. Delays in the availability of data for surveillance may be due to refresh rate intervals for data transformation (4–6 weeks at Lahey), as well as scheduled data cleaning and verification steps. Thus, in an optimized prospective active surveillance analysis, it would be expected that incomplete data availability would reduce the power of the analysis earlier in the course of the study, with expected convergence toward the current findings as the RWD were captured for all cases. Additionally, although we included a robust set of baseline covariates in our analysis, some variables that might be expected to affect mortality, eg, the number of implanted stents and adherence to antiplatelet and other drugs, were not captured. There is, however, no reason to expect that these would be significantly different between the two stent types. Also, our primary goal was to identify any potential safety signal, which, if found, would require further evaluation that would include factors not easily obtainable from real-world data sources.

From a technical perspective, the EHR data were ingested into a single cloud environment, then processed and pooled prior to processing by DELTA. The data harmonization enabling analysis from multiple health system sources was a good proxy for the challenges that will be associated with future work that is expected to be more complex when active surveillance is applied to medical devices that are less well characterized and when more complex privacy preserving techniques and analytic architectures are required by participating health systems.