Validity of Routine Health Data To Identify Safety Outcomes of Interest For Covid-19 Vaccines and Therapeutics in the Context of the Emerging Pandemic: A Comprehensive Literature Review

Background

In December 2020, within the context of a rapidly evolving pandemic where effective treatments were not yet available, there was a need for rapid generation of safety and effectiveness data in the real-world to supplement what was learnt during clinical trials.¹ Routine health data, that is, data captured during routine clinical care, such as electronic medical records (EMR) or healthcare insurance claims data,² are useful resources that are perfectly placed to quickly and efficiently provide information about important safety endpoints of interest in large patient populations.³ Furthermore, monitoring the prevalence of these endpoints in the general population can serve as an important baseline comparator against the observed rates occurring in the vaccinated/treated population.⁴

However, routine health databases are created for healthcare planning, monitoring and in some cases for insurance claims reimbursement coordination, not research.³ It is necessary to understand the validity of diagnostic algorithms that use diagnostic codes to accurately identify key endpoints of interest as this will impact the ability of epidemiological research to robustly detect safety signals that emerge for these therapies and vaccines within routinely collected health data.

We therefore conducted a literature review to provide an overview of the availability of validated diagnostic algorithms for selected safety endpoints for COVID-19 vaccines and therapeutics in routine health data in North America and Western Europe in order to understand the robustness of real-world evidence that may be generated in non-trial settings in the context of an emerging pandemic.

Methods

Search Strategy

The literature search was conducted in Medline and EMBASE up until the 1st December 2020. Additional searches were conducted by hand-searching reference lists of key articles. Outcomes were selected from the Safety Platform for Emergency Vaccines (SPEAC) as potential Adverse Events of Special Interest (AESI) for COVID-19 vaccines.⁵ The selected outcomes were cardiovascular outcomes (myocardial infarction (MI), arrhythmia, myocarditis, acute cardiac injury (ACI), vasculitis/vasculopathy), venous thromboembolism (VT), cerebrovascular outcomes (stroke), respiratory outcomes (respiratory distress syndrome (RDS), pneumonitis) and renal failure. Furthermore, we also included systemic outcomes (cytokine release syndrome (CRS) and multiple organ dysfunction syndrome [MODS]) in this study as they are considered key endpoints in assessing the effectiveness of therapeutics for severe COVID-19. We combined terms for the outcomes of interest with routine health data and validation terms. See the Supplementary Information for the full search strategy.

Screening, Selection Criteria, Data Extraction and Analysis

The studies found in the chosen databases were imported into Endnote. The titles and abstracts were screened by two investigators, any discrepancies were resolved by a third reviewer. We included validation studies reporting positive predictive values (PPVs) published in the last five years (2015–2020) and conducted in human adult populations within North America (USA, Canada) and Europe (France, Germany, Italy, Spain, the Netherlands, Denmark, Finland, Norway, Sweden and the UK). These geographies were chosen to reflect the availability of the majority of population-based healthcare data sources. We restricted the time period as we wanted to capture studies that better reflected the coding practices prior to the introduction of COVID-19 vaccines and therapeutics. However, to be exhaustive, if no studies were identified within the specified five years, the time period was expanded to fifteen years (2005–2020); this was the case for ACI, myocarditis, vasculitis, RDS, pneumonitis, CRS, and MODS. Only articles in the English language were considered given practical limitations. Information on the article characteristics (author, title, journal, year); study population (sample size, selection criteria, study setting), data source, index test algorithm definition, reference standard definition and PPV were extracted from the included articles into an Excel workbook and confirmed by the other investigator. No meta-analysis was conducted due to the heterogeneity of the included studies.

Quality Assessment

Full-text articles were reviewed by the same investigators. Quality was scored using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS) tool.⁶ The QUADAS tool was developed as a quality assessment tool to be used in systematic reviews of diagnostic accuracy studies. QUADAS is an evidence-based tool which consists of 14 items phrased as questions, each scored as “yes” (1 point), “no” (0 points) or “unclear” (0 points). These questions aimed to assess various aspects of the studies such as the selection of the study population, the adequacy of the reference test and index test, and the timing of administration of the tests: We evaluated the representativeness of the patient spectrum and the clarity of the selection criteria; the suitability of the reference standard for accurately classifying the target condition and verification; the replicability, independence and interpretation of the index test; Lastly, the adequacy of the time period between the reference standard and index test. Any disagreements were discussed and resolved by a third reviewer. Low quality defined as a QUADAS score <7 was considered a further reason for exclusion.

Results

Selected Studies

After the removal of duplicates, 5,862 titles and abstracts were screened and 148 were selected for full-text review, of these, 101 studies were excluded. The reasons for exclusion are presented in Figure 1. A total of eight of the 47 studies assessed against the QUADAS tool did not meet the quality criteria and were excluded, resulting in 38 studies extracted across all the events of interest in 2020. The majority of studies were conducted in the US (n=18), followed by Canada (n=7) and the UK (n=4). Only one included study was published prior to 2015.⁷

Figure 1 Study flowchart. Notes: Individual outcomes count sums to more than the total as five studies (Sundbøll et al,8 Ammann et al,9 Arana et al,10 Dalsgaard et al11 and Psaty et al12) provide information for multiple outcomes.

The most studied outcomes were MI (n=13), VT (n=10), stroke (n=9) and arrhythmias (n=5); a summary of the PPVs for these endpoints are detailed in the forthcoming sections. Only one validation study was identified for each of pneumonitis,⁷ myocarditis⁸ and RDS.¹³ When using diagnostic codes in the primary diagnosis field to define these conditions, the accuracy was middling to low at 72%, 64%, and 46%, respectively (see Table 1 for myocarditis and Table 2 for RDS and pneumonitis). To note, the PPV for pneumonitis was reduced when using codes in any diagnosis field. Meanwhile, no validation studies were retrieved for ACI, vasculitis, CRS or MODS.

Table 1 Characteristics of Selected Studies for Cardiovascular Outcomes

Table 2 Characteristics of Selected Studies for Pulmonary Conditions

The quality assessment of the 38 articles is available in Table 3. The majority of studies did not report if the results of the index test and reference test were interpreted without prior knowledge of the results of the other test, thus were categorised as “unclear” (Q10 and Q11).

Table 3 QUADAS Quality Assessment

Cardiovascular Outcomes

Myocardial Infarction

Thirteen validation studies were found for MI (see Table 1 for study details) with the majority (62% [8/13]) conducted in EMR data sources. Validation studies were more frequently conducted in data sources covering the secondary care (hospital) setting; only one study¹⁰ was conducted in primary care. The most common reference standard used was medical record review (n=6 studies),^{8,9,11,18,21,22} though other methods included validation against registry data (n=4),^12,15–17, GP questionnaires (n=1)¹⁰ and troponin measurements (n=1).¹⁹ The most common International Statistical Classification of Diseases and Related Health Problems (ICD) codes reported were ICD-9 codes 410.x and ICD-10 codes I21.x–I.24.x; all studies reported at least one of these codes. Some studies excluded 410.x2 (ICD-9) and I22.x–I24.x (ICD-10) which denote codes for MI in a subsequent episode of care, complications of MI or other ischaemic heart diseases.^9,16,17,19 The exclusion did not seem to have a significant impact on the PPVs.

MI was accurately determined using primary care data in the UK (PPV=98%).¹⁰ In the hospital setting, algorithms that used codes in the primary or secondary diagnosis field^{8,11,12,17–19,22,25} reported consistently higher PPV’s (PPV range 75%–98%) than those that used codes in any field (PPV range 44%–84%).^{9,12,15–17,21}

Arrhythmia

Five selected studies validated algorithms for identifying atrial fibrillation (AF),^{8,14,20,23,24} with one study additionally providing PPVs for bradycardia and ventricular tachycardia of 80% and 87%, respectively⁸ (see Table 1 for details). All studies were conducted in EMR databases and all but one¹⁴ were conducted in secondary care settings. For AF, the PPVs of all studies were above 70%. Generally, algorithms that used more than one instance of a code for AF had the highest PPVs. The highest PPV was recorded in primary care data (PPV=96%) in the Primary Care Practice Based Research Network at Massachusetts General Hospital (USA). AF was assigned with at least one ICD-9/10 code and one problem list entry term, or two ICD-9/10 codes within three years.¹⁴ The highest PPV in secondary care was reported in the Danish National Patient Register (DNPR) using codes in the primary or secondary field (PPV=95%).⁸

Coagulopathy Outcomes

Venous Thromboembolism

Of the 10 selected studies, only one study used healthcare insurance claims data,²⁹ the remainder used EMR data (see Table 4 for study details). All studies (n=10) used review of medical records as the reference standard. All but one study assessed the validity of ICD-9/ICD-10 in a hospital or emergency setting, the exception was conducted in primary care data in the UK using Read v.2. codes.⁴¹ All studies based on ICD codes used ICD-9/ICD-10 (415.x/I26.x) codes to identify pulmonary embolism (PE), and 451.x and 453.x/I80.x and I82.x to identify deep-vein thrombosis (DVT). Studies assessing the validity of VT used a combination of PE and DVT codes. To note, there were minor variations in the specific subcodes used between the studies.

Table 4 Characteristics of Selected Studies for Coagulopathy Outcomes

A total of eight studies provided algorithms for identifying PE.^{8,27–30,33,38,39} Algorithms using codes in the primary or secondary field reported PPV’s >85%^8,30,38,39 while those using codes in any field reported lower PPV values (PPV’s <85%),^27–30 with one exception (PPV=97%).³³ The latter may have been driven by the requirement for a procedure code for additional diagnostic imaging in the emergency department.³³ The lowest PPV was observed using codes in any diagnostic field in an outpatient setting (PPV=28%).³⁰

DVT and VT algorithms had lower PPV’s than PE. The highest PPV for DVT was observed in the Danish National Patient Register (PPV=86%).⁸ The highest PPV for identifying VT in North America was observed in the Cardiovascular Research Network Venous Thromboembolism (CVRN VTE) which used data from four integrated healthcare delivery systems in Canada (PPV=79%).³⁰ Both studies identified codes in a primary diagnosis field. The PPVs increased when using additional procedure codes that provided evidence of treatment, death or evidence of a diagnostic procedure to identify VT (PPV=92%).⁴² Low accuracy for the identification of VT in a primary care was reported, with a PPV of 40%.⁴¹ For both DVT and VT, there were no discernible trends in PPVs between algorithms, which used codes in the primary/secondary diagnosis field versus any diagnosis field.

Cerebrovascular Conditions

Nine studies were selected that validated stroke (see Table 5 for study details). Of these, two studies validated ischaemic stroke^10,43 and the remaining general stroke.^{11,12,31,32,37,45,46} Three studies were conducted in claims databases.^12,43,46 The validity of identifying ischaemic stroke with codes in any diagnosis field had a high PPV in both primary care and secondary care settings, with PPV’s ≥85%.^10,43 The PPVs for general stroke varied, depending on the definition ranging from 44% to 99%.^{11,12,31,32,37,45,46} ICD-9/ICD-10 codes for general stroke included 430.x, and 431.x/I60.x and I61.x (haemorrhagic stroke); 433.x and 434.x/I63.x (ischaemic stroke), and 436.x/I64.x (ill-defined stroke). The lowest PPVs were observed in studies where clinical trial or registry data were used as the reference standard to define general stroke: The PPVs for studies using medical record review ranged from 99% to 70% (median 80%),^11,37,45 while those using clinical trial or registry data as the reference ranged from 80% to 40% (median 72%).^{12,31,32,43,46}

Table 5 Characteristics of Selected Studies for Cerebrovascular Outcomes

Renal Conditions

Five studies validated renal failure (see Table 6 for study details). The reference standards used were creatinine measures (n=2), medical record review (n=2) and GP questionnaire (n=1). The selected codes varied according to the definition of renal failure. An algorithm identifying renal failure using unspecific codes (ICD-9 code 586.x and ICD-10 code N19.0) had the highest PPV of 70%.⁴⁰ The PPV’s for identifying hepatorenal syndrome (a form of kidney impairment that occurs in individuals with severe liver disease) were also high at 79%.³⁴ The PPV’s for identifying renal failure (defined using both AKI and chronic kidney disease codes) in claims data were very low at 13%.³⁶

Table 6 Characteristics of Selected Studies for Renal Conditions

Discussion

We identified 38 studies that validated at least one of our outcomes of interest. Validated algorithms are available to accurately identify cardiovascular and thrombotic events in routine health data such as PE, MI, AF, DVT and stroke where interest transcends multiple therapeutic research areas. The PPVs for renal failure were highly variable and depended on the definitions used. Validation studies for other events of interest, including those more specific to COVID-19 research, were less readily available (myocarditis, pneumonitis, RDS) or not available (ACI, MODS, CRS).

We updated the literature search for ACI, myocarditis, vasculitis, RDS, pneumonitis, CRS, and MODS in July 2023, to understand if any of the research gaps had been filled once the vaccines and therapeutics had been more widely dispersed among the population. We found that since our initial review diagnostic algorithms for myocarditis had been validated in EMR databases in Sweden⁴⁷ in the context of Covid-19 and found to be of acceptable quality (PPV Sweden: 96%). No further validation studies of diagnostic algorithms in EMR were found for the other safety outcomes of interest.

For events with validation studies available, some patterns emerged: algorithms for immediate life-threatening conditions such as MI or PE benefit from using codes within the primary and secondary diagnosis fields of the discharge summary in hospital settings. These fields are used to define the condition of admission during the relevant episode of healthcare. Meanwhile, conditions which often have a chronic presentation, such as AF benefit from using more than one instance of a code. DVT and stroke frequently occur as complications of care for other conditions,⁴⁸ as such, the use of any diagnosis field increases the sensitivity of the algorithm if it is not the primary reason for admission, is a comorbidity or event occurs during the hospitalisation.

Secondary care settings are optimal for identifying the outcomes in this study. Primary care data can be used to accurately identify MI, stroke and atrial fibrillation, though low accuracy was reported when identifying VT among anticoagulant users.⁴¹ VT is increasingly managed in an outpatient setting so may not be included in primary or inpatient secondary care datasets. However, there may be variation by country; hence, knowledge of a countries’ healthcare system and differences in the patient diagnostic and treatment journey are crucial when selecting data sources and defining outcomes of study.

There were fewer validation studies conducted in healthcare insurance claims data compared to EMR data sources. Furthermore, the use of different reference standards (medical records vs registry/clinical trial data) seemed to influence the study results. Recent FDA guidance focuses primarily on medical record review as a reference standard for the validation of outcomes in real-world evidence.⁴⁹ This is in line with the findings of our review, we found studies using clinical trial/registry data had lower PPVs than those using medical record review. The lack of healthcare insurance claims data validation studies may reflect a differential ability to link back to medical records versus EMR data sources. Therefore, to increase outcomes validation information additional strategies may be needed to increase validation opportunities for use with insurance claims data such as increased data linkages to EMR or registry data to allow cross validation.

Additional barriers to the generation of outcomes validation data may link to limitations in diagnostic coding guidance. For example, ICD coding guidelines explicitly recommend that ACI should be coded under the umbrella “ill-defined heart diseases” codes⁵⁰ decreasing the probability of high performing diagnostic PPVs based on these diagnostic codes. Similarly, coding guidelines indicate that MODS should be coded using the specific organ failure codes with no code available to specify “MODS” as a condition. The 2016 International Consensus Definitions for Sepsis and Septic Shock includes “life threatening organ dysfunction” as part of the definition of sepsis.⁵¹ However, caution should be exercised as numerous sepsis validation studies agree that this condition is under recorded in administrative data.⁵² In these cases where coding guidance is more non-specific, routine healthcare data may not be an optimal approach for the generation of benefit risk data.

Strengths and Limitations

The main strength of this study was the comprehensive collection of the most relevant AESIs for COVID-19 research in line with the SPEAC, or key effectiveness endpoints for therapeutics in the context of severe COVID-19.⁵

We acknowledge some limitations to our literature review. There is no standard term for “routine health data” thus it is not well catalogued in MEDLINE and EMBASE (eg no MeSH term). The time frame used may have caused us to miss relevant references. This may have been especially the case for more established data sources, such as the Clinical Research Practice Datalink (CPRD) or the DNPR which have been collecting data for over 30 years.^53,54 Nonetheless, we have identified various validation studies conducted in these data sources within our specified timeframe.^{8,10,11,22,38} We used data to provide a more accurate picture of how the diagnostic codes and algorithms fare in the landscape of an emerging pandemic prior to treatments becoming available to understand the capacity of electronic health records to capture emerging safety signals. Coding practices may have changed over time due to the implementation of new coding classifications such as the change from ICD-9 to ICD-10 in many databases, or due to changes in the delivery and recording of care. An example of the latter is the Quality Outcome Frameworks (QOF), a GP incentive program established in the UK to improve the quality of care. Another limitation is language bias, which may account for the fact that English-speaking countries are overrepresented in the data. Therefore, our findings represent one point in time as well as variations across healthcare systems and geographies for that point in time.

Most studies did not provide patient characteristics, the safety and effectiveness of treatments may be affected by different patient level factors.^55–57 Routine health monitoring is vital for high-risk individuals and as such, the validity of diagnostic algorithms should also be tested in different patient subgroups.

The lack of reporting of disease prevalence of the source population may have influenced the PPVs and hindered the generalisability of the disease algorithms in other populations. Furthermore, specificity and sensitivity are important measures of validation; however, due to the difficulty and resources needed to determine a reference population in routine healthcare data, few studies presented these measures. Also, the use of different gold standards seemed to influence the study results, thus making it difficult to compare validation statistics between diagnostic codes and algorithms using different reference standards. Given the varying PPVs, even for well-studied conditions, assessment is needed of the generalisability of an existing algorithm for a given event in a different data source, the degree of adaptation needed and addressing the influence of misclassification on the results with sensitivity analyses.

Conclusion

Validated diagnostic codes and algorithms are available to identify VT, MI, stroke and atrial fibrillation in routine health data. There is some evidence to support the identification of arrhythmias, pneumonitis, myocarditis and AKI within routine health data albeit with lower accuracy. These primarily reflect secondary healthcare settings. Nonetheless, when designing a study assessing medicinal/biological product benefit risk in the real world a number of factors should be considered when selecting an appropriate set of diagnostic codes given the wide range of PPVs often reported. These include diagnosis code field, minimum number of diagnostic codes required, coding guidance, source of validation data (medical record/registry/clinical trial data) as well as the characteristics of the disease, type of variable (outcome/exposure/covariate) to be identified and the type of data source and healthcare setting.

Emerging scientific areas, requiring development of new diagnostic codes and/or new outcomes of interest, highlight the importance of embedding validation within routine health data studies. Potential barriers to validation work should be better understood and solutions considered as this information is critical to evaluate the robustness of emerging evidence for safety and effectiveness of COVID-19 therapeutics and vaccines using routine health databases. Given increasing emphasis on outcome validation (49), increasing opportunities and methods for validation work is critical. This could be through regulatory frameworks setting clear expectations for validation work as well as facilitating additional data linkage opportunities that may offer an alternative to medical record review.