Sensitivity, specificity, positive and negative predictive values of identifying atrial fibrillation using administrative data: a systematic review and meta-analysis

Introduction

Atrial fibrillation (AF) increases risk of stroke, heart failure and death, and is one of the few cardiac conditions whose prevalence continues to rise.^1,2 Most developed health systems collect reasons for hospital and ambulatory encounters for administration, service planning, quality improvement and reimbursement. Health services researchers use these administrative electronic databases to monitor the burden of disease, quality of care, and ascertain exposures or outcomes. The accuracy of AF identification is central to these applications. Sensitivity and specificity, though theoretically independent, typically trade-off and are inversely related.³ The “optimal” approach to identifying AF depends on the purpose. High sensitivity more completely captures a population, improves generalizability and is important when defining AF as an exposure. By contrast, high specificity ensures persons identified truly have AF and is central to adjudicating treatment uptake, which appears inappropriately low if patients with sinus rhythm are misclassified as having AF.⁴

Conceptually, the AF patient journey involves ambulatory and acute contacts dissociated in time and space, between which information flows by varying amounts and rates. Interrogating data sources over short time intervals or single environments may miss infrequent encounters. A previous systematic review examined the accuracy of AF detection, but was limited to ICD-9 codes only, non-contemporary electronic sources, North American cohorts and narrative synthesis without consideration of the impact of different health care settings (indeed the focus was largely on hospitalization data).⁵ We, therefore, undertook a systematic review to address these evidence gaps.

Methods

Participants, outcomes and study designs

Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines were followed (Table S1). We examined the accuracy of AF case definitions in electronic administrative health data, namely sensitivity (SN), specificity (SP), positive and negative predictive values (PPV and NPV). Inpatient, outpatient and mixed populations were included. All study designs were accepted. The study protocol was not published.

Search strategy and data collection

MEDLINE was searched from January 2000 to February 2018, limited to adult humans and English language, excluding case studies, reviews and conference abstracts. Search terms were determined by literature review and database query. The search strategy combined Medical Subject Headings (MeSH) terms and keywords in title and abstract to define three groups: atrial fibrillation (including atrial flutter (AFL) if not differentiated); administrative and electronic medical databases; and studies examining accuracy of AF identification within these records (Table S2). The search returned 1007 unique records. Manual bibliography searches identified an additional 31 publications (Figure S1). Titles and abstracts were screened for inclusion, and 302 full-text articles reviewed. Studies fulfilling the participant, outcomes and study design criteria were included. Variables of interest were decided a priori, expanded iteratively after pilot and collected in Microsoft Excel. The following information was extracted: bibliographic details, sample size, population characteristics, inclusion and exclusion criteria, codes and algorithms, AF confirmation gold standard and accuracy parameter outcomes.

Data synthesis

Weighted averages of sensitivity, specificity, positive and negative predictive values were calculated using the DerSimonian-Laird random effects model.⁶ Forest plots of estimates with 95% confidence intervals (CI) were generated. Publication bias was assessed through visual inspection of funnel plots and the Begg-Mazumdar rank correlation test for asymmetry.^7,8 Heterogeneity was tested with visual forest plot inspection, Cochrane Q, I² and Tau² statistics.⁹ Estimates with significant heterogeneity (I²>90%) were examined manually and formally for moderating effects including country, publication year and reference standard, none of which were significant. The leave-one-out method was used to determine if the results were sensitive to the inclusion of extreme values from specific studies.¹⁰

Results

Study characteristics

Twenty-four studies were identified (Table 1). Most originated from countries with established administrative databases that are often interrogated by health services researchers, including 10 from the United States, 3 from Canada and 8 from Sweden or Denmark. The populations were heterogeneous, including general unselected, stroke and post-operative cohorts. Hospital, ambulatory and mixed data sources were assessed in 10, 4 and 10 studies, respectively. Only 3 studies outside Scandinavia examined mixed populations.^4,11,12 One Canadian study included administrative data from emergency departments separate from hospitalizations.¹¹

Table 1 Study characteristics

Coding and case definition algorithms

Most reports investigated International Classification of Diseases codes: ICD8 (427.93 AF, 427.94 AFL), ICD9 (4 digit code 427.3 and more explicit 427.31 AF and 427.32 AFL), ICD10 (I48). Overall, 9 different combinations of codes were studied (Table 2). The impact of coding position (primary versus secondary diagnosis in hospitalization data) was never examined. Four studies compared the accuracy of two versus one encounter coded as AF in ambulatory data sources within a single year.^4,11–13 This consistently increased specificity but decreased sensitivity. A single study compared 2 versus 1 year for case ascertainment in Veterans Affairs outpatient records, finding greater sensitivity with only slightly reduced specificity.¹³ Overall in that study, 2 diagnoses over 2 years were optimal for detecting AF.¹³ Only one study from Canada examined more complex algorithms including cardioversion codes and pharmacy dispensations for antiarrhythmic drugs.¹¹

Table 2 Characteristic and validation of atrial fibrillation in studies

Characteristics of AF

Prevalent and incident AF were assessed in two-thirds and one-thirds of studies, respectively (Table 1). Incident cases were typically defined by exclusion of prior AF diagnoses since the records began, or methods were not specified. The incidence and prevalence of AF varied markedly depending on the population studied, from 0.3% to 55%.^14,15 The incidence and prevalence was highest in studies following cardiac surgery (32–36%) and stroke (10–28%), lower in general hospitalizations (7–9%) and lowest in unselected outpatients (0.3–1%). No study distinguished between persistent and paroxysmal AF. Three studies reported from 7.0% to 20.4% of the coded AF to be transient.^4,16,17 Two defined transient as a single episode without recurrence,^4,16 while two added precipitants including cardiac surgery and/or hyperthyroidism.^16,17

Gold standard for diagnosis of AF

With the exception of two studies, medical chart review was considered the gold standard by which history of AF was classified (Table 2). Of these, 13 studies specifically included ECG review, of which 2 employed ECG alone for confirmation of AF.^18,19 No prospective protocols or frequencies for ECG were reported. A median of 11 ECGs per patient with AF was noted in a Swedish outpatient setting.¹⁵ Only 4 studies mentioned use of longer term rhythm monitoring such as Holter, although these results may also have been available in medical record review.

Sensitivity and specificity

Half the studies (n=12) sampled an unrestricted population including those without AF to assess sensitivity of case definitions and these ranged from 57% to 93%, median 81% (Table 3). The pooled random effects estimate for sensitivity was 80% (95% CI 72–86%) with significant heterogeneity (Q 439, I² 97.7%, Tau² 0.08). One-third of the studies (n=8) reported specificity. Estimates were consistently high in ambulatory, hospitalized and mixed populations, ranging from 91% to 99%, median 99% (Table 3). The pooled random effects estimate for specificity was 98% (95% CI 96–99%).

Table 3 Codes and accuracy for identifying atrial fibrillation

Positive and negative predictive value

Positive predictive value was reported in nearly all studies (n=22), and was the only parameter reported in half the studies (n=12). The PPV ranged from 71% to 99%, median 93% (Table 3). The pooled random effects estimate was 88% (95% CI 82–94%) with significant heterogeneity (Q 4997, I² 99.6%, Tau² 0.02) (Figure 1). The pooled estimate was similar in ambulatory, hospitalized and mixed populations (respectively: 87% (78–96%); 87% (79–95%); 90% (85–94%)). Negative predictive value was reported in 8 studies. Estimates were consistently high, ranging from 86% to 99%, median 98% (Table 3). The pooled random effects estimate was 97% (95% CI 94–99%).

Figure 1 Positive predictive values of atrial fibrillation (AF) algorithms stratified by population type.

Discussion

This analysis reports several key findings. The overall specificity and NPV of an AF diagnosis using the ICD case definitions was high, 98% and 97%, respectively. The sensitivity and PPV were lower though reasonable, 80% and 88%, respectively. Only half the studies sampled patients with and without an assigned diagnosis of AF to determine the sensitivity of the case definitions and thus the proportion potentially missed by using administrative data. Half the studies confirmed AF using electrocardiography as the gold standard, while the remainder employed medical record review, alternative databases (like primary care EMRs) and/or patient questionnaires. Only 3 studies reported all accuracy parameters and included rhythm monitoring in the gold standard definition.^15,17,20

Sensitivity

High sensitivity improves case finding as it more completely captures a population, increases the estimated incidence and prevalence and enhances generalizability. This is particularly relevant when estimating the burden of disease and to reduce bias when studying health inequalities. Sensitivity is also important when defining AF as an exposure. Misclassification of exposure (eg, AF) as non-exposure (eg, no AF) attenuates the association with outcomes such as stroke.²¹ By contrast, sensitivity is less of a concern when defining AF as an outcome, for example in pharmacovigilance studies. In these circumstances, estimates of relative risk are not biased providing misclassification occurs to the same degree in exposed and non-exposed patients.

Sensitivity is reduced when cases are missed and AF is misclassified as normal (ie, false negatives). This occurs in two circumstances. First, when recording or coding is incorrect. Second, when correctly recorded and coded diagnoses are missed in time or space. Examining shorter time frames may miss infrequent encounters, as evidenced in the Veterans Affairs study where sensitivity increased using a 2 versus 1 year period for case ascertainment.¹³ Information also flows by varying amounts and rates through health systems. Although the median time from AF on ECG to diagnosis in the Swedish Patient Register was 16 days, this time lapse exceeded 6 months in one-third of patients.¹⁵

Sensitivity may be viewed from different perspectives: local, horizontal level of care (eg, primary care), vertical (eg, health maintenance organization) or global (entire health care system). Examining a single health care setting may miss encounters meeting the AF case definition in another. For example, hospitalization data alone misses patients managed entirely in the community, causing under-estimates of prevalence rates and over-estimates of adverse outcome rates. There were insufficient studies to accurately compare sensitivity between ambulatory, hospital and mixed populations. However, one of the mixed population studies did compare the accuracy of coding between primary care, secondary care or both together. In that study from Ontario, the sensitivity was 45%, 39% and 75% for hospitalization, emergency department or outpatient data sources alone, respectively, and 83% combining the three sources.¹¹

The true population incidence and prevalence may also be influenced by access to rhythm monitoring (ECG, Holter, event or loop recorder), reporting standards (eg, training, quality assurance) and information transfer (eg, interface to electronic medical record). These factors are potentially more challenging in community than in hospital settings, particularly relevant to measuring inequalities, and difficult to quantify. None of the included studies described these aspects of access.

Positive predictive value

Since sensitivity and specificity are typically inversely related, higher sensitivity reduces specificity, which increases false positives and lowers PPV. The impact on PPV is magnified for diseases with a relatively low prevalence such as AF. A high PPV ensures persons identified truly have AF (fewer false positives). This is central to adjudicating treatment uptake, which will appear inappropriately withheld if patients with sinus rhythm are misclassified as having AF, unless OAC is prescribed for an alternate reason.⁴

A PPV value exceeding 85–90% suggested adequate for research purposes.^19,20 The reasons for false positives were rarely explored.²² Potential scenarios include: 1) miscoding eg, allergic rhinitis was written as “AR” and coded as AF;²² 2) rhythm misinterpretation such as atrial tachycardia; 3) misreporting if based on medical history alone; and 4) AF defined by an intervention shared with other conditions eg, cardioversion. PPV is also highly dependent on disease prevalence: as many studies focused on older or high-risk individuals they may overestimate the true PPV for that case definition if applied in a younger population.

Oral anticoagulation is the only treatment to improve survival in patients with AF, and thus a key quality indicator. Although the overall PPV was high (88%), the specificity and PPV to identify AF requiring anticoagulation (as opposed to any AF) could be lower for several reasons. First, up to 10% of incident AF is isolated with a defined precipitant, low recurrence, and may not require anticoagulation.^4,16,23 Only three studies reported or excluded such patients.^4,16,17 Second, anticoagulation adjudication requires accurate coding of embolic and bleeding risk factors, which like AF exhibit high specificity but are under-reported.^12,22 More subjective bleeding risks such as frailty and falls are particularly difficult to quantify, although a recently described frailty score based on administrative data (the Hospital Frailty Risk Score) has been described.²⁴ Finally, patient preferences are major drivers of anticoagulation decisions but are never captured in administrative databases.

Atrial fibrillation phenotype and coding considerations

The disease spectrum (permanent, persistent, paroxysmal, isolated unprovoked or provoked episodes) was rarely reported yet also impacts accuracy of AF detection. Permanent or persistent AF is associated with greater comorbidity and hence health care encounters during which arrhythmia is continuously present. By contrast, isolated or paroxysmal AF may be under-represented by health care encounters. Treatment including rate versus rhythm control and anticoagulation also varies based on symptoms, AF duration, risk of recurrence and thromboembolism.²⁵ The accuracy of administrative data to identify AF requiring anticoagulation is thus further lessened by limited phenotypic characterization.⁴ The AF phenotype may also impact the “gold standard” for diagnosing AF, whereby paroxysmal AF is missed by ECG alone but detected by chart review. In the only study examining this issue, ECG review did not improve sensitivity of AF detection over diagnosis codes alone.⁴

Most developed health systems collect reasons for hospital and ambulatory encounters for administration, service planning, quality improvement and reimbursement. A single primary or most responsible diagnosis is typically assigned, while conditions complicating or prolonging stay are coded in multiple secondary positions, sometimes further categorized as pre-existing or de novo disease. Differences in coding accuracy, treatment and prognosis are reported between primary and secondary positions for conditions such as heart failure.^26,27 To our knowledge, such differences have not been explored in patients with AF, and no study identified by our search compared coding positions.

Strengths and limitations

Several strengths and limitations merit consideration. Our analysis is contemporary, included varied health systems, ICD-8 to ICD-10 codes, and both ambulatory and hospital populations. However, most studies originated from North America or Scandinavia, and examined ICD codes in administrative data sources. This potentially limits generalizability to other health care systems. There was significant heterogeneity in terms of population, prevalence of AF and reported accuracy parameters. Most studies assessed accuracy in restricted cohorts as opposed to the broader population.

Directions for future research

Health service researchers and administrators may interpret administrative data using either our pooled estimates or locally relevant studies from among those identified. Jurisdictions would ideally conduct nationally representative validation studies to provide estimates specific to their populations and data sources. These should examine existing codes and test new case definition algorithms in all data sources with differences in coding practices and diagnostic accuracy (eg, hospitalization, emergency department, ambulatory primary and secondary care), and in scenarios with varying disease prevalence. Though challenging and costly, random sampling of representative populations is essential to define sensitivity, enhance generalizability and reduce bias when studying inequality. To understand the true disease burden, algorithms should combine primary and secondary care data sources.

More complex algorithms utilizing advanced analytics such as natural language processing and machine learning to mine free-text medical records merit investigation. Potential avenues include integrating corroboratory data such as medications and procedures, and temporal and spatial coding patterns. Future work should investigate the optimal gold standard including rhythm monitoring, electronic data sources and chart review. The reasons for false positives and negatives need to be explored in detail, as does the impact of AF phenotype and coding position. Finally, the accuracy of embolic and bleeding risk factor case definitions requires further validation in order to adjudicate appropriateness of anticoagulation management choices.

Conclusion

The overall accuracy of AF identification was reasonable for system planning and surveillance of prevalence, quality and outcomes. However, there is a marked disconnect between the volume of publications in these domains, and those examining the underpinning data. Sensitivity and PPV were the least accurate parameters with greatest uncertainty in terms of evidence and interpretation. This potentially underestimates the burden of disease and may bias estimates of outcomes and treatment quality. The optimal AF case definition should consider the purpose of the study and the data sources available. Health service administrators, researchers and clinicians should be mindful of these factors, and work together to refine our use of electronic data.