An analysis of the degree of concordance among international guidelines regarding alpha-1 antitrypsin deficiency

Introduction

Practice guidelines (PGs) represent an important strategy to harmonize and standardize clinical practice to optimize care. In this context, PGs have proliferated globally, often developed by disease experts and sometimes affected patients who are empaneled by professional societies.

In the specific case of alpha-1 antitrypsin deficiency (AATD), an uncommon autosomal codominant condition that predisposes to emphysema and cirrhosis, the first PG was issued in 1989 by the American Thoracic Society (ATS) with a subsequent 2003 PG co-sponsored by the European Respiratory Society (ERS), the American College of Chest Physicians, and the American Association for Respiratory Care.^1,2 The Canadian Thoracic Society (CTS) published its first iteration of an AATD guideline in 2001, later updated in 2012.^3,4 Since these earliest guidelines, multiple others have been published by a variety of international medical societies (Table 1).^5–14 More recently, the Alpha-1 Foundation empaneled a group of experts to prepare an updated PG in 2016.¹⁵ Notably, because AATD is uncommon and because clinicians, including pulmonologists, characteristically see few such patients in a career, reliance on published guidelines is commonplace.

Table 1 Alpha-1 antitrypsin deficiency (AATD) guidelines analyzed in this study

Naturally, the proliferation of guidelines regarding the same clinical condition begs the question: do all guidelines agree? In this context, the current study was undertaken to assess the degree of agreement among the available guidelines regarding the management of individuals with AATD.

Methods

To identify candidate guidelines, Medline and Embase were searched from 1974 to January 24, 2018, using the search terms “alpha-1 antitrypsin deficiency,” “COPD,” and “guidelines.” References in sourced documents were reviewed. Eligible guidelines were published in English and were issued by official respiratory organizations/medical societies and/or by national organizations. Guidelines issued by insurance organizations were excluded. Because initial review of more generic COPD guidelines indicated that few offered specific recommendations regarding AATD, such guidelines were not included. Furthermore, because the earliest report from the Canadian Thoracic Society (in 1992) was confined to a statement regarding augmentation therapy and two subsequent, more comprehensive guidelines were issued by the Canadian Thoracic Society (in 2001 and 2012), only the latter two guidelines were included.^3,4,16

To analyze the degree of concordance among the 15 eligible guidelines (Table 1), 24 statements regarding key management issues were formulated by two pulmonologists (JKS, AA) with deep experience with AATD (Table 2). The statements were developed after we compiled every recommendation by an AATD guideline and divided each recommendation into categories. Using the compilation of guideline questions as a model, we summarized these overall themes by the development of our summary statements.

Table 2 Individual clinical statements and adjudicated guideline ratings

To allow specific ratings for each guideline on each of the 24 statements, each statement endorsed a specific clinical action in managing a patient with AATD (eg, “get CT scan at baseline”). The 24 statements were bundled into 5 categories: When to initiate a diagnostic workup (“When to test”), how to test for AATD (“How to test”), how to manage individuals with AATD apart from augmentation therapy (“How to manage”), when to initiate augmentation therapy (“When to treat”), and how to treat with augmentation therapy (“How to treat”).

Two independent reviewers (AA and UM) read the 15 guidelines and initially coded each guideline on each of the 24 statements, assigning one of four ratings to summarize the guideline’s treatment of that statement (Table 3). The four possible ratings included: Yes (ie, the practice was endorsed); Yes, conditional (the practice was endorsed subject to qualifying conditions); Equivocal/no comment (the guideline was ambiguous or no comment was made); No (the practice was not endorsed).

Table 3 Four rating categories applied to each of the 24 clinical statements for each guideline

After independent ratings by the two reviewers for each statement on each guideline, all ratings were compared. In the case of discordant ratings, a third reviewer (JKS) helped to adjudicate and consensus was achieved for all ratings. Thus, a complete set of adjudicated ratings of the 24 statements was analyzed.

Notably, several guidelines cross-referenced each other. For example, some guidelines mimicked statements in the ATS, ERS, or CTS guideline. In tabulating agreement among PGs, the statements in each individual text were attributed to that specific society guideline even when derived or quoted from another guideline; for example, the recommendations of the Belgian Thoracic Society were ascribed to that guideline despite mirroring the ATS guideline.^2,8

Some texts were noted to have “internal discordance,” which was defined as conflicting recommendations within the individual guideline, usually found between the text and a table/graph (Table 4). In this situation, recommendations from the text were favored.

Table 4 Parameters for quantifying guideline agreement

Whenever diametric discordance occurred (ie, one guideline rating was “No” and another guideline was rated “Yes”) on a statement, the three reviewers convened again to confirm consensus. For example, in contrast to other guidelines, the CTS guideline recommends targeted testing for AATD only when suggestive features are present (eg, emphysema of early-onset or basilar hyperlucency), whereas most others endorse testing all patients with COPD, whether or not suggestive features are present. In this instance, all three authors reviewed all guidelines again on this issue to assure accuracy in their ratings.

After final adjudicated ratings for each guideline were obtained, the number of guidelines receiving each particular rating (Y, YC, Eq/NC, N) was calculated for each clinical statement (Table 2). In addition to this guideline rating distribution – called “individual statement agreement” – we also aimed to quantify the degree of overall guideline concordance, which we call the “overall guideline agreement percentage.” Guideline concordance requires two guidelines to share the same rating (eg, “Yes,” “Yes, conditional,” etc.) for a specific clinical statement. Therefore, calculating the “overall guideline agreement percentage” involves comparing each guideline to every other guideline on each of the 24 clinical statements. In total, with 15 guidelines, there are 105 guideline comparisons for each clinical statement (ie, each guideline with each other one) and an overall total of 2520 comparisons (105 guideline comparisons for each statement × 24 clinical statements). While the number of guideline comparisons is rather straightforward, concordance also requires that the type of ratings for each statement (eg, Yes; Yes, conditional; Equivocal/No Comment; No) on each guideline match. The number of matching guideline comparisons is a direct result of the number of guidelines receiving a particular rating (Table 5, see legend for further explanation).

Table 5 Number of matching guideline comparisons (“overall guideline agreement percentage”)

Importantly, a guideline comparison match is possible when neither guideline addresses the specific clinical issue posed by one of the 24 statements, ie, both compared guidelines were rated “Equivocal/No Comment” on the statement. To distinguish topics on which PGs explicitly agreed in endorsing an action or agreed in disapproving an action, parameters called “affirmative agreement percentage” and “negative agreement percentage” were developed (Table 4).

Finally, in the context that the available guidelines have spanned several decades and that the emergence of new knowledge over time can affect recommendations in guidelines and could be the source of discordance, a “time-clustered” analysis of guideline concordance was conducted. Specifically, guidelines were stratified into three clusters by year of publication: 2013–2018, 2008–2012, and 2007 and earlier. The same analysis of concordance among guidelines as was conducted for the total group was conducted within each time cluster. If observed discordance is attributable to the emergence of new knowledge about AATD, then the degree of concordance between guidelines should be lower within time clusters than between guidelines published at largely separated time intervals.

Statistical analysis was performed using R software version 3.5.1 (R Foundation for Statistical Computing, Vienna, Austria).

Results

As shown in Figure 1 (which depicts the guideline selection using the PRISMA method and format), the search strategy produced 936 unique records, of which 805 were eliminated based on the review of title and abstract. The 131 remaining articles were reviewed in full text; 116 were excluded, leaving 15 eligible guidelines that comprise the study sample (Table 1).¹⁷ Eligible guidelines emanated from 11 countries in North America, Europe, and South America, with authors from 53 countries. Several guidelines represented multiple iterations from the same medical society (eg, the ATS and ERS both issued two guidelines, one jointly; and the CTS issued two guidelines, one in 2001 and the later one in 2012).^2–4

Figure 1 PRISMA flow diagram. Medline and Embase were searched using search terms “alpha-1 antitrypsin deficiency,” “COPD,” and “guidelines.” Eligible guidelines were published in English and were issued by official respiratory organizations/medical societies and/or by national organizations.

The distribution of adjudicated guideline ratings for each clinical statement shows substantial variability (Table 2). There is not a single clinical statement where all 15 guidelines receive the same rating. Also, notable is the large number of guidelines rated “Eq/NC” for many clinical statements.

The overall level of agreement among the 15 guidelines on the 24 clinical statements was 47% (1190/2520 comparisons [Table 6]). Affirmative agreement accounted for 42% of the matching comparisons (501/1190 comparisons), whereas 48% of the matching guideline comparisons were rated as “Equivocal/No Comment” on a specific statement (568/1190 comparisons). Higher agreement surrounded concepts of “When to treat” and “How to treat”; however, even these only agreed approximately half the time.

Table 6 Overall guideline agreement percentage and by rating bundle (ie, when to test, etc.)

In considering individual clinical statements (Figure 2), the highest affirmative agreement percentage (guidelines agreeing that the following should be performed) was for the statements: “Initial testing should include a serum AAT level.” (74%), “I should encourage smoking cessation” (63%), and “I should only treat when the serum AAT level is below a protective threshold value and/or when there is a severe deficient genotype/phenotype” (63%). Conversely, the highest negative agreement percentage (guidelines agreeing that the following should not be performed) was for the statements: “Only patients with suggestive features of AAT deficiency should be tested” (43%), “Patients with airflow obstruction that is fully reversible with bronchodilators should be tested (ie, ‘pure’ asthma)” (34%), and “Testing should always include both a level and genotype/phenotype” (34%).

Figure 2 Affirmative agreement among guidelines on individual clinical statements. For each clinical statement, the affirmative proportions of guideline comparisons endorsing an action (with or without an added condition, Y + YC) are plotted. Negative agreements are also plotted. The highest affirmative agreement percentage was for the statement: “Initial testing should include a serum AAT level” (74%). The highest negative agreement percentage was for the statement: “Only patients with suggestive features of AAT deficiency should be tested” (43%).Abbreviations: Y, yes; YC, yes, conditional; AAT, alpha-1 antitrypsin.

Many guidelines were concordant in not addressing some of the 24 statements. For example, 74% of the guideline comparisons were rated “Eq/NC” regarding, “I should treat patients with panniculitis as first-line therapy.” Other statements rarely receiving definitive recommendations were “I should refer my patient to a pulmonologist and/or tertiary center” (63%), “I should get a CT chest at baseline” (52%) and “I should get a liver ultrasound on my patients at baseline” (52%).

Finally, to address the question of whether discordance between guidelines reflects changing knowledge over time, Table 7 presents the results of a time-clustered analysis. Specifically, the degree of discordance between guidelines published in 5-year windows (2013–2017, 2008–2012, 2007 and earlier) was similar (ie, for guidelines published between 2013 and 2017 [N=6 guidelines], overall discordance =51%; 2008–2012 [N=3], overall discordance =35%; 2007 and before [N=6 guidelines], overall discordance =49%), thereby suggesting that the passage of time and acquisition of new knowledge about AATD over time did not account for the observed discordance between guidelines.

Table 7 Agreement percentage among guidelines stratified by time periods: 2013–2017, 2008–2012, and 2007 and earlier

Discussion

The main finding in this comparison of 15 available AATD guidelines from multiple medical societies in multiple countries is that there is substantial variation in recommendations regarding how to manage routine clinical issues in AATD. Sources of disagreement among the available guidelines include the degree to which the various guidelines addressed the 24 statements that were posed (ie, many were rated equivocal/no comment) as well as disagreement regarding specific managerial decisions that were addressed in the guidelines. The time cluster analysis of concordance among guidelines published within 5-year intervals shows that the degree of discordance within the time clusters was also high, suggesting that the emergence of new knowledge over time did not account for the observed discordance between guidelines.

While our study could not examine specific reasons for discordance among the guidelines (which would require querying the guideline authors), several explanations are possible. First, disease prevalence and manifestations vary across countries, likely causing recommendations to differ.¹⁸ For example, the recommendation to test all patients with fixed airflow obstruction for AATD in a country where the prevalence of severe deficiency of AATD is high (eg, countries with substantial populations of North European descent) may be inappropriate in countries where the prevalence is lower (eg, countries in Asia or Africa).¹⁹ As evidence of this influence of disease prevalence on recommendations, some guidelines made diagnostic recommendations that are conditional on disease prevalence.² For example, the 2003 ATS/ERS guideline recommends testing all symptomatic adults with fixed airflow obstruction in populations where the prevalence of AATD resembles that in North America and Northern Europe (level A recommendation) but lowers the strength of the recommendation (to level B) in populations where the prevalence is lower.²

A second possible reason for discordance among guidelines from different countries regards the availability of specific therapy in the source country and how health care is funded there. For example, in countries where augmentation therapy for AATD has not yet been approved by regulatory bodies (eg, the United Kingdom), guidelines would not be expected to recommend augmentation therapy, whereas treatment with augmentation therapy is endorsed in some guidelines from countries where augmentation therapy has been approved (eg, the United States and some European countries).²⁰ Similarly, in countries with universal health care coverage, recommendations regarding the allocation of costly therapy (like augmentation therapy) or expensive testing strategies may be especially conditioned by the perceived incremental cost-effectiveness of the recommendation.²¹

Of course, a third potential reason for discordance is that physicians and regulatory bodies can interpret available data about the efficacy of treating AATD differently. For example, the United States Food and Drug Administration (FDA) and the European Medicines Agency (EMA) have interpreted the results of a recent randomized trial of augmentation therapy (eg, the RAPID trial) differently, resulting in different approvals.²² Specifically, the drug Respreeza (CSL Behring, King of Prussia, PA) was approved with text stating that the drug slows lung destruction on CT by the EMA but not by the FDA.^23,24

Finally, as available guidelines span the interval from 1989 to 2016, the evolution of knowledge over time could account for varying recommendations.^2,15 However, the time-clustered analysis which showed that the degree of discordance among guidelines published within 5-year clusters was similar to the overall rate of discordance suggests that the emergence of new knowledge was not a primary driver of discordance among available guidelines.

The current study extends available knowledge in several ways. First, to our knowledge, this is the only study to compare recommendations among available guidelines regarding AATD. Second, this is, to our knowledge, one of only a few studies to assess concordance among available guidelines for any specific disease.^25–28 In 2001, LaCasse et al compared the quality and recommendations from 15 extant guidelines regarding the management of COPD.²⁸ Though lacking a systematic comparison of guideline concordance on each management element, the earlier analysis reported substantial discordance among the available guidelines. Details regarding AATD management recommendations were not available in the report. Mortensen and Nordestgaard examined variation among 5 medical societies’ guidelines in recommending statins.²⁵ As with our findings, frequent discordance was observed, eg, the prevalence of eligible candidates for statin use among an index Danish population using each of the 5 guidelines varied more than two-fold, ie, from 15% to 42%. Similarly, the other two analyses – which addressed relatively common conditions like managing urinary incontinence and imaging for dental and maxillofacial indications – also demonstrated broad heterogeneity of recommendations.^26,27 Beyond the general degree of discordance, guideline generation for uncommon diseases like AATD is especially challenged by the relative paucity of large, definitive studies upon which to make management recommendations.

Several shortcomings of the current study warrant discussion. First, although a Delphi process to adjudicate discordant ratings from individual reviewers was used to derive the final adjudicated ratings, there was, not surprisingly, some discordance among the reviewers (AA and UM) in their initial reviews. Such initial discordance raises the possibility that the 24 statements used to assess the AATD guidelines (Table 2) were insufficiently clearly formulated. At the same time, our practice of involving three independent raters and serially reviewing the guidelines to assure rating accuracy (eg, when there was diametric discordance) affirms the possibility that some discordance resulted from ambiguity in some of the specific guideline recommendations. As a specific example of ambiguity, the Belgian Thoracic Society guideline states:

Augmentation therapy should be considered in non-or ex-smokers with an AAT serum level under 50 mg/dL and with moderate to severe obstruction (FEV1 between 30–65% predicted) or with a rapid decline in FEV1.

This specific text could easily be variously interpreted as allowing augmentation therapy for specific FEV1 parameters or for unspecified changes in FEV1 over time, leading to potential ambiguity in the recommendation, especially for patients whose FEV1 initially exceeds 65% but appears to be declining.

Another potential shortcoming of the analysis is that ascribing recommendations to a guideline that mimics another guideline could cause overestimation of the degree of concordance. For example, if the Belgian Thoracic Society guideline directly quoted the ATS/ERS guideline, the two guidelines would be deemed concordant.² Notably, because such direct quotation of one guideline from another was infrequent in this study, we submit that this potential bias toward overestimating concordance is minimal. Given that our study showssubstantial discordance idespite a bias that would inflate concordance, this emphasizes the variability among guidelines and the resultant confusion that both clinicians and patients may experience in trying to offer and receive optimal care for AATD.

Conclusion

To our knowledge, this is the first study to analyze the degree of concordance among various international guidelines regarding AATD, and one of very few comparisons of guidelines for any disease. The analysis recognizes population and treatment milieu differences as potential sources of guideline variation across diverse populations. Beyond the specific implications for AATD, which might include attempts to harmonize the various guidelines by empaneling a broadly representative international group of disease experts, our findings also encourage similar analyses of guideline concordance for other diseases.