Psychometric Validation of the EQ-5D-3L in Patients with Nontuberculous Mycobacterial (NTM) Lung Disease Caused by Mycobacterium avium Complex (MAC)

Introduction

Nontuberculous mycobacterial (NTM) lung disease (NTM-LD) is a rare but emerging global health concern.^1,2 Although more than 200 species and subspecies of NTM have been identified, the most predominant species causing NTM-LD is Mycobacterium avium complex (MAC).^3,4 MAC lung disease (MAC-LD) can become a chronically progressive and debilitating disease. It can lead to severe lung injury and cavitary lesions and be associated with poor survival prognosis.^5,6 Patients with MAC-LD experience diminished pulmonary function,⁷ which could lead to a reduction in patients’ health-related quality of life (HRQoL), self-perceived health status, and functional abilities.^8,9 Additionally, treatment of MAC-LD necessitates the use of a combination of antibiotics, often given over 18 months or more; thus the risk of treatment-related toxicities is relatively high, which can further worsen patients’ HRQoL.^5,10

To accurately assess a disease’s impact on patients’ HRQoL, it is imperative to select a valid and reliable HRQoL instrument. In the absence of any MAC-LD-specific HRQoL instrument, previous studies have used generic measures, such as EuroQol 5-dimensional 3-level (EQ-5D-3L), to assess the HRQoL of MAC-LD patients.^8,9,11 However, limited information is available regarding the psychometric properties of EQ-5D-3L in MAC-LD. In order to build on this gap in evidence, the current study aimed to investigate the reliability, validity, and responsiveness of the EQ-5D-3L among patients with MAC-LD who were refractory to guideline-based treatment (GBT).

Methods

Study Design

This analysis includes data from CONVERT (INS-212; NCT02344004), a Phase III, randomized, open-label clinical trial that studied the effectiveness of adding amikacin liposome inhalation suspension (ALIS) to GBT in subjects with MAC-LD who were refractory to GBT.¹² This post hoc analysis uses the HRQoL and relevant clinical data collected from baseline to month 6 to evaluate the psychometric properties of the EQ-5D-3L among patients enrolled in the CONVERT study.

The CONVERT study enrolled adults with active MAC-LD, according to the American Thoracic Society and the Infectious Diseases Society of America criteria. Patients were required to be MAC-positive while on GBT ≥6 months and to have been on GBT within 12 months before screening.¹² Patients were randomized 2:1 to receive ALIS and GBT or GBT alone. Details of the CONVERT trial and the results have been published elsewhere.¹² All patients in the intent-to-treat (ITT) population who had a baseline assessment of the EQ-5D-3L in the trial were included in this analysis.

Study Measures

Clinical Measures

Three clinical measures recorded in the trial were used in this analysis: sputum culture conversion, six-minute walk test (6MWT), and percent predicted forced expiratory volume in 1 second (ppFEV₁). The proportion of patients achieving sputum culture conversion within the first 6 months was the primary endpoint assessed in the CONVERT trial, and was based on patient sputum cultures collected monthly from baseline through month 6.¹² Culture conversion was achieved if patients had 3 consecutive negative sputum cultures. The 6MWT provided a measure of the patient’s exertional capacity by requiring the patient to walk a prescribed course on a flat surface while recording the distance covered in 6 minutes in meters, with higher scores representing better capacity.¹³ The 6MWT was administered at baseline, month 4, and month 6. The FEV₁ is a lung function test that measures the “maximal volume of air exhaled in the first second of a forced expiration from a position of full inspiration.”¹⁴ The ppFEV₁ (predicted for age, race, gender, and height) was generated using the prediction equations from Hankinson et al for baseline and month 6.¹⁵

Health-Related Quality of Life (HRQoL) Measures

The 2 HRQoL measures used in this analysis were the EQ-5D-3L and the St. George Respiratory Questionnaire (SGRQ). The EQ-5D-3L is frequently used for both clinical and economic appraisals and allows for comparisons between different disease states.¹⁶ It is a self-administered questionnaire that includes 2 components: a descriptive profile and a visual analogue scale (VAS).¹⁷ The descriptive profile measures 5 single-item health dimensions: mobility, self-care, usual activities, pain/discomfort, and anxiety/depression. Each dimension has 3 levels of response: no problems, some problems, and extreme problems. The responses to each item were used to generate a population-normed index summary score, anchored at 0 for death and 1 for perfect health, based on the published United Kingdom time trade-off value set.¹⁸ The VAS assesses a respondent’s self-perceived current general health status on a 100-point scale, with 0 representing “worst imaginable health state” and 100 representing “best imaginable health state.” The EQ-5D-3L was administered at baseline, month 3, and month 6. The SGRQ is a self-administered 50-item questionnaire evaluating 3 health domains: symptoms, activity, and impact.¹⁹ Each domain is scored from 0 to 100, with a lower score indicating better HRQoL, and the domain scores are combined into a weighted total score ranging from 0 to 100. The SGRQ was administered at baseline, month 3, and month 6.

Statistical Analysis

Item Distribution, Floor and Ceiling Effects

Descriptive statistics (N, mean, standard deviation [SD], median, 25^th and 75^th percentile, and range) were reported for the EQ-5D-3L index and VAS scores at baseline, month 3, and month 6. The distribution of responses and missing data for each item was reported at baseline in terms of frequencies and percentages. Additionally, the presence of floor and ceiling effects was examined at baseline for each item. An item was considered to demonstrate a floor or ceiling effect if more than 33% (100/the number of response options on the item, ie =100/3) of responses were in the lowest or highest response categories.²⁰ The proportion of patients reporting the lowest and maximum possible values on the EQ-5D-3L index and VAS scores at baseline were also reported.

Test–Retest Reliability

Test–retest reliability is the ability of an instrument to reliably replicate the results more than once (ie, at two or more different time points) in the same population.²¹ Test–retest reliability of the EQ-5D-3L in this analysis was evaluated by comparing the EQ-5D-3L responses at month 3 and month 6 among stable patients. A patient was considered to be stable if they met all of the following criteria: (1) change of <25 meters in the absolute 6MWT score between month 4 and month 6, (2) an absolute change of <4 points in the composite score of the SGRQ between month 3 and month 6, and (3) no change in culture converter status between month 3 and month 6 (ie, converted patients who do not have a relapse or non-converters who do not convert). Intraclass correlation coefficients (ICCs) along with 95% confidence intervals (CIs) were calculated for the EQ-5D-3L index and VAS scores between the 2 time points (ie, month 3 and month 6). The reliability of single-item dimensions of the EQ-5D-3L were evaluated using the weighted kappa statistic between month 3 and month 6. ICC values >0.75 and weighted kappa values >0.4 were considered to indicate acceptable test–retest reliability.²²

Validity

Known-groups validity is the ability of an instrument to differentiate among patients based on their disease severity.²³ Known-groups validity was assessed for the EQ-5D-3L index and VAS scores, grouping patients based on their ppFEV₁ (≤80%, >80%) and separately based on their 6MWT (≥350 m, <350 m).^19,24 Differences between known groups were examined using the Mann–Whitney U-test.

Construct validity is a measure of how well a measure correlates with a previously validated measure.²¹ Construct validity (convergent and divergent validity) was tested by examining the strength and direction of the correlation between the EQ-5D-3L individual dimensions, index, and VAS scores with the SGRQ activity scale, SGRQ impact scale, SGRQ symptom scale, SGRQ total score, and the 6MWT. Pearson’s correlation was calculated for VAS and index scores, while Spearman’s rank correlation was used for the individual dimensions. Correlations between 0.1 and <0.3 were defined as weak, 0.3 and <0.5 as moderate, and ≥0.5 as strong.²⁵ The hypothesized strength and direction of each of the correlations are depicted in Table 1.

Table 1 Hypothesized Direction and Strength of Correlations Between EQ-5D-3L, SGRQ, and 6MWT

Responsiveness to Change

Responsiveness to change is an instrument’s ability to detect a clinically meaningful change over time. In this analysis, meaningful change was defined based on 4 separate anchors: (1) ppFEV₁: relative change of >10% versus <5%,²⁶ (2) SGRQ total score: absolute change of >4 versus <4,²⁷ (3) 6MWT: absolute change of >25 meters versus <25 meters,²⁸ and (4) culture conversion status: patients who achieved culture conversion on or before the month 6 visit without relapse versus those who did not achieve culture conversion. Change in anchors and the EQ-5D-3L scores was calculated as the difference between the baseline and month 6 values on the respective measures. The responsiveness of the EQ-5D-3L was assessed by calculating the Guyatt responsiveness statistic and evaluating point biserial correlations. The Guyatt statistic was calculated as mean change in responder groups divided by the SD of change in the non-responder group. The point biserial correlations quantified the correlations between change in the EQ-5D-3L scores and responder categories (responders vs non-responders) based on respective anchors.²⁹ The Guyatt responsiveness statistic thresholds for weak, moderate, and strong responsiveness were <0.2, 0.2–0.5, and >0.8, respectively,³⁰ and the point biserial correlations' thresholds for weak, moderate, and strong correlations were 0.1 and <0.3, 0.3 and <0.5, and ≥0.5, respectively.²⁵ Cumulative distribution function (CDF) plots of change in EQ-5D-3L index scores and VAS scores, stratified on meaningful change for each anchor (conversion status, ppFEV₁, SGRQ, and 6WMT), were also plotted to examine the degree of separation between responders and non-responders at various score change values.

Patients with missing values for a measure at a particular assessment time point were excluded from the analyses involving that measure at that particular time point. For example, if a patient had missing data for the 6MWT but not for the SGRQ at month 6, the patient was excluded for analyses involving 6MWT data at month 6 but not for analyses involving SGRQ data at month 6. All analyses were performed using SAS 9.4.

Results

Descriptive Analysis

Out of the ITT population (N=336) for the CONVERT trial, only 1 patient did not have a baseline EQ-5D-3L assessment and hence was excluded in this analysis. The baseline demographic and clinical characteristics of the ITT population have been described in detail elsewhere.¹² Among the 335 patients with a baseline EQ-5D-3L assessment, 4 patients had missing data on 1 or more of the single-item dimensions, thereby preventing the calculation of an index score.

Minimal changes in the mean index and VAS scores from baseline (Index: 0.82; VAS: 65.88) to month 6 (Index: 0.83; VAS: 68.19) were observed (Table 2).

Table 2 EQ-5D-3L Index and VAS Scores at Baseline, Month 3, and Month 6

The distribution of responses to each single-item dimension at baseline is shown in Figure 1. No floor effects were observed among any of the EQ-5D-3L items. However, substantial ceiling effects were seen for all the items, with >50% of the respondents reporting “No problems” for each item. None of the respondents had the lowest possible index or VAS score, but 32.6% of respondents had the maximum possible index score of 1, representing perfect health. Minimal ceiling effects were seen for the VAS, with only 2.4% of respondents reporting a perfect score of 100.

Figure 1 Distribution of responses to each of the EQ-5D-3L items at baseline. Abbreviation: EQ-5D-3L, EuroQol 5-dimensional 3-level.

Test–Retest Reliability

50 patients were classified as stable, according to our criteria using culture conversion, 6MWT, and SGRQ. Among the stable patients, the ICCs for both the index (0.802) and VAS (0.852) demonstrated adequate test–retest reliability.²² Likewise, across all items, the kappa statistic was >0.4, indicating acceptable test–retest reliability (Table 3).

Table 3 Test–Retest Reliability of the EQ-5D-3L Items, Index Score, and VAS Score

Validity

The index scores were able to differentiate between the hypothesized groups (p<0.05) based on 6MWT (Figure 2A) and ppFEV₁ (Figure 2B), thereby suggesting adequate known-groups validity of index scores.

Figure 2 EQ-5D-3L index scores across known groups defined based on (A) 6MWT and (B) ppFEV1. Abbreviations: 6MWT, six-minute walk test; EQ-5D-3L, EuroQol 5-dimensional 3-level; ppFEV1, percent predicted forced expiratory volume in 1 second.

Similarly, statistically significant differences in the VAS scores were also observed between the known groups defined by 6MWT (<350m: 57.82; ≥350m: 68.79; p<0.001) and ppFEV₁ (≤80%: 60.38; >80%: 69.23; p<0.001).

Correlations of the EQ-5D-3L single-item domains, index, and VAS scores with the SGRQ domains, SGRQ total score, and the 6MWT were largely as hypothesized, supporting the instrument’s construct validity (Table 4).

Table 4 Spearman Correlations Between EQ-5D-3L, SGRQ, and 6MWT

Responsiveness to Change

Point biserial correlations between change over time in the response categories and the EQ-5D-3L index were small (Table 5). Similarly, based on the estimated Guyatt statistics, EQ-5D-3L index scores did not appear responsive to meaningful changes defined by culture conversion status, ppFEV₁, or 6MWT (Table 5). The EQ-5D VAS scores (results in Appendix Table 1) also displayed small to moderate responsiveness to change in culture conversion, 6MWT, and SGRQ.

Table 5 Point Biserial Correlations and Guyatt Responsiveness Statistics for EQ-5D-3L Index Scores by Each of the Anchors

CDF plots of change in EQ-5D-3L index scores, stratified on meaningful change for each anchor (conversion status, ppFEV₁, SGRQ, and 6WMT) are shown in Appendix Figure 1. The EQ-5D-3L index scores could not discriminate well between patients who had a meaningful change on the anchors versus those that did not, as indicated by the large degree of overlap in the curves. The median change in the EQ-5D-3L index scores was 0 for both groups, further showing the non-responsiveness of the index scores to meaningful changes in the anchors. The CDF plots for change in EQ-5D-3L VAS scores also showed that the change scores were similar among responders and non-responders for all anchors (Appendix Figure 2).

Discussion

In this study, we investigated the psychometric properties of the EQ-5D-3L among patients with MAC-LD who are refractory to GBT. Item and summary score distributions showed that the EQ-5D-3L VAS score had limited ceiling effects, but the individual domains and utility scores were subject to substantial ceiling effects. These findings complement a recent study by Hong et al, which also found ceiling effects in EQ-5D-3L domains and utility scores, with >70% of MAC-LD patients reporting no problems on individual domains and utility scores ranging from 0.91 to 1.00.¹¹ Additionally, we found that while the EQ-5D-3L is a valid and reliable instrument, it may not be responsive to clinically important changes among MAC-LD patients refractory to GBT.

We tested the reliability of the domains using the weighted kappa statistic and the index and VAS scores using the ICC. Both measures of reliability exceeded the thresholds for good reliability, with the weighted kappa for domains ranging from 0.50 to 1.00 and the ICCs for index and VAS scores being >0.8. These findings support the reliability of the EQ-5D-3L and suggest that it has good reproducibility and is unlikely to be subject to measurement error.³¹ Similar performance of the EQ-5D-3L has been noted in previous studies among patients with other chronic respiratory diseases, such as chronic obstructive pulmonary disorder and asthma.³²

We studied the construct validity of the EQ-5D-3L instrument by testing its concurrent and known-groups validity. The concurrent validity was supported with the correlations between the index and VAS scores and 6MWT and SGRQ falling in hypothesized ranges. The correlations reported in our study were similar to the estimates previously reported in the COPD population.^33,34 Specifically, Lin et al found a correlation of 0.29 and 0.45 of 6MWT scores with the EQ-5D-3L VAS, and Nolan et al reported correlations between SGRQ sub-scales and total scores, with EQ-5D-3L index and VAS scores ranging from 0.3 to 0.6.^33,34 Our results also demonstrate evidence supporting the known-groups validity of the EQ-5D-3L, with the index and VAS scores successfully differentiating between patient groups defined using 6WMT and ppFEV₁ thresholds. This finding is in agreement with previous studies which have shown that the EQ-5D-3L index and VAS scores were able to discriminate COPD patients based on their ppFEV₁ values.^35,36

However, the EQ-5D-3L index or VAS scores did not demonstrate good responsiveness in the MAC-LD population. The low point biserial correlations as well as weak Guyatt responsiveness statistics showed that detectable changes in EQ-5D-3L index or VAS scores were not observed for clinically significant changes over time detected on key study anchors. Hence, the EQ-5D-3L may not be a suitable instrument to detect longitudinal changes in HRQoL among MAC-LD patients refractory to GBT. The lack of responsiveness to changes in the ppFEV₁ is not surprising considering that longitudinal changes in HRQoL and FEV₁ have a poor relationship in other chronic respiratory conditions as well.^37,38 The lack of responsiveness of the EQ-5D-3L scores to sputum culture status is noteworthy, given that sputum culture samples are regarded as the primary biomarkers for assessing treatment efficacy in this population.^12,39 These findings are in line with Hong et al, who also reported that the EQ-5D-3L index scores were unable to distinguish between MAC-LD patients and healthy controls.¹¹ Previous studies assessing the responsiveness of EQ-5D-3L in other respiratory conditions, such as COPD and asthma, also conclude that the EQ-5D-3L is better suited to detecting change in moderate-to-severe patients, with limited responsiveness in mild disease states.^40–42

The lack of responsiveness of the EQ-5D-3L index to changes in the anchors may be partially attributable to the high ceiling effects (~33% of patients reported perfect health at baseline), which prevented us from detecting further improvements in these patients. Another potential reason for the lack of responsiveness could be that the range of responses for each dimension of the EQ-5D-3L may be too narrow to capture changes in HRQoL in the MAC-LD population.⁴³ This problem could be circumvented in future studies by opting to use the EQ-5D-5L in favor of the 3L. Nonetheless, a recent study by Nolan et al among COPD patients showed weak correlations between changes in EQ-5D-5L utility scores, with clinically significant changes in these patients.³⁴ Hoogendoorn et al proposed another potential solution for future consideration, which involved the addition of a respiratory dimension to the EQ-5D (3L or 5L).⁴⁴ They found that addition of a “breathing problem” dimension significantly improved the responsiveness of the EQ-5D to clinically significant changes in chronic lung disease patients.⁴⁴ However, such an approach would require valuation of the new dimension in order to obtain preference-based utility scores, which can be burdensome in terms of time and resources. Additionally, the comparability of the utilities obtained using this instrument with established national tariffs may also be questionable.

The results of this study are subject to some limitations. There is no “gold standard” measure of HRQoL in MAC-LD patients; hence we used a range of HRQoL and clinical measures available to test the psychometric properties of the EQ-5D-3L. The minimal clinically important difference thresholds for each anchor used in the responsiveness evaluation have been established among COPD and asthma patients and may not be applicable to MAC-LD patients. Another limitation is the limited duration of the study (6 months). Improvements in HRQoL as a result of culture conversion may take longer than 6 months to be realized. However, since CONVERT patients who did not achieve culture conversion by month 6 were either transitioned to a single-arm study or lost to follow-up, we were unable to test responsiveness using longer-term data. Lastly, our population of MAC-LD patients was restricted to those who were refractory to GBT, indicating that they had had MAC-LD for a considerably long duration (5.7 years on average). Previous studies have reported that patients with chronic disease tend to adapt to their condition,⁴⁵ which may have resulted in the high ceiling effects at baseline we observed in this study. Additionally, since our study population included only MAC-LD patients refractory to GBT, our findings may not be generalizable to the overall MAC-LD population.

Conclusions

Within this cohort of MAC-LD patients refractory to GBT, the EQ-5D-3L exhibited evidence of validity and reliability but may not be best suited to studying changes in HRQoL over time. The lack of responsiveness of the EQ-5D-3L may partially be attributable to high ceiling effects at baseline. Overall, this study highlights the need to develop HRQoL measures that can accurately capture and correlate with clinical improvements in the MAC-LD population. With the growing importance of HRQoL measures in the context of drug development and regulation,¹⁴ evaluating existing measures as well as identifying or developing new measures that can capture the HRQoL changes in this population is an important unmet need.