Do Generic Preference-Based Measures Accurately Capture Areas of Health-Related Quality of Life Important to Individuals with Amyotrophic Lateral Sclerosis: A Content Validation Study

Introduction

Amyotrophic lateral sclerosis (ALS) is a neurodegenerative disease characterized by selective and progressive loss of voluntary motor neurons.¹ Individuals with ALS experience a range of symptoms related to the loss of muscle control in limb, bulbar and respiratory functions.^2–4 Consequently, activities of daily living, independence, and health-related quality of life (HRQL) are all impacted by the disease and as no curative treatment is currently available, optimal treatment includes addressing symptoms and improving HRQL.^2,5–8

Improvement in HRQL, “an individual’s perception of how an illness and its treatment affect the physical, mental and social aspects of his or her life,”^9,10 is often considered the ultimate goal in healthcare,¹¹ and measures of HRQL can be used for treatment decision-making and outcome evaluation purposes.^11,12 Generic preference-based measures (GPBMs) are a type of HRQL measure designed to assess the cost-effectiveness of interventions due to their ability to produce a single index score, typically anchored from 0.0 (death) to 1.0 (perfect health).^13–15 This value can be used to calculate quality-adjusted life years (QALYs) by capturing the effect of an intervention on one’s quality of life (morbidity) and length of life (mortality).^13,16 GPBMs have been used to assess the HRQL of individuals with ALS and to aid researchers, policymakers and health care professionals in evaluating the cost-effectiveness of different treatment options.¹⁷

While generic measures include a set of common domains relevant across a variety of health conditions, they may not capture all the domains that are impacted by specific health conditions. When this occurs, scores from GPBMs may be higher than the true impact, resulting in incorrect comparisons across interventions and populations.¹⁸

Before a measure can be applied in practice, it must be tested to ensure that it is both reliable and valid.^19,20 Content validity is the degree to which the content of an instrument accurately reflects the construct to be measured: a fundamental aspect in considering whether a measure can be used in a population.²⁰ However, the content validity of GPBMs in individuals with ALS has not yet been evaluated. The Patient Generated Index (PGI) is an individualized measure that was developed to focus on the impact of a specific health condition on HRQL.²¹ It has been previously utilized to identify areas of quality of life important to patients in studies of content validity.^18,22 Therefore, the primary objective of this study was to assess the content validity of GPBMs in ALS. The secondary objective of this study was to examine the convergent validity of the EuroQol 5 Dimension 5 Level (EQ-5D-5L), a widely used GPBM for HRQL, against the Patient Generated Index (PGI), in ALS.

Methods

Participants

Participants were recruited from the Canadian ALS Research Network (CALS) outpatient clinics across Western (Edmonton, AB), Central (Hamilton, ON), and Eastern (Fredericton, NB) Canada. Participants were eligible for inclusion if they : 1) were 18 years of age or older, 2) had a clinical diagnosis of ALS, and 3) able to communicate, verbally or electronically, in English. Individuals with severe frontotemporal dementia were excluded.

Outcome Measures

This study involved the administration of an online or hard copy (paper and pen) questionnaire consisting of sociodemographic and clinical information, the PGI, the EQ-5D-5L and the Amyotrophic Lateral Sclerosis Functional Rating Scale-Revised (ALSFRS-R).

Sociodemographic and Clinical Information

Sociodemographic information consisting of age, sex, gender, residing region, highest education level, marital status, living situation and employment status was obtained. Clinical information consisting of year of diagnosis, ALS symptom onset location, clinic location and clinic visitation frequency was recorded.

Patient Generated Index (PGI)

The PGI²¹ is an individualized measure previously utilized to identify areas of quality of life important to individuals in studies of content validity. It has been used with chronic conditions such as cancer,^23–25 Parkinson’s disease,²² Multiple Sclerosis (MS)¹⁸ and Ankylosing Spondylitis.²⁶ It is completed in three stages. First, participants are asked to identify up to five of the most important areas of their lives affected by their health condition (ie, ALS). Second, they are asked to rate the extent of impact of each area on their lives from 0 (the worst you could imagine) to 10 (exactly as you would like it to be). A supplementary sixth item is provided to rate all other areas of life not mentioned. This can include additional areas of life affected by the health condition, as well as non-health related areas. In the third phase, participants are asked to imagine that they could improve some or all their chosen areas. Participants are given twelve weighting points to distribute across the five potential areas they would like to have improved, as well as the sixth item indicative of all other areas not mentioned. They can distribute these weighting points in any manner they choose but cannot use more than 12 points in total. More points allocated to an area indicate greater importance and hope of improvement. An average of 2 or more weighting points per area is considered meaningful.²²

The rating and weighting points allocated to each area are then multiplied and summed to produce a single index score of overall HRQL from 0 to 10, with higher scores indicating better HRQL.²¹ This score is typically reported as a percentage²⁷ and is intended to represent the extent to which reality matches expectations of perceived quality of life for those areas of life patients most value an improvement.^21,24 If there are missing data, an overall PGI score cannot be calculated.

EQ-5D-5L

The EQ-5D-5L,²⁸ developed by the EuroQol Group, is a well-established and widely used GPBM of HRQL that consists of two parts.^15,29 The first part (the descriptive system) assesses health in five domains: mobility, self-care, usual activities, pain/discomfort and anxiety/depression. Each domain has five response levels, scored from 1 (no problems) to 5 (unable/extreme).²⁸ A health utility value is derived from the five domains and is transformed into a single index score using a time-trade off (TTO) derived scoring system; a method of assigning values to health states from the population by asking respondents to choose between a shorter life in a state of perfect health or a longer life in a state of impaired health.³⁰ Canadian health utilities for the EQ-5D-5L range from −0.148 for the worst possible health state (55,555; a score of 5 for each of the health domains) to 0.949 (11,111; a score of 1 for each of the health domains) for the best possible health state.³⁰ The EQ-5D-5L describes a total of 3125 health states (5⁵), has been translated into more than 170 languages world-wide, and takes only a few minutes for participants to complete.³¹ The second part of the EQ-5D-5L consists of a visual analogue scale (VAS) of self-rated health, scored from 0 to 100.^31,32 The scores from the VAS cannot be used directly as weights in QALY calculations, as they not produce a single index value from a preference-based scoring system; however, the scores can be used as a subjective measure of one’s self-perceived health.³²

Self-Administered ALSFRS-R

The ALSFRS-R³³ quantifies degree of functional impairment in ALS, and consists of 12 questions across 3 domains: bulbar, motor and respiratory. The questions are rated on a 5-point scale from 0 (complete dependence) to 4 (normal function) and a total ALSFRS-R score, ranging from 0 to 48, is produced through summation of the individual item scores; with higher scores indicating better health and increased predicted survival.³³ The self-administered version of the ALSFRS-R has demonstrated excellent reliability (intra-class correlation = 0.93, 95% CI0.88 to 0.96) and sensitivity to change over time.³⁴

More recently, studies have evaluated the measurement properties of the ALSFRS-R using Rasch analyses^35,36 and longitudinal and survival analyses.³⁷ From their findings, researchers caution the reporting of a global score and recommend reporting domain-specific subscale scores organized into bulbar, motor and respiratory domains.^33,35–37 A total subscale score for each domain is produced through summation of the corresponding items: items 1 to 3 (bulbar), items 4 to 9 (motor) and items 10 to 12 (respiratory).³⁵ Bulbar and respiratory domains range in score from 0 to 12 whereby the motor domain ranges in score from 0 to 24; with higher scores indicating better function.

Procedure

Ethics approval for this cross-sectional study was obtained from McMaster University (HiREB #5664) and all sites in accordance with their respective research ethics boards. This study was conducted in accordance with the Declaration of Helsinki and all participants provided written informed consent. A designated clinician or research nurse located in clinic, identified, and recruited eligible participants. All participants were diagnosed with ALS using valid diagnostic criteria by board-certified neurologists specialized in motor neuron disorders. If interested, participants were given a hard copy of information including the invitation letter, consent form, and survey. Alternatively, interested participants could provide their email and a link to complete the LimeSurvey³⁸ was then sent by the research team. Participants could complete the questionnaire package by themselves or with the aid of a caregiver.

The domains generated from the PGI were classified independently by two reviewers (NP and JVD) using the World Health Organization’s International Classification of Functioning, Disability and Health (ICF).³⁹ The ICF was used as it provided a framework for coding and a standardized description of health-related problems at various levels (impairments, activity limitations and participation restrictions). A third and fourth reviewer (AK and VDBH) were consulted if consensus was not reached or ambiguity in responses were present. Established guidelines from Cieza and Stucki⁴⁰ were used for the coding process to ensure accuracy of coding between reviewers and to capture all relevant domains:

1. The domain nominated by the participant was coded to the most specific ICF code; if the reported area covered more than one code, then all codes were assigned.
2. To eliminate subjectivity, reviewers coded all possible interpretations of the domains.
3. Broader categories were used for coding if there were inconsistencies between reviewers in order to be as conservative as possible.

All individual and overarching ICF domains were then mapped onto the seven leading GPBMs: the EQ-5D-5L,²⁸ the SF-6D,⁴¹ the Health Utilities Index Mark 2 and 3 (HUI 2 and 3),⁴² the Assessment of Quality of Life 8-Dimension (AQoL-8D),⁴³ the 15-Dimension (15D)⁴⁴ and the Quality of Well-Being Self-Administered (QWB-SA) scale.⁴⁵ Mapping was performed by two independent reviewers (NP and JVD), with a third reviewer (AK) for consultation, if needed. The content coverage of GPBMs was determined by the percentage of domains included in the GPBM that were also nominated by individuals with ALS using the PGI. High and low percentages correspond with high and low content coverage, respectively. This methodology, as outlined in Figure 1, followed that of similar studies assessing content validity of GPBMs using the PGI.^18,22,46

Figure 1 Outline of the study procedures. Abbreviations: EQ-5D, EuroQol 5 Dimension; SF-6D, Short Form 6-Dimension; HUI2, Health Utilities Index 2; HUI3, Health Utilities Index 3; AQoL-8D, Assessment of Quality of Life 8-Dimension; 15D, 15 Dimension; QWBSA, Quality of Well-Being Self-Administered scale.

Sample Size

There are no sample size recommendations for content validation.⁴⁷ Therefore, our sample size calculation was based on: the number of participant responses needed to achieve data saturation (when no new relevant knowledge is being obtained from participants)⁴⁸ and; the recommended sample size for construct validation studies, which is a minimum of 50 patients total.²⁰ Studies have demonstrated that sample sizes around 15 to 20 are sufficient for saturation,^49–51 therefore, to satisfy both recommendations we aimed to recruit between 45 and 60 participants across the 3 clinical sites.

Data Analysis

Descriptive statistics consisting of parametric measures such as mean, standard deviation, frequency and percentage were calculated to analyze participants’ sociodemographic and clinical information. Scores for the PGI, EQ-5D-5L, and ALSFRS-R were computed according to their respective guidelines. Pearson’s correlation coefficient was used to measure the strength of the association between PGI and EQ-5D-5L scores, as the data were normally distributed. Only complete data were used to assess association. Correlations with instruments measuring similar constructs should be greater than or equal to 0.50,⁴⁸ therefore to assess convergent validity, a correlation of at least 0.5 was hypothesized between the PGI and EQ-5D-5L.⁵²

Results

Table 1 presents the sociodemographic and clinical characteristics for the sample (N=52). No participants were excluded due to severe frontotemporal disorder. A total of 35 participants completed the PGI in full. The mean age of the sample was 61 years old and 54% were females. Approximately half of the sample (52%) completed the questionnaire in hard copy format, with the remaining completing the online format. Of the study participants, 67% completed the questionnaire without the assistance of a caregiver. Participants were distributed across Western (46%), Central (37%) and Eastern (17%) Canada, with 33% of individuals visiting their designated clinic every 3 months. The mean time since diagnosis was 3.5 (2.95) years. The time since diagnosis ranged from less than 1 year ago (27%) to five or more years (12%). For our sample, ALS symptoms first began to appear in the upper and/or lower limbs for 75% of the sample. The mean total ALSFRS-R score was 30.4 (9.4). The following subdomain scores were calculated for the ALSFRS-R: bulbar was 8.5 (3.5), scored out of 12; motor was 12.2 (6.2), scored out of 24; and respiratory was 9.8 (2.8), scored out of 12. The mean PGI score was 25.4 (14.1) and the mean EQ-5D-5L score was 0.55 (0.24).

Table 1 Sociodemographic and Clinical Characteristics of Sample (N=52)

Table 2 outlines the ICF domains identified by participants to be most affected by ALS and their frequency of appearance (n). There were 78 individual domains identified which resulted in 25 overarching ICF domains. The top 10 overarching domains identified were: recreation and leisure (17%), lower limb mobility (11%), interpersonal relationships (9%), self-care (7%), housework and preparing meals (6.5%), speaking (6%), eating and swallowing (5%), work and employment (4%), upper limb mobility (4%), and daily routine and independence (4%).

Table 2 Overarching ICF Domains Identified More Than Once by Participants from the Patient Generated Index, Organized from Most Frequent to Least Frequent (Total n=291)

Figure 2 outlines the mean impact scores for each overarching ICF domain identified by participants. The three least impacted HRQL domains were upper limb mobility (mean score of 4.4), self-care (mean score of 3.6) and lower limb mobility (mean score of 3.2). The domains work and employment (mean score of 1.3) and recreation and leisure (mean score of 1.7) were the most impacted.

Figure 2 Distribution of mean impact scores, from 0 (the worst one could imagine) to 10 (exactly as one would like it to be), for the overarching ICF domains identified by the sample. Abbreviations: ICF, World Health Organization’s International Classification of Functioning, Disability and Health; n, frequency of appearance. Note: *Missing data (N=1).

Figure 3 outlines the mean number of points (out of 12 points) that participants allocated to each of the overarching ICF domains. The most desired areas for improvement were interpersonal relationships (30% of points, mean of 3.0 points), muscle and movement functions (29% of points, mean of 2.9 points) and speaking (29% of points, mean of 2.9 points). The area with the least desire for improvement was housework and preparing meals (15% of points, mean of 1.5 points).

Figure 3 Mean desire for improvement displayed as a distribution of mean number of points (out of 12 points, where higher points indicate a greater importance for improvement) allocated to classified overarching ICF domains. Abbreviations: ICF, World Health Organization’s International Classification of Functioning, Disability and Health; n, frequency of appearance. Notes: *Missing data (N=3); **Missing data (N=2); ***Missing data (N=1).

Table 3 presents the mapping of overarching ICF domains identified by individuals with ALS against GPBMs. The GPBM that covered the highest number of ICF domains identified by participants was the QWB-SA scale at 87% coverage. The HUI3 addressed the least number of domains with 33% coverage. The remaining GPBMs identified between 53% and 67% of domains. The domains covered by all GPBMs were lower limb mobility and emotions. The domains most commonly missing from GPBMs were: structures involved in voice and speech, which was only included in the QWB-SA scale; and caring for household objects, only included in the AQoL-8D. Supplementary Figure 1 presents a scatter plot of EQ-5D-5L scores plotted against PGI scores. A positive moderate Pearson’s correlation coefficient of 0.52 was observed between the two measures.

Table 3 Mapping of Overarching ICF Domains, Identified by ALS Patients, Onto Generic Preference-Based Measures

Discussion

This was the first study evaluating the content validity of GPBMs in individuals with ALS. Participants completed an individualized measure, the PGI, to evaluate the impact of ALS on their HRQL. Commonly reported domains, identified as areas impacted by ALS and rated in terms of desire for improvement, were classified using the ICF and consequently mapped onto GPBMs to estimate the extent to which these generic measures captured domains that were important to individuals with ALS.

Individualized measures provide a standardized method to identify aspects of a health condition that impact patients’ HRQL.^53,54 The PGI²¹ allowed individuals with ALS to identify the areas of their lives affected by their condition and assign a weight to each identified domain. The majority of GPBMs included approximately half of the areas reported on the PGI. The domains lower limb mobility and emotions were the only two areas identified by all GPBMs, however there was no one GPBM that included all the domains nominated by participants.

Domains self-identified as being affected by ALS encompassed three out of four ICF components - body structures (7%), body functions (13%), and activities and participation (80%). Domains nominated were relatively severely impacted, rated as very poor to poor²¹ with an average score of 2.7 out of 10 (See Figure 2), and important to their quality of life with an average of 2.4 weighting points out of 12 (See Figure 3) allocated across domains for desire for improvement. For example, recreation and leisure was the most commonly reported domain and not only was it severely impacted by ALS but the desire for improvement in this domain was heavily weighted. This was true to some fashion for all domains nominated by participants as the average impact of ALS on domains was rated as severe. For example, it is well known that ALS progression results in a decline in physical health.² Studies have shown that HRQL is not necessarily dependent on patient’s physical well-being but on their mental and social well-being.^55–57 The results of this study further demonstrate the impact of ALS on patients’ social well-being and independence. Therefore, we would thus expect GPBMs used in ALS to capture these nominated domains, yet this was not the case.

In assessing the content coverage of GPBMs, the GPBM with the least coverage was the HUI3; evolved from the HUI/HUI1 and HUI2.⁴² The HUI3 has been widely used in clinical populations, including neurological conditions.^58–60 It includes eight HRQL domains that focus on bodily functions: vision, speech, hearing, dexterity, ambulation, cognition, emotion and pain.⁶¹ Only a third of the domains identified in our sample were covered by this measure ie, lower limb mobility, speaking, upper limb mobility, emotions and undertaking a task. The HUI3 does not include domains relevant to social well-being, which was identified as important in our sample, and is missing many areas identified as important to their HRQL.

The QWB-SA scale had the highest percentage of included domains, with 87% of the domains deemed relevant to individuals with ALS. The QWB-SA scale is a comprehensive measure of HRQL that combines scales of functioning with a measure of symptoms and problems.⁴⁵ However, the measure is very symptom and limitation focused, and did not seem to translate to our sample. ALS affects all areas of life including participation areas, and our study showed that the effects of the symptoms, rather than the symptoms themselves, were the most impacted in our sample. The QWB-SA scale did not capture this well. Furthermore, the administration of the QWB-SA scale is lengthy, ie, takes around 14-minutes to complete,⁶² compared to the EQ-5D which only takes a few minutes to complete;³¹ this may be one reason why it is not as widely used. Additionally, a study by Smith et al (2000)⁶³ utilized the QWB-SA in ALS and found evidence corroborating decreased usage of the measure in this population, which may contribute further insight into the low utilization in ALS. Specifically, the measure demonstrated poor convergent validity with other measures of HRQL (ie, the SF-36 and Sickness Impact Profile/ALS-19). The authors explained their results in relation to the general makeup of the test; with items and valuations that may not accurately capture the physical symptoms of individuals with ALS or provide equal weighting to items that are associated with ALS (such as self-care).

The EQ-5D-5L is a widely used GPBM and has previously been used in the ALS population.^17,64–68 It covered 53% of domains identified by individuals with ALS (8/15 domains). However, it did not cover relevant and important domains such as speaking, eating, and swallowing or upper limb mobility, which are affected by ALS. For instance, domains comprised by the EQ-5D-5L (Mobility, Self-care, Usual Activities, Pain/Discomfort and Anxiety/Depression)²⁸ are comparable to the impairments and activity limitations identified by our sample. Nonetheless, the social aspects of ALS identified by our sample as being impacted, were not explicitly addressed by the EQ-5D-5L as a distinctive domain, which resulted in a lower coverage than expected.

The mean EQ-5D-5L score in our sample indicates a moderate health state, whereas the mean PGI score indicates poor HRQL. A lower HRQL score on the PGI in comparison to the EQ-5D-5L, suggests that GPBMs may underestimate the effects of ALS on the HRQL of patients. Furthermore, a moderate Pearson’s correlation between the PGI and the EQ-5D-5L was found in our study. A higher correlation was anticipated between the two HRQL measures,¹³ however it is not surprising considering there were identified domains not included in the EQ-5D-5L. Therefore, both the magnitude of correlation and percentage of content coverage in the EQ-5D-5L provide evidence of an overall lack of items relevant to individuals with ALS.

Strengths and Limitations

Our study is not without its strengths and limitations. To strengthen the generalizability of our study, participants were recruited from 3 different regions across Canada. Furthermore, we had a fairly equal distribution of men and women in our sample. An additional strength is in the number of years since diagnosis in that it is reflective of the ALS population in large epidemiological studies;⁶⁹ with the majority of individuals diagnosed within two to five years. Moreover, in our sample 75% of participants indicated upper and/or lower symptoms as the primary symptom recognized at onset; and limb-onset ALS affects 65% to 75% of individuals diagnosed.^70,71 Lastly, there was a wide range of ALSFRS-R scores in our sample, indicating mild to severe functional impairment, which is again reflective of the ALS population.⁶⁹

One limitation to this study was in utilizing the ICF as a coding framework. While the ICF has been used in similar studies, the framework is not all encompassing; therefore, some domains nominated by the sample, such as balance, were difficult to code. The second limitation was the amount of missing data present (N=17), particularly related to the completion of the hardcopy version of the PGI. As a result, we could only determine the magnitude of the association between the EQ-5D-5L and PGI for the 67% of the sample that had complete data.

Future Research

As this was a cross-sectional study, we did not evaluate the impact of symptom progression on HRQL and ALSFRS-R scores. It has been shown that rate of disease progression is a simple and sensitive clinical prognostic biomarker in ALS, when reflected over time by ALSFRS-R change score.⁷² This is an area for future research to investigate whether slow or fast progression differentially impact the type and order of domains generated by the ICF procedure and PGI. Another area for future research would be to compare ALSFRS-R scores to GPBMs and the PGI. As the ALSFRS-R is not a measure of HRQL, this calculation was not within the scope of this study. Such analysis by future studies could strengthen findings from this study by comparing quality of life and degree of patient ability.

In addition, due to a limited sample size, analysis on clinical information of the sample was restricted. For instance, epidemiological characteristics of ALS such as ALS phenotype (classic, bulbar, respiratory, flail arm, flail leg, pyramidal, pure lower motor neuron, and pure upper motor neuron)⁷³ was not established by researchers a priori. Scores from the ALSFRS-R may therefore be impacted due to perceived burden of disease which may vary with ALS phenotype. This is an area for future research as the impact of ALS and perceived burden of the disease may be reflected by GPBMs.

Conclusion

Content of preference-based measures needs to be reflective of the population’s values for accurate economic evaluation of treatments. Our results demonstrated that the majority of well-recognized GPBMs included only approximately half of the domains important to those living with ALS. The most commonly used GPBM in ALS, the EQ-5D-5L, correlated moderately with the PGI, however it underestimated the impact of ALS on the HRQL of patients. Likewise, in assessing the content validity of GPBMs, there were domains that were not identified, or that were inaccurately represented and not relevant to our sample. Findings from this study suggest the need for the development of an ALS specific preference-based measure with items that will capture the areas of life important to people with ALS and provide population-specific values that can be utilized for the assessment of treatment implications.