Content Validation of the ATTR Amyloidosis Patient Symptom Survey: Findings from Patient and Clinician Cognitive Debriefing Interviews

Introduction

Amyloid transthyretin (ATTR) amyloidosis is a rare, progressive, degenerative, and fatal disease in which insoluble amyloid fibrils, comprised of non-native forms of transthyretin (TTR) protein, deposit and accumulate in various organs and body systems, most typically in the heart and nerves.^1,2 There are 2 primary types of ATTR amyloidosis: wild-type ATTR (wtATTR) amyloidosis and hereditary ATTR (hATTR amyloidosis).

wtATTR amyloidosis occurs in the absence of a genetic mutation, and often manifests in men over the age of 60.^1,3 Prevalence of wtATTR is not frequently reported in published literature due to difficulty estimating true rates, and it is likely that wtATTR is significantly underdiagnosed, partly due to its complicated clinical presentation. However, 1 post-mortem study estimated a prevalence of 25% in individuals at least 85 years of age; likewise, other post-mortem studies have suggested that 10–25% of individuals over the age of 80 show signs of TRR amyloid deposition.^3–5 Clinical characteristics and symptoms of wtATTR include congestive heart failure, atrial fibrillation, and carpal tunnel syndrome.^1,3,6

hATTR amyloidosis is caused by a TTR gene mutation inherited from a parent; the worldwide prevalence is estimated to be 50,000, and the US incidence is estimated to be 1 in 100,000.^1,3,6 Clinical characteristics and symptoms of hATTR amyloidosis vary greatly; they include the development of peripheral neuropathy (damage to the peripheral nervous system), polyneuropathy ([PN] damage to multiple peripheral nerves), autonomic neuropathy, and cardiomyopathy (CM), due to the deposition of non-native TTR proteins in nerves and the heart.³ Other symptoms – due to deposition of TTR proteins in various tissues and organs – include gastrointestinal symptoms (eg, diarrhea, constipation, weight loss), ocular involvement, carpal tunnel syndrome (damage to the median nerve of the hand due to protein deposition in the surrounding tissues and ligaments), and renal dysfunction.^2,3,7 The precise organs involved, and subsequent symptoms experienced, depend in part on the specific TTR genotype mutation experienced by a patient. There are over 120 recognized mutations, some of which are more prevalent among descendants of specific geographic regions.² For example, the V30M mutation is typically seen among patients from Portugal, Sweden, and Japan; this mutation impacts the peripheral nervous system, autonomic nervous system, and heart. The V1221 mutation is more common among patients from the United States, Caribbean, and Africa, and impacts the peripheral nervous system and heart.⁸ While some patients may experience predominantly cardiac or neuropathic involvement (ATTR-CM, ATTR-PN, respectively), other patients experience genotype mutations associated with a mixed phenotype, developing both CM and PN.^3,9-11 This mixed phenotype illustrates the varied nature of the disease, which is often characterized by multi-organ involvement.

Patients with ATTR amyloidosis experience widespread impacts due to the disease, particularly in areas related to physical functioning,^12,13 with a burden of disease approximately equivalent to that experienced by patients with multiple sclerosis or congestive heart failure.¹² Given these impacts, as well as the mortality associated with the disease,^14,15 it is particularly important to track the symptoms that may contribute to these poor outcomes. A standardized way of tracking patient-reported symptoms would be especially useful for monitoring improvement in clinical practice and clinical trials aimed at identifying safe and effective treatments for ATTR amyloidosis. However, given the wide variety of symptoms experienced by patients with ATTR amyloidosis, existing patient-reported outcome (PRO) measures are often relevant and useful, but not sufficient. For example, the Norfolk Quality of Life–Diabetic Neuropathy (Norfolk QOL-DN) is a PRO developed for patients with diabetic neuropathy but has been used in many studies of patients with ATTR amyloidosis (despite a lack of evidence to support its content validity in this specific patient population). The Norfolk QOL-DN is limited to the measurement of symptoms related to neuropathy, including numbness, tingling, electric shocks, weakness, pain, lack of sensitivity to temperature, vomiting, diarrhea, and dizziness.^16,17 While these are known symptoms of ATTR amyloidosis, the Norfolk QOL-DN fails to capture other symptoms that patients with ATTR amyloidosis frequently experience, such as shortness of breath, chest pain, headaches, and urinary and fecal incontinence – among others. Thus, a PRO measure developed specifically to capture the symptoms of ATTR amyloidosis is necessary to allow for a more comprehensive and targeted assessment of patients’ disease experience.

The ATTR Patient Symptom Survey (ATTR-PSS) is a PRO developed in 2017 to assess the type, frequency, severity, and degree of impact of symptoms experienced by patients with ATTR amyloidosis. The ATTR-PSS was designed to be applicable to patients with either hATTR or wtATTR. Establishing the content validity of an instrument is a necessary component of PRO measure development, as it provides evidence that the measure assesses content areas that are appropriate and comprehensive given the intended population, and that items are easily understood and accurately interpreted by respondents.¹⁸ The initial development of the ATTR-PSS was informed by 1) literature review, 2) initial content validation through informal review and discussion with clinicians, 3) concept elicitation input from a patient advisory board, 4) review by a patient advocacy group, and 5) a formal concept elicitation and cognitive debriefing with a patient focus group that resulted in minor revisions to the survey (Figure 1). While initial evidence from the patient focus group supported the content validity of the ATTR-PSS, the content validation efforts to-date had not included a formal review by clinicians and included only a small number of patients in 1 focus group. Thus, it was determined that additional research should be conducted. This report presents the findings of a series of cognitive debriefing interviews with both clinicians and patients, with the primary objective of providing additional evidence of the content validity of the ATTR-PSS.

Figure 1 Development of ATTR-PSS.

Materials and Methods

Two sets of cognitive debriefing interviews were conducted consecutively to evaluate the content validity of the ATTR-PSS: 1) interviews with clinicians who treat patients with ATTR amyloidosis, and 2) interviews with patients diagnosed with ATTR amyloidosis.

Participants

Seven clinicians and 10 patients were recruited for the study. Clinicians with experience treating patients with ATTR amyloidosis were contacted via email. Efforts were made to invite both male and female clinicians who practiced in different geographic regions. Clinicians who responded to the email were scheduled for a telephone interview; each clinician was provided a toll-free number to use for the call and a copy of the ATTR-PSS prior to the interview.

Patients were recruited for the study through collaboration with the Amyloidosis Support Groups (ASG); information about the study was distributed through ASG’s social media pages. Patients were eligible to participate if they were at least 18 years of age, reported having been diagnosed by a doctor with ATTR amyloidosis, and were comfortable reading and communicating in English. A quota system was developed to include representation of patients with different types of ATTR amyloidosis, to ensure that different symptom experiences were captured in the interviews. The quota was set to include at least 2 patients with each of the following types: ATTR-PN, ATTR-CM, ATTR-PN and CM, and wtATTR. Because efforts were focused primarily on achieving diversity in ATTR amyloidosis type, and ATTR amyloidosis is a rare disease (making recruitment especially challenging), no formal quotas were implemented for other patient characteristics such as age, gender, education, or time since diagnosis.

ATTR-PSS

The draft version of the ATTR-PSS included a list of 40 different symptoms experienced by patients who have ATTR amyloidosis. In reference to the symptom list, patients are asked to indicate 1) how often they have experienced each symptom, 2) the severity of each symptom, 3) the 5 symptoms that have had the greatest impact on their daily life, and 4) the overall severity of their symptoms. All 4 items include a recall period of “the past month.”

Study Procedures

The clinician interviews were conducted by phone in March 2019. Approximately 1 week prior to the interview, clinicians were emailed a copy of the ATTR-PSS to review. At the start of the interview, the interviewer provided a brief description of the purpose of the interview and obtained permission from the clinician to audio-record the interview. The interviews followed a semi-structured interview guide; the topics included in the guide are depicted in Figure 2.

Figure 2 Content included in clinician and patient cognitive debriefing interviews.

Prior to the start of the patient interviews, the study and all associated patient-related materials were reviewed and approved by the New England Independent Review Board (IRB #120,190,082). The patient interviews were conducted by phone in April and May 2019. Approximately 1 week prior to the interview, patients were emailed an informed consent form (ICF) and a copy of the ATTR-PSS. Patients were asked to read, sign, and return the ICF in advance of the scheduled interview; it was requested that patients do not review the survey prior to the interview. At the start of the interview, the interviewer provided a brief description of the purpose of the interview, answered any questions on the ICF, and obtained permission from the patient to audio-record the interview.

The interviews followed a semi-structured interview guide. While the interview guide was primarily developed prior to the clinician interviews, it was revised to incorporate questions and probes related to topics that emerged during the clinician interviews. The topics included in the guide are depicted in Figure 2.

The interview started with a brief conversation regarding patients’ experiences with ATTR amyloidosis, and then turned to evaluate the ATTR-PSS. This part of the interview used cognitive debriefing methodology, which is an interview technique that prompts participants to discuss the relevance of items and their understanding of each aspect of the survey.^19–21 As part of this methodology, patients were instructed to verbalize their thoughts while reading and completing each part of the ATTR-PSS.²² During this think-aloud process, patients were asked to describe any aspects of the ATTR-PSS they found challenging or confusing. The interviewer then asked a series of targeted questions about the survey, including its overall relevance and the clarity of instructions, items, recall period, and response choices; patients were asked to comment on each of these and provide feedback on any survey-related topics that had not already been covered.

Data Coding and Analysis

All interview recordings were transcribed verbatim. Clinician and patient interview data were coded and analysed separately but followed the same procedure.

At the conclusion of each interview, a Microsoft Excel spreadsheet was populated with any issues that emerged from the interview that suggested a change be made to the ATTR-PSS. Such issues included survey elements perceived as confusing or difficult to answer, or suggestions provided by interviewees to improve the clarity of the survey. Each unique suggestion was recorded in a single row of the spreadsheet, with a separate column for each interview. As such, the spreadsheet tracked 1) each of the suggestions provided throughout the interviews, and 2) the number of interviewees who made the same suggestion. This preliminary coding took place prior to receiving transcripts and was solely based on interviewer notes. Next, transcripts of each interview were reviewed for quality and then cross-checked against the Excel spreadsheet in order to confirm all feedback had been accurately recorded.

After transcripts were reviewed against the information included in the spreadsheet, formal content coding was conducted; all remaining relevant data from the interviews were coded. All coded data were reviewed and analysed by a primary coder to identify necessary survey modifications. Finally, modifications were made to the survey through a consensus-based approach. Consensus was reached when all team members agreed on the modifications to the survey. Clinician or patient suggestions that were primarily personal preferences for wording or presentation style, or suggestions made by a single individual, generally did not lead to survey modifications. Suggestions aimed at improving the clarity or comprehensiveness of the survey (eg, modifying symptom descriptions) that were suggested by multiple clinicians or patients were evaluated by the research team and implemented as described in subsequent sections. The research team evaluated the perceived importance of each suggestion and subsequently decided whether a modification was needed. In certain limited instances, the research team was unable to reach consensus regarding whether a modification was necessary, or how a patient’s suggestion should be implemented. When this occurred, the research team reviewed existing literature, meeting minutes from the patient advisory board meeting, and transcripts from the earlier patient focus group to better understand the evolution of the survey items and the totality of evidence in favor of any particular modification. In 1 instance, researchers also contacted the clinicians who participated in interviews to gain additional insight regarding the best way to modify the survey in response to patient suggestions.

The survey was modified twice: once after clinician interviews, and once after patient interviews. As such, patients reviewed a draft of the survey that had been modified as a result of the information obtained from the clinician interviews.

Results

Clinician Cognitive Debriefing

Clinician Demographics

Characteristics of the 7 clinicians who were interviewed are summarized in Table 1. Five haematologists/oncologists, 1 neurologist, and 1 cardio-oncologist were interviewed. All clinicians had at least 5 years of experience treating patients with ATTR amyloidosis, reported experience treating both patients with hATTR and wtATTR, and had used PRO measures in clinical practice and/or clinical trials.

Table 1 Demographic Characteristics of Clinicians Who Participated in Cognitive Debriefing Interviews

Summary of Clinician Cognitive Debriefing Results

Clinicians generally agreed the ATTR-PSS symptom list was relevant, comprehensive, and inclusive of symptoms patients with hATTR and wtATTR experience. Clinicians found the order of the list logical and easy to follow. Representative quotations from clinicians regarding their evaluation of different aspects of the ATTR-PSS, and suggested modifications to the survey, are provided in Table 2. Clinicians confirmed that the ATTR-PSS captured the full extent of the patient experience of ATTR amyloidosis. Initially, 4 clinicians found the survey too long, but after an explanation of planned skip logic designed to alleviate respondent fatigue, 3 of the 4 agreed the length was appropriate. Clinicians were in agreement that the recall period of 1 month is an appropriate timeframe in which to ask patients to reflect back on their symptoms. Two clinicians worried that someone else might complete the form for the patient due to neuropathy in the hands; as a result, an instruction was added stating all answers should reflect the patient experience, and not the impressions or experience of a caregiver. Simple edits (eg, adding the words “each” and “had”) and formatting changes (eg, underlining) were made to 4 of the items in the ATTR-PSS to improve the clarity of the questions, based on clinicians’ suggestions.

Table 2 Overview of Results from Clinician Cognitive Debriefing

All clinicians identified several conditions on the symptom list they felt were better characterized as medical diagnoses than as symptoms (malnutrition, dementia, sleep apnea, spinal stenosis, carpal tunnel syndrome, stroke, depression, anxiety, and seizures) and as such might be difficult to evaluate on the scales provided (eg, it would be difficult for a patient to evaluate the frequency of carpal tunnel). Clinicians suggested these be removed from the symptom list, and added to a new item asking if patients had been diagnosed with or experienced any of the conditions. Likewise, clinicians indicated the symptoms of unintentional weight loss and weight gain were difficult to evaluate using the scales provided; these were removed from the symptom list and included as a separate item which asked whether the patient had experienced unintended weight loss or weight gain of 10 or more pounds.

Across clinicians, a total of 12 symptoms were recommended for addition to the ATTR-PSS. Of these 12 symptoms, 4 were recommended for inclusion by multiple clinicians: falls/sudden falling when trying to stand, fecal incontinence, rapid heartbeat/heart palpitations, and loss of taste/altered taste. Three of these symptoms were subsequently added to the ATTR-PSS, while the fourth (loss of taste/altered taste) was not added to the survey, but noted for interviewers to discuss with patients.

Clinicians found that, generally, the language used to describe each of the symptoms included in the ATTR-PSS was clear. However, 1 symptom was modified as a result of clinician feedback to more clearly align with the actual patient experience (original description: loss of sensitivity to temperature; revised description: loss of sensitivity to hot and cold). Two other symptoms, sensitivity to alcohol and pain (other than neuropathic pain), were not endorsed by clinicians as part of the ATTR amyloidosis disease experience but were left in the ATTR-PSS and explicitly probed in the patient interviews for clarity and interpretation.

Patient Cognitive Debriefing

Patient Demographics

Characteristics of the 10 patients who participated in cognitive debriefing interviews are summarized in Table 3.

Table 3 Demographic Characteristics of Patients Who Participated in Cognitive Debriefing Interviews

Summary of Patient Cognitive Debriefing Results

Overall, patients voiced favorable impressions of the ATTR-PSS and found the length of the survey to be acceptable. Representative quotations from patients regarding their evaluation of the ATTR-PSS are provided in Table 4. Cognitive debriefing interviews with patients confirmed the relevance of the symptom list and items of the ATTR-PSS, and provided positive feedback on the overall comprehensibility of the items, instructions, recall period, and response options.

Table 4 Overview of Results from Patient Cognitive Debriefing

The greatest difficulty patients experienced while completing the ATTR-PSS was deciding whether or not to endorse a symptom they experience but could not definitively attribute to their ATTR amyloidosis. To address this challenge, relevant text in the instructions for each item were bolded to emphasize that patients should only report on symptoms that are related to ATTR amyloidosis.

Four patients failed to notice that item 3 of the survey asks them to choose only 5 symptoms that have the greatest impact on their daily life. While this could be prevented in an electronic version of the survey through programming logic, additional formatting was also added to the instruction text of this item (eg, bolded font, revision of line breaks).

Patients recommended edits be made to 6 symptoms of the ATTR-PSS to improve clarity; the wording was revised on 5 of the symptoms, while 1 was removed entirely. Three different pain-related symptoms were originally included in the ATTR-PSS. Two described symptoms of neuropathic pain (“pain, numbness, or tingling in the feet or legs;” “pain, numbness, or tingling in the arms or hands”), while a third was meant to encompass any other type of painful symptom, and appeared in the symptom list simply as “pain.” While patients understood the 2 symptoms related to neuropathic pain, they (similar to the clinicians) had difficulty understanding how to interpret the third type more ambiguous pain symptom, particularly in comparison to the other specific types of pain on the list. After additional consultation with the clinicians who had participated in the cognitive debriefing interviews, this symptom was ultimately revised to read “any other type of pain,” which was placed after the other 2 pain-related symptoms and more accurately described pain that was not neuropathic in nature but could occur in any part of the body. The symptom related to sexual dysfunction was revised with language more clearly inclusive of sexual dysfunction experienced by individuals of any sex rather than having examples that would only be experienced by men. The symptom “stress” was changed to “stress due to ATTR amyloidosis,” to alleviate patient-reported confusion stemming from uncertainty regarding whether they should endorse the symptom if they are experiencing stress due to their condition or due to non-disease-related factors. Minor modifications to the wording of 2 other symptoms were made to increase clarity and facilitate accurate interpretation (“severe headaches or migraines” was revised to “headaches;” “other (please specify)” was revised to “other symptoms (please specify)”). The symptom “sensitivity to alcohol” was removed entirely from the symptom list, as none of the patients in the study reported experiencing it, and both clinicians and patients expressed confusion regarding how to appropriately interpret its meaning.

In response to patients’ suggestions, 2 conditions, congestive heart failure and Crohn’s disease, were added to the item asking whether patients had been previously diagnosed with certain medical conditions.

Discussion

This qualitative research study was designed to elicit feedback from both clinicians and patients on the comprehensiveness, comprehensibility, and ease of use of the ATTR-PSS. Due to the varied nature of ATTR amyloidosis, developing a single survey that captures all relevant symptoms while not being overly burdensome for patients to complete is especially challenging. Nevertheless, our study provides strong evidence that the ATTR-PSS overcomes these obstacles. The ATTR-PSS is the first known PRO measure designed specifically for patients with ATTR amyloidosis, with demonstrated evidence of content validity following the FDA’s guidance for measurement development.^18,23

The cognitive debriefing interviews with both clinicians and patients provided ample evidence that the ATTR-PSS is relevant to patients with different types of ATTR amyloidosis, comprehensive, appropriate, and easy to understand and complete. While the revised survey includes a total of 32 symptoms and 12 other medical diagnoses or complications, the majority of participants found the length of the survey acceptable, especially given the likelihood it will be available as an online survey or mobile application, which will include electronic programming logic to the facilitate survey administration. Importantly, the cognitive debriefing interviews provided evidence to support the use of the ATTR-PSS in patients with hATTR and wtATTR, as the symptom list is comprehensive enough to capture symptoms associated with CM, PN, and other organ involvement. Clinicians who reviewed the ATTR-PSS not only agreed that it would be useful to incorporate into clinical trial research but also acknowledged its potential benefit in a clinical practice setting (especially if it was formatted for electronic administration). Given its potential for widespread use across a variety of contexts, future research should focus on evaluating the psychometric properties of the ATTR-PSS and developing a scoring algorithm.

Table 5 depicts the revised survey items, while Figure 3 summarizes the modifications made to the ATTR-PSS symptom list as a result of clinician and patient feedback. In total, 6 symptom descriptions were revised, 3 symptoms and 2 medical diagnoses were added, and 1 symptom was removed. In addition, weight loss and other complications were removed from the symptom list, and added to 2 new separate items. In total, the 6 revisions and 5 additions accounted for only 25% of all symptoms/conditions included in the revised survey, while the majority of the survey remained unchanged. Together these revisions – along with minor modifications to formatting and instruction text – ensure all relevant symptom-related patient experiences can be reported through the ATTR-PSS, and help increase the ease with which patients can accurately interpret and respond to the survey.

Table 5 ATTR-PSS Survey Items

Figure 3 Modifications to ATTR-PSS symptom list. Conditions listed in “medical diagnoses” and “other” categories were originally included as part the symptom list. At the suggestion of clinicians, these conditions were removed from the symptom list and added as 2 new items to the ATTR-PSS (as represented by the * symbol). Revised symptoms (represented by the * and § symbols) were modified for language/clarity; the updated language is reflected in the figure.

This study had several strengths. Among them, the patient sample included both those with hATTR and wtATTR. Of the patients with hATTR, some had symptoms of CM, some had symptoms of PN, and some had symptoms of both, confirming the survey is relevant across patients with mixed phenotype and diverse symptom experiences. Second, the 2-part design of the study allowed for an iterative approach to survey modification. In interviewing clinicians first, researchers were able to modify both the survey and the patient interview guide to take into account feedback from the clinicians. In some cases, feedback from the patients was then presented back to the clinicians before a final determination was made. In this way, it was ensured that modifications to the survey were appropriate and accurately reflected the input of both those living with the disease and those with experience treating the disease.

As with any study, limitations also existed. While collaboration with an advocacy group allowed for relatively fast recruitment of 10 patients who fit both the study’s inclusion criteria and sampling quotas, it is possible this approach also led to the inclusion of more well informed, engaged participants. All 10 patients were college educated, and 6 of the 10 patients had post-graduate degrees, thus their education and literacy levels may be higher than the average patient. As such, aspects of the survey they found understandable or easy to interpret may not be similarly clear to patients with lower literacy levels. In addition, while younger patients did participate in earlier phases of survey development, all of the patients in this study were over the age of 50, thus primarily representing individuals experiencing late-onset presentation. The clinicians involved in review of the survey were primarily based in the US, and the majority were haematologists/oncologists. It is possible that clinicians in Europe, Asia, or Latin America would have additional experience treating patients with genotype mutations that are less common in the US. While modifications to the ATTR-PSS based on patient interviews were not major in nature, best practice would suggest the modified ATTR-PSS undergo additional interviews with patients to ensure modifications made did not add complexity or confusion. However, of the 4 symptoms that were revised after the patient interviews, 2 were reviewed by clinicians or subject matter experts to ensure that confusion or unnecessary complexity would not be introduced by the survey modifications.

Given the results of the study, which support the content validity of the ATTR-PSS, future work should focus on evaluating the psychometric properties of the instrument, adapting it for use in languages other than US English, and formatting it for electronic administration.

Conclusions

The ATTR-PSS is an understandable and easy to use assessment of the symptoms of ATTR amyloidosis and is intended for use in patients regardless of the type of ATTR amyloidosis with which they have been diagnosed. This survey provides a comprehensive evaluation of symptoms and experiences not measured by other PROs. Use of this survey, whether as part of routine clinical care or to measure an endpoint in clinical trials, can help contribute to a more complete assessment of a patient’s health status.