Evaluating the Content Validity of the Modified Neuropathy Total Symptom Score-6 Self-Administered (mNTSS-6-SA) in a Painful Diabetic Peripheral Neuropathy Population

Introduction

Diabetic peripheral neuropathy (DPN) results from nerve damage caused by type 1 or type 2 diabetes mellitus (DM). DPN is the most common form of distal sensory polyneuropathy (DSP), affecting the longest nerve fibers first, typically symmetrically starting in the toes, feet, and lower limb and then affecting the hands.^1,2 DPN is often progressive and bothersome, resulting in clinical complications and impairments in functioning. DPN is frequently accompanied by pain (pDPN), affecting approximately 2.3 million Americans and 30% of the diabetes population globally.^1–3 A variety of treatments have been used for pDPN and new treatments are being developed.^2,4 Patients continue to call for increased awareness of the consequences of pDPN, highlighting the need for more focused research.⁵ Robust outcome measures are needed to assess and monitor symptoms of pDPN.

The Neuropathy Total Symptom Score-6 (NTSS-6) was developed as a 12-item, clinician-reported outcome (ClinRO) measure to evaluate sensory symptoms of neuropathy in individuals with DPN.⁶ The measure was developed based on a literature review and consultations with experts in neurology and peripheral neuropathy⁶ and has been translated into 40 different languages. The NTSS-6 evaluates the severity (items A) and frequency (items B) of six neuropathic sensory symptoms on a 4-point verbal response scale with a 24-hour recall period. Symptoms measured include, 1) numbness and/or insensitivity, 2) aching pain or tightness, 3) burning sensation, 4) prickling and/or tingling, 5) sharp, shooting, or lancinating pain, 6) allodynia and/or hyperalgesia.⁶ The measure produces scores of 0–21, where a higher score indicates more frequent/severe symptoms. A self-administered (NTSS-6-SA) version of the measure was also developed and its’ psychometric properties evaluated among people with DM.⁷ The NTSS-6-SA includes the same 12-items and response options with a 4-week recall period. Both versions were found to provide a reliable assessment of treatment in neuropathic symptoms that shows evidence of validity in the DPN population.^6,7 Several recent studies have used the NTSS-6^8–10 or NTSS-6-SA¹¹ to assess neuropathy in DM and other conditions. The NTSS-6 has also been used as a secondary endpoint in a randomized clinical trial for treatment of symptomatic DPN.¹²

Patient-focused drug development guidance from regulators recommend using clinical outcome assessments (COAs) that measure how a patient feels, functions, or survives for the evaluation of treatment benefit in clinical trials, and are critical for patient-focused drug development.¹³ Evidence of content validity is essential to promote confidence that a selected COA is measuring the intended construct and that patients understand the patient-reported outcome measure (PROM) as intended, and this evidence is best gathered at an early phase (eg, when selecting or developing COAs for clinical trials). Regulators recommend assessing the content validity of the instrument by considering patients’ experience of the disease (eg, identifying concepts that are relevant and important), what type of COA is most appropriate to assess the concept of interest (ie, the aspect of an individual’s clinical, biological, physical, or functional state), and the recall reference period.¹³ The suitability of specific PROM recall periods depends on several factors, including the specific aims of each trial, the types of experiences respondents are asked to recall, the level of burden imposed by repeated assessments (if applicable), and the risk of recall bias that applies to memory of experiences over the selected time frame.¹⁴

This underscores the value of establishing the validity of a given PROM with more than one option for the recall period. For example, a PROM using a 7-day recall period may decrease patient burden and increase compliance in completing a measure compared with a daily assessment with a 24-hour recall period, and may result in less recall bias than a 4-week assessment.¹⁵ The current study aimed to evaluate the content validity of the NTSS-6-SA, modified for use with a 7-day recall period, in a sample of individuals with pDPN, with the goal of gathering evidence to support its inclusion as an outcome measure in future clinical trials.

Materials and Methods

Study Design and Participants

This was an observational, cross-sectional qualitative study consisting of one-time interviews with adults with a physician-confirmed diagnosis of DPN, who self-reported moderate-to-severe pain (ie, people with pDPN). Interviews consisted of concept elicitation and cognitive debriefing of multiple PROMs, though only data relative to the modified NTSS-6-SA (mNTSS-6-SA) are presented in this manuscript; the order in which PROMs were presented varied across interviews to mitigate the risk that data collection was biased by interviewer or participant fatigue. A sample size of 25 was targeted to optimize the opportunity to achieve data saturation for concept elicitation, which refers to the point at which no new relevant concepts are being identified during data collection.^16,17 In cognitive debriefing, sample size was justified by obtaining evidence that the items were interpreted and understood as intended by most individuals, items were relevant to participant experience, and there was no missing content. Interviews lasted 90 minutes via web-conferencing and were conducted by members of the research team with training and experience in qualitative data collection; interviewers were allotted a minimum of 17 minutes for debriefing the NTSS-6-SA. Interviews were conducted in two waves: an initial wave of 13 interviews that provided data indicating that revisions to the NTSS-6-SA would be beneficial and a second wave that provided data affirming that the revisions were well understood and appropriate. One interview took place while revisions were prepared and that participant did not review the NTSS-6-SA. All interviews were recorded and professionally transcribed, then deidentified for analysis. Participants signed an informed consent form that included consent for publication of deidentified quotes in scientific publications. Ethical approval was provided by Advarra CIRBI, an independent institutional review board, on May 23, 2023, and interviews were conducted June through September 2023.

Participants were recruited and screened by third-party professional research recruiting firm (Global Perspectives, an IQVIA business), and participants who met the study eligibility criteria were enrolled. All participants provided informed consent to participate in the study. Individuals were eligible for the study if they were a resident of the United States (US), aged 18 years or older, experienced DPN in their feet or toes, had experienced pain from DPN for at least one year, and self-reported moderate-to-severe pain from DPN (a score of 4 to 9 on a 0–10 scale). Participants also provided a confirmation of diagnosis of DPN (eg, a screen capture or photo of information from a medical chart portal, letter from the participant’s provider confirming diagnosis of DPN). Individuals were excluded if they were currently participating in a clinical trial for treatment of DPN or pDPN or if they self-reported any of the following conditions that could affect their ability to distinguish symptoms specific to pDPN: lumbar sacral radiculopathy present currently and in past month, active sciatica, fibromyalgia, or amputations of lower limbs or toes due to diabetes. Individuals were also excluded if they self-reported having a disorder that could compromise their ability to give informed consent or participate in an interview (eg, untreated schizophrenia or bipolar disorder, stroke, traumatic brain injury, dementia), and if they were an employee or family member of an employee of the sponsor, researchers, or any other collaborators associated with this project. Participants who took part in an interview were compensated for their time and expertise.

Interviewers conducted virtual concept elicitation and cognitive debriefing interviews of several measures, including the NTSS-6-SA with a 7-day recall period, using a semi-structured interview guide. Approximately 20 minutes of each interview included a concept elicitation portion focusing on understanding patients’ perceptions and experiences of living with pDPN and elicited descriptions of symptoms and impacts of pDPN. The cognitive debriefing of the NTSS-6-SA aimed to gather evidence from the pDPN population related to comprehension and ability to recall and respond to the measure using the provided response options for the requested 7-day time period. Participants were also asked about relevance of item concepts. Interviews were conducted in two sequential waves. Wave 1 interviews assessed the 7-day measure, which resulted in modification to the instructions and items. Wave 2 interviews assessed the modified measure (see Figure 1).

Figure 1 Implementation of modifications to the NTSS-6-SA.

Analytic Approach

Content and thematic analysis methods were used to identify pDPN symptoms concepts reported by participants.^18,19 Using NVivo Windows qualitative data analysis software, analysts reviewed and coded the transcripts for concepts and terminology related to pDPN symptoms and impacts, then organized codes into hierarchical groups based on similarity of concept content. A saturation analysis was conducted to assess the number of new concepts identified in each interview and determine when concept saturation was reached. For the cognitive debriefing data, analysts assessed whether each participant understood NTSS-6-SA instructions, items and recall period as intended; had any difficulties with comprehension or selecting a response; found concepts relevant to their own experience; or had any other issues or suggestions for the measure (eg, suggestions for wording, missing concepts, perceived redundancies). Three coders independently coded the first three transcripts and then met to discuss coding agreement and refine the codebook. The remaining transcripts were then independently coded, with analysts meeting regularly to discuss the definitions, scope, and application of codes. Transcript analysis was conducted in NVivo Windows, a qualitative data analysis program.

Results

Participant Demographics

The study included 25 participants with pDPN of varied age, sex, race or ethnicity, educational background, and employment status (see Table 1). Participants self-reported pain from DPN on a numerical rating scale of 0–10, with 80% (n=20) reporting moderate pain (scores of 4–6) and 20% reported severe pain (scores of 7–9). More than half of the sample (n=15, 60%) reported using prescription medication to treat their pain and most (n=21, 84%) reported using over-the-counter medication. Duration of participants’ experiences with pDPN symptoms ranged from 1 to 10 years, with a sample mean of 3.6 years.

Table 1 Participant Demographic Characteristics

Of the 25 study participants that completed concept elicitation, 24 completed cognitive debriefing of the original (Wave 1) or modified version (Wave 2) of NTSS-6-SA. When comparing the demographic characteristics of the two waves, Wave 1 (n=13) and Wave 2 (n=11) differed slightly. Specifically, Wave 2 was less racially diverse, with the majority of participants being white (n=10, 91%) compared to Wave 1 (n=6, 46%), but Wave 2 was more ethnically diverse (n=4, 36% vs n=1, 8% were of Hispanic background). Wave 2 participants also included more older individuals, aged 60–69 (n=5, 45% vs n=3, 23%).

Concept Elicitation Results

All 25 participants completed the concept elicitation portion of the interview, which focused on understanding patients’ perceptions and experiences of living with pDPN, with questions eliciting descriptions of symptoms and impacts of pDPN. Analysis identified 9 domains for pDPN symptoms (pain, sensory abnormalities, muscle weakness) and impacts on functioning (physical, social, emotional, sleep, cognitive, sexual), as described below. A saturation analysis demonstrated that no new concepts were identified following the 21^st interview.

pDPN Symptoms

Participants described their symptoms in a variety of ways that have been categorized in 3 domains: pain, sensory abnormalities (numbness, tingling, sensitivity to touch/allodynia), and muscle weakness. Descriptors of pain reported by many participants included burning sensation (n=18, 72%), sharp or stabbing pain (n=17, 68%), and aching (n=15, 60%). Less frequently reported pain descriptions included tightness (n=9, 36%), electric shock-like pain (n=7, 28%), shooting (n=5, 20%), squeezing (n=4, 16%), cramping (n=4, 16%), stinging (n=2, 8%), and pulling (n=2, 8%). Sensory abnormalities of numbness (loss of sensation) (n=23, 93%) and tingling (n=20, 80%) were commonly reported by participants. As participant 2 described, “I started just getting a lot of numbness and it just kind of progressively got worse”.

Impacts of pDPN

When describing pDPN impacts on quality of life, nearly all participants reported that pDPN in the feet or legs interfered with basic daily physical functioning (n=24, 96%), including any activity or task that required walking or standing. For example, participant 4 shared, “I do not want to stand when I’m in pain because standing on it, actually, I think increases the pain intensity”. Most participants (n=23, 92%) reported experiencing an impact on their emotional well-being, describing feelings of irritability towards others when experiencing pain; frustration due to function impairments; anxiety about symptom management or a worsening of one’s condition; fear of falling; and depression related to perceived deterioration of their quality of life. About half of the participants (n=14, 56%) reported experiencing negative impacts on their social lives as a result of living with pDPN. Participant 10 explained that experiencing pain “makes you angry sometimes and irritated and just not a fun person to be with”. Participants (n=13, 52%) also reported impairments in cognitive function related to pDPN. These experiences were most often described as having a diminished mental acuity due to pain consuming one’s thoughts, making it challenging to think and focus normally. Other reported impacts related to sleep, sexual functioning or desire, or discussion of treatment burden.

Cognitive Debriefing Results

Analysis of interview data from the first wave of participants (n=13) indicated that the lengthy instructions were considered unnecessary and identified issues with comprehension and readability for some of the items (see Table 2). Modifications were made to the instructions and item rephrasing, and the mNTSS-6-SA was then reviewed in the remaining 11 interviews (Wave 2).

Table 2 Modifications to NTSS-6-SA After Wave 1

Wave 1

The NTSS-6-SA retained the original instructions, which consisted of one full page of text that described the purpose of the measure, instructed the respondents to answer based on experiences with symptoms in the feet and legs only and provide answers based only on symptoms of pDPN, and provided detailed definitions of each of the response options. Although most Wave 1 participants (80%) were able to provide at least a partial summary of the original instructions, evidence suggested that many participants were not reading the full instructions (Table 2). For example, participants were not able to answer basic questions about the content after their initial reading, made spontaneous comments indicating that they thought that the instructions were hard to recall, and made comments suggesting that they were too long and unnecessarily detailed. Participant 20 shared, “I just think it could be condensed, you know”. Other comments from participants suggested that the final sentence alone provided sufficient instruction. This sentence directed respondents to select one response that best described their symptoms. After seeing this sentence, for example, participant 5 remarked, “See, that makes sense. That makes sense”. Following these findings, the instructions were modified to only include the original last summary sentence. This change then resulted in the removal of the phrase “see definitions above” in item 1B.

Two items (2A and B, 3A and B) were well understood and not modified following Wave 1. Modifications were made to the other four questions based on participant feedback, which focused on comprehension issues (see Table 2). For example, 39% of Wave 1 participants (n=5) reported issues with the term “boring” (Item 1A), resulting in removal of this term. As participant 20 described, “I was confused. When you said boring, I thought it was a misspelled word”. Phrasing in item 4A was modified as four participants (31%) demonstrated issues with interpretation; two participants misunderstood the item and perceived the item to be asking about when they are asleep, and two participants expressed difficulty with the word “prickling”. Participant 5 preferred the other word used in the item, “Yeah, I’m just using, you know, instead of the ‘prickling,’ I’m just using ‘tingling’ in my head”. Data also demonstrated the need for clarification around the word “asleep”. Consequently, the item was modified to remove the phrase “without a prickling feeling” and change the phrase “asleep feeling” to “like your foot falling asleep”. Item 4B was modified to remove the phrase “without prickling” to align with modifications to item 4A. All participants understood the concepts in 5A to be asking about pain associated with DPN. However, when participants were asked which term they prefer or use most often to describe their experiences, the most frequently mentioned term was “stabbing”. As a result, the item was modified slightly to clarify that respondents should report on experiences that matched any of the terms in the question. Based on findings and modification to include the phrase “any of these” for 5A, all of the terms listed in the original item 5A were included in 5B. Finally, for item 6A, data indicated that this item was complex, requiring participants to assess both walking and situations where their feet were touched, and determine what constitutes “unusual sensitivity or tenderness”. The phrase “when your feet are touched” was modified to read “when something touches your feet” for ease of understanding, as some interpreted the original phrasing to mean when another person (as opposed to something) touches your feet.

With few exceptions, participants in Wave 1 found the items relevant. Two participants found item 4 (numbness without prickling) not relevant because they experienced only numbness accompanied by prickling. Additionally, two participants indicated that item 6A (allodynia, hyperalgesia) was not relevant. One found the phrasing confusing and the other viewed the item as less important than the other symptoms addressed in the measure.

Wave 2

Participants found the mNTSS-6-SA instructions and items clear and easy to understand and could recall and rate their experiences over a 7-day period using the available response options. All participants in Wave 2 (n=11) provided an accurate interpretation of the instructions. Reducing the details of the instructions did not affect the participants' understanding or ability to respond to items and did not lead to any noticeable differences in the way responses were selected. Refinement of the phrasing of items 1, 4, and 6 increased comprehension of the items. These modifications improved the comprehension of the scale without introducing new challenges and participants did not identify the need for further modifications to any items.

Overall, participants in Wave 2 found the measure to be an appropriate tool to evaluate neuropathic symptoms. The majority of participants reported that the mNTSS-6-SA was relevant to their experience with pDPN. However, about a quarter of participants (27%) reported that item 6 was not relevant, as two did not experience this type of sensitivity in their feet, while one other participant did not find this item relevant due to the inclusion of the word “unusual”. This individual noted these experiences to be usual for them and thus perceived that the question did not apply to them.

Overall

The cognitive debriefing results indicated that participants generally found the mNTSS-6-SA relevant to their experiences (Table 3) and found it easy to recall and rate their experiences within a 7-day period. Participants interpreted the recall period as “the last week” and 18 of the 20 (90%) participants who discussed the recall period perceived no difficulty in recalling their experiences over that time period. Participant 19 discussed the ease of recalling the last 7 days, “…it’s fairly fresh. Anyone can remember the last seven days, can remember details, things that they did, or that happened to them”. Furthermore, the concepts and terms reflected in the measure items aligned with those spontaneously reported by participants during the concept elicitation portion of the interview.²⁰ Collectively, participants did not perceive the measure to be missing any important concepts. One participant suggested adding an item about emotional impact and another participant suggested adding an item about cramping. These isolated suggestions for missing content underscore the overall finding that the measure addresses the central symptom experiences pertinent to pDPN.

Table 3 mNTSS-6-SA Item Relevance

Discussion

The NTSS-6 is a brief instrument that has demonstrated the capacity to provide valid assessments of the severity and frequency of neuropathic sensory symptoms in patients with pDPN in clinical or research settings.^8–10 As the NTSS-6 was designed based on the literature and clinical expertise,⁶ this study expands the evidence of content validity of the scale for populations with pDPN based on patient perspectives obtained through concept elicitation and cognitive debriefing interviews. Findings from this study resulted in a modified NTSS-6-SA (ie, the mNTSS-6-SA) that incorporated patient feedback on the length of instructions and the clarity of items, but did not alter the conceptual content of the measure. Use of a 7-day recall period was also assessed, as this time period reduces participant burden compared with a daily assessment and may reduce recall bias compared with a 4-week assessment, potentially improving data quality, which are important considerations when assessing changes over time or treatment effects within clinical trials. Use of a 7-day recall across multiple PROMs within a clinical trial may further reduce recall bias as participants consistently think about the same time period when responding to these items. Study participants found the mNTSS-6-SA easy to understand and complete, as well as comprehensive and relevant to their experiences of pDPN symptoms. Participants found it easy to recall and rate their experiences over the 7-day reference period using the provided response options.

Individuals with pDPN in the feet or legs experience symptoms that impact their physical, social, and mental functioning, which in turn affects their quality of life.^1,2 A rigorous, evidence-based, patient-reported assessment of the frequency and intensity of sensory neuropathic symptoms is needed to support the development and evaluation of patient-centered treatments for the condition. This study provides evidence that the mNTSS-6-SA is a viable option for assessing neuropathic symptoms through a PROM for individuals with pDPN. Alignment between the concepts measured in the mNTSS-6-SA and the key symptoms of pDPN described in the concept model,²⁰ in addition to the cognitive debriefing results, highlight that the mNTSS-6-SA is well understood, comprehensive, and is relevant to the pDPN population. In parallel to the results of previous studies of the clinician-administered and self-administered versions of the NTSS-6 that demonstrate good psychometric properties,^6,7 the current study complements these results, indicating that the mNTSS-6-SA is a suitable measure of neuropathic symptoms.

The main strength of this study was the robust, representative sample of adults with pDPN. The sample included a balanced mix of men and women, along with substantive diversity in race, ethnicity, age, employment status, and level of education. Additionally, all participants provided a confirmation of diagnosis by a physician as opposed to self-reporting DPN.

Several limitations apply to this study. First, a primary goal of cognitive debriefing is to assess whether participants understood the items as intended. However, the original developer’s intended meaning of the items was not always defined. Consequently, assessments of whether the patients understood and responded to the items as intended are based on this research teams’ best interpretation about the item’s intent. Second, the hybrid interview format, which included concept elicitation and the debriefing of several other PROMs in addition to the NTSS-6-SA, resulted in challenges in addressing all topics for each of the items in a timely manner during the cognitive debriefing, particularly when participants were talkative. Consequently, some participants were not asked every follow-up question for every item. However, all included PROMs were discussed in relation to participants’ experiences of pDPN, and data from these participants reflected significant uniformity and do not suggest that improved consistency in item-by-item data collection would have altered the overall findings. All participants supplied data used to determine comprehension and relevance of items. Finally, the sample was limited to the US-based, English-speaking participants and, as with any small qualitative study, the results may also be unintentionally influenced by the specific sample of self-selected participants.

Conclusion

The final modified NTSS-6-SA exhibited robust content validity for US adults with pDPN. This measure was well understood, interpreted as intended, and includes relevant concepts related to patient experience. Participants were able to rate their experiences with the provided response scale within the requested 7-day recall period. The concepts in this measure aligned with the concept elicitation results. Further studies are needed to assess other psychometric/measurement properties (eg, reliability, responsiveness) of the mNTSS-6-SA within the adult pDPN population to support the measure as fit-for-purpose for clinical outcome assessment in a treatment trial.