The Use of an Iterative Strategy of Cognitive Interview and Expert Consultation to Revise the Quality of Life Scale for Patients with Aplastic Anemia (QLS-AA)

Introduction

Aplastic anaemia (AA) is a rare, life-threatening and heterogeneous disorder. It is defined as pancytopenia with a hypocellular bone marrow in the absence of an abnormal infiltrate or marrow fibrosis.¹ The excessive apoptosis of hematopoietic stem cells caused by cytotoxic T cells and/or lymphokines is the main pathogenesis of acquired AA.² The incidence of AA in the United States and Europe is below 2.5/million, while the incidence of AA in Asia is 2–3 times higher.^3,4 In China, the annual incidence of AA is 7.4/10⁵ population, and has a biphasic distribution, with peaks at 10–25 years and over 60 years.² Anemia, hemorrhage and infection are the initial manifestations of AA, and with the development of the disease, the clinical symptoms are gradually worsen, which may be accompanied by refractory anemia, multi-organ hemorrhage and sepsis, which pose a great threat to their life and health.^3,5

Treatment plans for AA include hematopoietic stem cell transplantation (HSCT), immunosuppressive therapy (IST), androgin-based hematopoietic therapy and new attempts of new drugs for refractory AA.⁶ In recent years, with the development of new drugs, the progress of HSCT technology and the maturity of IST treatment, more and more patients can obtain the best treatment plan, and the survival time of AA patients has been significantly prolonged. Studies have shown that the 5-year survival rate of patients with severe aplastic anemia treated with HSCT or IST can reach more than 80%, while the 9-year survival rate of patients treated with HSCT is as high as 87–89%.⁷ However, aplastic therapy is often accompanied by side effects such as decreased physical function, fatigue, diarrhea, nausea and vomiting, oral ulcer, hair loss, rash and respiratory problems. In addition, patients face a variety of risks such as disease recurrence, secondary clonal disease, infection, graft-versus-host disease (GVHD) and so on.⁸ Therefore, AA has the characteristics of long treatment cycle, repeated disease changes and heavy disease burden, resulting in about 15–35% of AA patients with depression and anxiety,⁹ and even serious consequences such as giving up treatment and suicide.¹⁰ It can be see that as the survival time of AA patients increases, the quality of life of AA patients should no longer be neglected. Reliable assessment tools play a fundamental role in the study of the quality of life level and support strategies of AA patients, but there is no mature and specific assessment tool that can be used at present.

We have been committed to develop the Quality of Life Scale for Patients with Aplastic Anemia (QLS-AA). In the preliminary work, we obtained the item pool and formed the pretest scale through literature study and qualitative interviews.¹¹ The test approaches of the pretest scale include focus groups, cognitive interviewing and expert consultation.^12,13 Cognitive interview is a methodology based on the interdisciplinary field termed cognitive aspects of survey methodology (CASM), initiated in the 1980s, which involves theories and results from studies on the cognitive processes involved in survey responding.¹⁴ Cognitive testing is designed to identify unobservable problems in item comprehension, recall, and other cognitive processes that can be remediated through question rewording, reordering, or more extensive instrument revision. Cognitive interview techniques were developed mainly for face-to-face interviews, and the interviews rely on intensive verbal probing of volunteer participants by a specially trained interviewer.¹⁴ It has been widely used to guide the revision and cultural adjustment of scales, the development of questionnaires and other self-report measures.^15,16 Expert consultation is defined as a method in which some experts with highly specialized technology or experience provide personalized technical assistance, which is a core component of many research projects and can reflect our attention to special issues.¹⁷ The participation of clinical experts can provide efficient information, and expert consultation is adopted when the information is uncertain.¹⁸ However, expert consultation also has shortcomings such as strong subjectivity and empiricism, so it is mainly used to the early stage of decision-making or the soliciting opinions.

We used an iterative strategy of cognitive interview and expert consultation to reduce errors that could affect measurement validity, and included preferences for layout of the measurement.¹³ Our objective was to use the cognitive testing to identify otherwise unobservable problems with item comprehension, recall, and other cognitive processes that can be remediated through question rewording, reordering or more extensive instrument revision. We also hope to use the results of cognitive interview and expert consultation to reduce factors that may affect content validity, including differences in perceptions and patient preferences.

Methods

Participants and Recruitment Procedure

The purposive sampling method and maximum variance strategy were used. The sample size was calculated according to the recommendations of the cognitive interview.¹⁹ The first round of interviews was based on the criterion that each item was evaluated by at least 5 participants with no more than 35 items per evaluation. The second round of interviews usually requires 3–5 participants. We divided the pretest scale into 3 sets in a balanced manner to facilitate completion. AA patients who were admitted to the Department of Hematology of the First Affiliated Hospital of Zhejiang Chinese Medical University from August to October 2021 were selected as interview subjects.

Inclusion Criteria

In line with the diagnostic criteria for aplastic anemia proposed in the Standards for Diagnosis and Efficacy of Hematological Diseases;²⁰ age ≥18 years; the patient is aware of the diagnosis and development of the disease; has basic reading and writing skills and able to independently fill in our questionnaire; interested in participating in this research.

Exclusion Criteria

People with communication disorders; those with severe changes in heart, lung, liver, and kidney functions; those with previous mental disorders. The ethical principles of the Declaration of Helsinki were followed in this study. We actively briefed the participants on the purpose and significance of the study, and they were assured that their responses would be anonymous and their personal information confidential until they signed the informed consent form. Ethics approval was obtained from the Ethics Committee of the First Affiliated Hospital of Zhejiang Chinese Medical University (Ethical approval ID: 2021-KL-055-01).

Formation of the Pretest Scale

QLS-AA is based on the concept category of quality of life proposed by WHO and the theoretical framework of quality of life proposed by Dr. Ferrell.^21,22 Combined with cultural background and clinical environment, our research identified a pretest scale with 107 items. We used an iterative strategy of cognitive interview and expert consultation to reduce errors that could compromise a measures validity (Figure 1).

Figure 1 Schematic diagram of the iterative strategy of cognitive interview and expert consultation.

The Outline of the Interview

Cognitive interviews are based on the cognitive theory proposed by Tourangeau,²³ in which the response process is divided into four components: comprehension (attend to questions and instructions, represent logical form of question, identify question focus, link key terms to relevant concepts); retrieval (generate retrieval strategy and cues, retrieve specific/generic memories, fill in missing details); judgment (assess completeness and relevance of memories, draw inferences based on accessibility.^14,24 Table 1 illustrates domains, example questions and corresponding cognitive interview probes.

Table 1 Domains and Examples of Draft Items and Cognitive Interview Probes

Data Collection

Before the interview, explained the research purpose and privacy protection to the participants in detail, signed the informed consent form and completed the general information questionnaire and QLS-AA subscale. Semi-structured interviews were administered throughout the process of interview, and the whole interview would be recorded. Verbal-probing technique was used to encourage open-ended dialogue.²⁵ Participant responses were documented during the interview using a predefined data collection sheet.

Data Analysis

The content analysis method was used to review the responses of the participants and an independent summary was formed.²⁶ Qualitative data analysis is based on hermeneutic, which was conducted within cognitive interviews framework. In 1808, Friederich proposed a circular approach to hermeneutics whereby the foundations of knowledge and understanding was to find the spirit of the whole through the individual, and to grasp the individual through the whole.²⁷ The hermeneutic circle has become the most resonant idea in hermeneutic theory, which is concerned with the dynamic relationship between the part and the whole.²⁸ The questions were independently classified by 2 researchers, and the questionable items were initially revised through special discussion to ensure validity and reliability. The revised items, especially controversial or questionable ones, would be further discussed and reviewed through expert consultation, and then proceed to the next round of cognitive interviews until all items could be understood by the participants, that is, the data is at a relatively saturated state.

Results

Characteristics of Cognitive Interview Participants and Experts

2 rounds of cognitive interviews and expert consultation have been completed. The first round of CI included 16 participants (1-S1~16), and the second round of CI included 5 participants (2-S17~21). According to the research purpose and the representativeness of the experts, 6 experts (E1-6) with multidisciplinary backgrounds were identified as the axis members of the expert consultation group. The characteristics of the participants are shown in Table 2. More information on the specialist background and self-assessment of these experts is provided in Supplementary 1.

Table 2 Characteristics of Cognitive Interview Participants and Experts

Results of the First Round of Cognitive Interviews

The first round of cognitive interviews involved 16 patients and 107 draft items were tested, and the total interview time was about (28.88±10.18) mins. Item problems identified as “clarity” and “response categories” (vague and overlapping) were the most common types of specific question. In addition, 4 items were reflected by participants as diversified response processes and even the views are contradictory, while the linear problem coding is not conducive to understanding the problems, or even misleading the modification of the items. Therefore, we temporarily categorize it as “other” and carry out a further interpretative analysis in expert consultation along with preliminary revised terms. Table 3 shows examples of the analysis of those items and their preliminary revisions.

Table 3 Examples of Analysis of Item-Specific Problems and Its Revision

Results of the First Round of Expert Consultation

The expert consultation meeting reviewed the preliminary revisions undertaken by the researchers after the first round of cognitive interviews. 6 experts put forward a total of 16 revision opinions, of which 11 were revised, 2 were merged, and 4 were added (Supplementary 2). Subsequently, the 4 items with diversified response process were discussed and analyzed, finally a consensus revision was reached, of which 3 items were revised and 1 item was deleted.

Results of the Second Round of Cognitive Interviews

Items verified by expert consultation enter the second round of cognitive interviews. A total of 5 participants were involved and 64 items were evaluated by cognitive interview. The results showed that 1 item was suggested to modify by one participants, but it was not adopted after discussion by the research group (Table 4). There are no problems with other items in terms of response process.

Table 4 Opinions and Amendments

Results of Second Round of Expert Consultation

Following the second round of expert consultation, there was a consensus amongst all experts that all revisions were appropriate. Assess consensus by inviting experts to comment on all items individually, and particular attention was paid to obtaining comments from experts who were less active. We believe that the data has reached relative saturation and the Pre-QLS-AA scale is ready for further testing.

Discussion

Cognitive interview is helpful to study test response processes, improve survey construction and map cognitive models of complex thought processes.²⁹ As we expected, cognitive testing identified otherwise unobservable problems in item comprehension, recall, and other cognitive processes that could be remediated by question rewording, reordering, or more extensive instrument revision, which largely contributed to the improvement of QLS-AA.

Approaches to qualitative analysis tend to be described in a linear, step by step manner, and cognitive interview is no exception. Classical cognitive interviews provide a unique way to capture participants’ cognitive process, and their structured questioning and linear analysis approach suffice for most studies to test response processes.³⁰ However, classical cognitive interviews are challenged in increasingly diverse cognitive process, contradictory opinions and some sensitive health topics.³¹ Hermeneutics provides important theoretical insights into cognitive interviews, and provides a dynamic, non-linear style of thinking that can take a part-to-whole perspective on complex content and diverse responses to meet the specific needs of similar research.³² This kind of technical assistance is realized in the form of expert consultation. In particular, a key tenet of expert consultation is that the process of analysis is iterative, where we may move back and forth through a range of different ways of thinking about the data, rather than completing each step one after the other. In this work, we describe the application of the integrated cognitive interviews in the Pre-QLS-AA. Under the research context of the complexity of the test content and the diversification of cognitive responses, we propose this iterative analysis method based on hermeneutics and implement it through expert consultation. An interpretive cycle from cognitive interviewing to expert consultation until the item being tested is fully understood and accepted.

Classical coding systems tend to categorize linearly, which has the advantage of being organizational and cognitive interview.³³ But for diverse viewpoints, interpretive analysis methods appear to be more conducive to gaining a deeper understanding of the response process and linking individual items to the goals of the project as a whole.³² We achieved interpretive analysis through expert consultation, not only because expert consultation is one of the inherent methods of pre-testing, but more importantly, experts with rich clinical experience and multidisciplinary backgrounds can provide understanding and analysis from different perspectives.³⁴ This ensures that we make careful changes to every draft item. The results of the second round of cognitive interviews confirmed that all the items of Pre-QLS-AA (revised) can be understood, clear and easy to fill out, laying a foundation for the evaluation of the test efficiency in the next step.

Limitations

There are limitations to this study. First, our research method is a comprehensive exploration, although this exploration is carried out within the framework of cognitive interview, this process lacks standard procedures. Although the revision of Pre-QLS-AA has produced significant positive effects, its generalization and applicable scenarios still need to be verified and discussed. Moreover, the results of our study were reported sequentially within the framework of interpretive study, rather than the general reporting paradigm of qualitative research. How to report the results in a more standardized way and present the advantage of complementary combination of methods is a challenge that has not been fully solved in this study. In addition, our study did not conduct a national survey and interview due to resource limitations, but the impact of regional differences on patients’ quality of life and their cognition should not be ignored, which may make QLS-AA lose some applicability in remote areas.

Reflection

In our study, we encountered the challenge of having a diverse response process for 4 draft items. This may be due to the lack of predictability of cognition that the researchers used the cognitive interview for the first time. In response to this existing dilemma, the research team proposed expert consultation based on interpretive analysis to clarify the relationship between diverse viewpoints and the overall purpose of the project, thereby helping us revise the project. Through reflection on our research, we think that we can provide two inspirations for other researchers: (1) Newcomers to cognitive interviews need to be very well prepared for research, especially to anticipate projects as well as keep an open mind to avoid preconceptions. Timely reflection during interview is also helpful to discover the shortcomings of the researcher himself as a research tool. (2) For the existing diversified response processes or contradictory viewpoints, the methods explored in this study may provide some references.

Conclusion

In this study, we report a revision of the Pre-QLS-AA with an comprehensive cognitive interview method. Our improvement lies in incorporating expert consultation based on interpretive analysis within the framework of cognitive interviews to address the challenges of item revision. The method showed a significant effect on the revision of Pre-QLS-AA in this study, and also provides ideas for the development of cognitive interviewing multimodality. In particular, our research methodology can inform the development of cognitive interviews or newly developed drafts.