Initial development of a patient-reported outcome measure of experience with cognitive impairment associated with schizophrenia

Introduction

Cognitive impairment is a serious, often distressing aspect of schizophrenia. Among the array of symptoms of schizophrenia, cognitive difficulties contribute significantly to the burden of the disorder because they interfere with the ability to manage day-to-day tasks. Cognitive impairment has been shown to be the strongest predictor of functional impairment in schizophrenia, typically predicting more accurately than psychiatric symptoms: a patient’s poorer response to psychosocial interventions, employment status, and social functioning.¹

Currently, no effective pharmacological treatments are available for cognitive impairment in schizophrenia.² Traditional antipsychotic medications have no beneficial effect and may worsen cognitive functioning due to sedation and other adverse effects. The second-generation antipsychotic medications also do not benefit cognitive functioning, and the high anticholinergic effects in several of these medications can significantly impair attention and memory functioning³ and reduce response to cognitive remediation.⁴

Several instruments can be used to assess the components of functional capacity or cognitive performance in schizophrenia. For example, the University of California Performance Skills Assessment (UPSA)⁵ and Specific Levels of Functioning (SLOF) scale^6,7 are measures of functional capacity; the Measurement and Treatment Research to Improve Cognition in Schizophrenia (MATRICS),⁸ Cambridge Automated Neuropsychological Test Battery (CANTAB),⁹ Brief Assessment of Cognition in Schizophrenia (BACS),¹⁰ Repeatable Battery for the Assessment of Neuropsychological Status (RBANS),^11,12 and CogState¹³ batteries can be used to assess cognitive performance. However, none of these instruments were developed based on patient reports of their subjective experiences of and reactions to cognitive impairment.¹⁴

Although several measures of the subjective experience of cognitive impairment in schizophrenia are available, all of them have shortcomings that preclude their use as patient-reported outcome (PRO) measures. For example, the Schizophrenia Cognition Rating Scale (SCoRS)¹⁵ is an interview-based assessment of cognitive functioning that is meant to be used with patients and informants. However, it is not truly a patient-reported measure because the items used in this assessment were either modifications of items from dementia rating scales or developed by researchers based on data on neuropsychological impairment in schizophrenia. There was no patient input into the development of this scale, and none of the SCoRS items included, or were based on, patients’ reports of their own subjective experiences of cognitive impairment. Similarly, the Self-Assessment Scale of Cognitive Complaints in Schizophrenia (SASCCS) was constructed by experts without patients’ input.¹⁶ The Subjective Scale to Investigate Cognition in Schizophrenia (SSTICS)¹⁷ is a brief questionnaire, but its format does not allow for verbal input from the patient; in addition, it focuses primarily on memory and concentration, and internal consistency of subscale scores is not strong. The Subjective Cognitive Impairment Scale (SCIS)¹⁸ was developed with input from patients; however, it has been validated only in patients with first-episode psychosis, and an English version of this scale is not available.

Importantly, the subjective experience of cognitive difficulties may not be correlated directly with the performance on neuropsychological tests.¹⁸ Moreover, performance in real-world settings may be uncorrelated with scores on laboratory measures of neuropsychological functioning, leading to the distinction in the brain injury literature between impairment (reflecting abnormally low test scores) and disability (reflecting difficulty in performing everyday tasks).¹⁹ This dissociation between impairment and disability can also be observed in patients with schizophrenia.¹⁴ Because cognitive impairment associated with schizophrenia is likely to affect a patient’s quality of life, distress, motivation for treatment, and perhaps other outcomes, a mechanism is needed to capture the patient’s experience of cognitive difficulties in a sensitive and reliable way. Providing a truly patient-reported, English-language instrument was the motivation for the development of the PRO measure that is described in this article.

As recommended by the US Food and Drug Administration (FDA) guidance for PRO measures,^20,21 the initial development of the new PRO measure incorporated the existing knowledge, expert input, and patients’ perspectives. Qualitative research methods were used to extract the most common and relevant themes from patient narratives regarding their own experiences of cognitive disability, and these qualitative data were used in creating the final item pool. The resulting draft PRO measure is currently undergoing psychometric testing in a clinical trial as a precursor to broad-based clinical and research use.

Methods

Design

The PRO instrument was developed from multiple sources of information. First, a conceptual model was constructed based on the existing knowledge. Next, two sets of semi-structured qualitative interviews were conducted to elicit the perspectives of people living with schizophrenia: 1) concept elicitation interviews to understand patients’ experiences, leading to the development of draft items and 2) cognitive debriefing interviews, resulting in item revision and reduction. Input from experts in the field was elicited throughout the study.

Conceptual model development

In order to develop the conceptual model, existing literature was reviewed, and expert input regarding neurocognition and schizophrenia was incorporated. A comprehensive literature search in PubMed using Medical Subject Heading terms related to schizophrenia, cognitive impairment, and psychometric testing yielded 121 published articles. Based on a systematic review of the articles, unpublished results from a qualitative study of people with schizophrenia and their family members, and expert input, an initial conceptual model of seven domains contributing to the subjective experience of cognitive impairment associated with schizophrenia was developed (Figure S1).

Concept elicitation interviews

Based on the initial conceptual model, a semi-structured interview guide was developed with open-ended questions to allow for unanticipated concepts to emerge. Discussion topics included the patient’s experience with attention, memory, executive functioning (eg, planning, organization, and flexibility); communication and social cognition; “flat” cognition (ie, lacking sharpness of thought); metacognitive abilities (eg, self-awareness, distinguishing self from others, and application of information); impaired visual and auditory perception; and an “other” category to elicit unanticipated experiences associated with cognitive difficulties. Aligned with the iterative process of qualitative methods, one additional concept (“unmooring of thought from context and one’s surroundings”) was added based on the unpublished findings from a doctoral dissertation completed during the study.²²

During the interviews, special attention was paid to the specific terminology that the respondents used to describe their experiences, respondents’ reports of the frequency of these experiences, conditions under which the experiences typically occurred, and the effect of cognitive difficulties on their day-to-day lives, including emotional reactions. The time duration of interviews ranged from 45 minutes to 2 hours. After each interview, the interviewer completed a field note about the patient’s level of cognitive insight and engagement in the interview to assist with an accurate analysis.

Due to varying levels of cognitive insight among patients and the abstract nature of some of the domains, four probing strategies were used. The interviews were begun by asking patients directly about the concepts in an open-ended manner. As required, particularly for abstract topics, examples from either the interview guide or the patient’s life were used. As a third strategy, scenarios related to particular domains were used, and the patients were asked about how they would respond. If these three strategies were not yielding meaningful responses, we occasionally presented a prescripted scenario, provided two options for how someone could respond, and asked the participant to tell us which response was closest to his/her experience; because this strategy was less open-ended than the others, it was used infrequently.

Item generation

Based on the results from the concept elicitation interviews, initial items were generated to reflect the most common descriptions and language participants used to describe their experiences with cognitive difficulties and the most common conditions under which these experiences occurred. For concepts with variation in common respondent descriptions, multiple items were developed per concept. A third-grade reading level was targeted throughout to accommodate a variety of literacy levels.

Cognitive debriefing interviews

The objectives of the cognitive debriefing phase were to evaluate whether the draft items reflected the patients’ experiences (ie, face validity); to elicit suggestions for other items that could be included to better represent their experiences (ie, content validity); and to assess comprehension of items, instructions, and response options. The time duration of cognitive debriefing interviews ranged from 52 minutes to 2 hours and 16 minutes. Similar to the concept elicitation phase, the interviewers completed a field note after each interview to assist with an accurate analysis.

The interviewer began by asking the participants to complete the draft PRO measure in its entirety. The participants were then asked an open-ended question about their overall impressions of the questionnaire. A series of general questions followed regarding the clarity of the instructions and response options as well as the participants’ perceptions on the length of the questionnaire. Then, the participants were asked several questions about each item in the questionnaire: what they understood it to mean, the ease of answering the item, whether their answers would have changed for a different time frame, how closely the item described their actual experiences, and, for those who indicated that an item did not apply, whether the time frame of the item mattered for this determination. In addition, they were also asked about missing topics related to their experiences with cognition and the degree of bother associated with the categories included in the questionnaire.

Setting and participants

The interviews were conducted in person by staff members at clinical sites serving people with schizophrenia in the Northeastern United States. Four sites conducted the concept elicitation interviews between November 2013 and April 2014, and three of these sites also conducted the cognitive debriefing interviews between July and August 2014. Clinical staff members were chosen to conduct interviews due to their experience in interacting effectively with people with schizophrenia.

Training of the interviewers included practice with open-ended interviewing techniques and nonleading probing, review of the conceptual model, conduct of mock interviews, and practice with multiple strategies for engaging patients in discussion about cognitive difficulties. Throughout the study, the interviewers across all the study sites participated in quality assurance sessions to receive feedback and facilitate consistency in interviewing styles.

People with schizophrenia were recruited to achieve a sample with the demographic and clinical characteristics typical of patients receiving treatment for schizophrenia. Clinical sites recruited patients from clinic databases and community events. Patients participating in a clinical trial within 30 days prior to screening were not eligible. Potential participants were screened to achieve a sample including both older and younger people (with at least 10% aged 18–25 years and at least 10% aged 40–55 years), both genders (with at least 30% women), and both white and nonwhite racial groups (with at least 25% nonwhite). Patients of Hispanic ethnicity were included in either racial group.

Eligible patients were clinically stable for at least 8 weeks with mild or moderate disease severity, defined by Positive and Negative Syndrome Scale item scores <5 for hallucinations, delusions, and conceptual disorganization as well as scale scores <21 for the negative syndrome total score. Patients with mild or moderate comorbid depression (assessed by using the Calgary Depression Scale) were considered for inclusion, although efforts were made to limit the extent of comorbid psychiatric illnesses, particularly depression, in the study sample. Eligible patients were maintained on stable doses of antipsychotic and concomitant psychotropic medications for at least 6 weeks and on a current dose for at least 2 weeks. This study was approved by the New England Research Institutes Institutional Review Board, and written informed consent was obtained from all of the study participants.

Analytic approach

All the interviews were audio-recorded and transcribed verbatim. ATLAS.ti qualitative analysis software (Scientific Software Development GmbH, Berlin, Germany) was used to facilitate data management and analysis.

For both rounds of interviews, two analysts used consensus-based coding. In order to develop codebooks, the analysts independently conducted line-by-line open coding²³ on an initial set of transcripts, met to determine an initial code list and definitions, and entered these into ATLAS.ti to facilitate consistency. In order to verify the code lists, they independently coded additional sets of transcripts (two sets for concept elicitation interviews and one for cognitive debriefing interviews) and met to compare coding and reach a consensus about additional codes and modified definitions.

Coding also followed a similar approach for both rounds of interviews. For the concept elicitation interviews, the analysts independently coded three sets of five transcripts and met after each to resolve discrepancies, at which point inter-rater reliability for the most recently coded transcript exceeded 90%. Consensus coding continued with one analyst applying the codebook to subsequent sets of transcripts and the second analyst reviewing the full coding; again, the analysts reviewed discrepancies. This process continued for three additional sets of five transcripts, at which point inter-rater reliability for the most recently coded set was 89%. For the remaining transcripts, one analyst applied the code list and reviewed the areas that were unclear or may have required a new code with the second analyst.

During concept elicitation, saturation was assessed by using sets of five transcripts. Saturation of concepts²⁴ was considered achieved when no new concepts arose within the previous two sets of transcripts (ie, 10 transcripts). Throughout the coding process, the analysts wrote memos to capture emerging connections between categories. Thorough analysis of quotations identified the most common ways in which the participants described their experiences with cognitive difficulties and conditions under which the difficulties occurred. Results from this process formed the basis of the initial PRO item pool.

For the more straightforward cognitive debriefing interviews, coding proceeded with one analyst applying the codebook, the second analyst reviewing the full coding, and both the analysts discussing to reach a consensus. The analysis entailed calculating frequencies of responses and categorizing quotations to assess face validity and comprehension of each item and overall content validity. Particular attention was paid to identifying ways to revise items to more closely reflect the participants’ experiences.

Throughout the analysis and interpretation of both rounds of interviews, the analysts implemented procedures to follow Green and Thorogood’s general principles for enhancing rigor and credibility: transparency of method, maximization of validity (including attention to deviant cases), maximization of reliability (including frequency counts of themes), constant comparison within the data set and within a case, and a reflexive approach to analysis.²⁵

Item reduction

In order to begin the iterative process of item reduction, response thresholds to indicate adequate comprehension and ease of response were preselected. For categories expected to apply to all people to some degree (ie, difficulties with memory, communication, control, understanding others, planning, handling problems, and attention), items with <90% endorsement of comprehension were omitted from further consideration. For categories in which some people may not experience difficulties (ie, sharp thinking and remaining connected with one’s surroundings), items with <80% endorsement as understandable were omitted. In addition, items with <90% of respondents reporting that the item was “easy” or “very easy” to answer were also omitted. Items that met or exceeded the response thresholds received further discussion to address any difficulties in interpretation that the respondents reported; if an adequate revision was identified, the item was modified and retained.

Results

Concept elicitation

In total, 80 people with schizophrenia participated in the concept elicitation interviews (Table 1). This relatively large sample for a qualitative study was recruited to ensure that saturation of relevant concepts was achieved,²⁴ particularly given anticipated variations in subjective experiences of cognition related to an illness with heterogeneous symptoms and potential communication difficulties. Overall, the results supported the domains of the original conceptual model but yielded substantive changes in the concepts within domains and the terminology used in order to more closely reflect the respondents’ experiences (Figure 1).

Table 1 Participant characteristics

Figure 1 Revised conceptual model of CIAS based on concept elicitation interviews. Note: Two general categories (“Memory-general” and “Sharpness of Thought” – unspecified) are not represented as distinct concepts. Abbreviation: CIAS, cognitive impairment associated with schizophrenia.

The initial item pool included the full range of reported difficulties with cognition that demonstrated three criteria: 1) emerged with some consistency, indicating that it is a common experience for people with schizophrenia; 2) addressed difficulties that respondents were aware of (a prerequisite for accurate reporting by patients) given varying levels of insights into cognitive impairment among people with schizophrenia; and 3) were central to the overall concept of interest. Based on the analysis of interviews and these criteria, one domain and six concepts were omitted from the development of items.

The domain Intermittent Impaired Perception (including perceptual difficulties with visual, auditory, olfactory, time, and body) was omitted owing to the overall low frequency of participants reporting difficulties, as well as concerns by some members of the development team that this domain may be perceived by patients as overlapping with positive symptoms of schizophrenia and did not fit conceptually with the other domains. The concept of “planned movement/motor skills” (originally in the domain of Impaired Perception and later moved to Executive Functioning) was also omitted owing to lack of centrality to the overall concept of interest.

The concepts of “intrusive thoughts” (domain: Attention), “unspecified descriptions of lack of sharpness of thought” (domain: Sharpness of Thought), and difficulties with “self-awareness/self-reflectivity” (domain: Metacognitive Abilities) were omitted because an insufficient number of respondents reported difficulties in these areas. Within the domain of Executive Functioning, “abstract thinking” was omitted because most respondents with difficulties in this area were not aware of them and, therefore, would not be able to report accurately about them. Within the domain of Metacognitive Abilities, the concept of “mastery/application of information” was omitted because this concept is more appropriately measured through objective testing.

Initial item pool

In total, 26 concepts across six domains were retained for item generation based on the participant descriptions (Table 2). A draft instrument comprising 53 items was developed, including two items to assess the level of bother (refer to “Item Tracking Grid” in Table S1).

Table 2 Initial item pool domains and concepts with illustrative quotations

Response categories were modeled based on the validated Patient-Reported Outcomes Measurement Information System (PROMIS) response options;²⁶ specifically, a 5-point response scale ranging from “not at all” to “very much” was used. Because the concept of interest was the degree of difficulty that patients have with cognition (rather than self-reported functioning), the PROMIS response categories were modified to include the word “hard”; that is, the response categories for most items were presented on a 5-point scale, ranging from “not at all hard” to “very hard.” Items not amenable to this format were constructed with a stem that indicated difficulty (eg, “I had trouble with…”) or to clearly indicate that the experience was unwanted (eg, “I kept thinking about things even when I wanted to let them go”).

Questionnaire instructions were developed to provide respondents with a clear definition of the concept of interest and to distinguish it from the positive symptoms of schizophrenia. The instructions also directed respondents to consider their overall experience with an item, rather than focusing on one particular difficult experience. A recall period of the past week was selected to optimize recall while accounting for the frequency of occurrence of difficulties with cognition associated with schizophrenia (ie, the finding from the concept elicitation interviews that these difficulties do not necessarily occur daily). Expert advisors reviewed the draft instrument to consider whether additional modifications were required based on content validity considerations (refer to “Item Tracking Grid” in Table S1).

Cognitive debriefing

In total, 22 respondents participated in cognitive debriefing interviews (Table 1). As the goal of these interviews was narrower and more straightforward than the goal of the concept elicitation phase, a smaller sample was sufficient to reach saturation.²⁵ The participants addressed their overall impressions of the questionnaire, its length and comprehensiveness, the clarity of the instructions and response categories, and their comprehension of individual items (refer to Table 3 for illustrative quotations).

Table 3 Illustrative quotations for cognitive debriefing results

The majority of participants (13 of 22; 59%) reported a positive overall impression of the questionnaire. The main reasons were that the questions were meaningful and relevant to their experiences and were easy to understand. Eight of 22 participants (36%) reported a neutral or mixed reaction, and one participant (5%) had a more negative response due to the concern that completing the questionnaire would be difficult for people with impaired concentration. The average time for completing the questionnaire was 9.4 minutes (range: 4.6–14.7 minutes). Most participants (19 of 22; 86%) stated that the time required to complete the questionnaire was acceptable, with the remaining (3 of 22; 14%) reporting that it was too long.

Regarding comprehensiveness, most participants (19 of 22; 86%) indicated that there were no missing topics related to their experiences with cognition; the remaining three participants (14%) identified the effects of medication, changes over time, and an ability to explain the impacts of each item as areas where coverage could be more detailed or complete.

As an additional indicator of comprehensiveness, 19 participants rated the completeness of the questionnaire on a 5-point Likert scale from “very complete” to “not at all complete.” Sixteen of these 19 (84%) participants rated the questionnaire coverage as “very complete” or “quite a bit complete.” Of the three participants rating the questionnaire as “somewhat complete” or “a little bit complete,” the explanations provided indicated that some of the questions were “a little hard” to answer, suggesting that their responses reflected their experiences with answering some items rather than an assessment of how well the questionnaire covered their difficulties with thinking.

Overall, a majority of participants (17 of 22; 77%) reported that the instructions were clear and easy to follow. The main reason for lack of clarity involved the instruction to separate difficulties with cognition from delusions, hallucinations, or paranoia, which participants indicated, may be difficult for some. As a result, the instructions were modified to include further explanation and examples to clarify the distinction between cognitive difficulties and positive symptoms.

Among the respondents who addressed the definition of “bother” provided in the instructions (ie, being “annoyed” or “concerned” by their experiences), 95% (20 of 21) of them reported that the definition made sense. When asked about alternative words to explain what being bothered by difficulties with thinking feels like, the only additional term to be endorsed by more than one participant was “uncomfortable,” which was added to the definition.

Response options were considered easy to use by 77% of the participants (17 of 22) commenting on Section 1 (response options: “not at all hard” to “very hard”) and 79% (15 of 19) commenting on Section 2 (response options: “not at all” to “very much”). The other participants suggested minor wording changes (eg, to use the word “difficult” instead of “hard”), preferred a 10-point scale instead of a 5-point scale, or were not clear that the response options were orderable. As a result, instructions were added to indicate that response options ranged “from the lowest to the highest” and to choose the “not at all” option if an item did not apply.

The analysis of responses for individual items resulted in 18 items being omitted (Table 4). The participants demonstrated lack of adequate comprehension of 15 items. These items ranged across nine concepts and five domains and were omitted from further consideration. Three additional items were also omitted: one item was identified by the participants as being redundant (concept: general difficulties with memory), and two items were not well understood by some participants and were redundant with other items that had better comprehension.

Table 4 Number of items by domain, concept, and development phase Notes: aOne item for general memory difficulties was included for cognitive debriefing but omitted from the draft PRO measure due to redundancy, as reported by the respondents. bItems related to the level of bother were added to capture the perceived impact of cognitive difficulties related to schizophrenia. Abbreviation: PRO, patient-reported outcome.

The remaining items exceeded the minimum threshold for comprehension. Minor modifications based on participants’ feedback were made to 22 items to increase the level of understanding (Table 4). Further expert review yielded additional minor clarifications to four items. A total of 35 items were retained for psychometric testing, representing 23 concepts within six domains plus two items about the level of bother (see “Draft PRO” in Table S2).

Discussion

This report describes the development phases of a new PRO measure to assess the subjective experience of cognitive impairment associated with schizophrenia, the Patient-Reported Experience of Cognitive Impairment in Schizophrenia (PRECIS). Existing neurocognitive performance measures assess a patient’s cognitive functioning abilities but have not been developed based on the patients’ experiences and do not capture the extent to which cognitive tasks are hard or bothersome for them. The development of the instrument included the construction of an initial conceptual model based on the existing literature, concept elicitation through in-depth qualitative interviews with patients, item generation based on the most common descriptions and language participants used to describe their experiences, and item reduction resulting from a second set of cognitive debriefing interviews with patients.

The resulting 35-item draft PRO instrument represents 23 concepts from six domains and two items assessing bother. The factor structure and other psychometric properties of the draft PRO measure are being assessed and will be reported in a future article. A final version of the instrument and scoring algorithm will be proposed based on the results of psychometric testing.

Although following FDA guidance for developing PRO measures yielded a study design that elicited patients’ opinions, the study has several limitations. For example, the presence of depression among the participants could have affected their responses during qualitative interviews or the ability of the interviewers to build rapport. Although efforts were made to limit the number of participants with comorbid depression, the frequency of this condition among people with schizophrenia meant that it was not feasible to fully exclude these patients. Similarly, some people with schizophrenia lack insight into cognitive difficulties, making it difficult for them to report on experiences with those difficulties in an interview. As noted previously, trained clinical interviewers used four strategies to adjust to a participant’s level of cognitive insight, and field notes included interviewers’ assessments of each participant’s level of awareness to aid analysts. It is unknown how the participants’ medications may affect cognitive insight, although there is no evidence to suggest that medications decrease awareness, and the stabilizing benefit of medication may increase it. Future studies could stratify the analyses by the presence of comorbid depression and level of cognitive insight in order to further account for these conditions.

Even with these limitations, the cognitive debriefing results suggested that the PRO measure reflects many patients’ experiences. The majority of participants in this phase reported a positive overall impression of the questionnaire because the items were meaningful and relevant to their experiences and were easy to understand. Cognitive impairment associated with schizophrenia typically impacts many areas of patients’ everyday lives, including their education, work lives, and interactions with friends and family. Cognitive difficulties can also affect self-esteem and generate distress among patients. The new PRO measure will provide a patient-based perspective on this clinically important area of schizophrenia. It will complement the existing measures of cognitive functioning by providing a tool to describe patients’ statuses in an open-ended manner and with a language that captures their own experiences.

Given the importance to patients of receiving understandable information about their conditions and treatment choices,²⁷ this new measure could prove valuable in building a therapeutic alliance. Following validation, this PRO measure and the conceptual model on which it is based could serve to define key patient-based endpoints for future clinical trials, along with the development of a new endpoint model incorporating both objective and subjective endpoints for assessing cognitive impairment in schizophrenia.

Acknowledgments

The authors thank Masami Tabata, MA, for data management and assistance with qualitative coding and analysis. Editorial support and formatting assistance for this manuscript were provided by Sarah Brown, BSc (Hons), CMPP of Fishawack Communications Ltd., funded by Boehringer Ingelheim.

Disclosure

This work was supported by Boehringer Ingelheim Pharmaceuticals, Inc. (BIPI). The authors meet criteria for authorship as recommended by the International Committee of Medical Journal Editors and were fully responsible for all the content and editorial decisions. Jeremiah J Trudeau and Michael Sand were employees of BIPI, which owns the PRECIS instrument. Raymond C Rosen and Lisa C Welch were employees of New England Research Institutes (NERI), the company contracted to conduct this study. Steven M Silverstein was a paid consultant for BIPI and NERI during the process of scale development, and David C Henderson received institutional support for this study.

References

1.		Green MF, Kern RS, Braff DL, Mintz J. Neurocognitive deficits and functional outcome in schizophrenia: are we measuring the “right stuff”? Schizophr Bull. 2000;26(1):119–136.
2.		Barch DM. Pharmacological strategies for enhancing cognition in schizophrenia. Curr Top Behav Neurosci. 2010;4:43–96.
3.		Minzenberg MJ, Poole JH, Benton C, Vinogradov S. Association of anticholinergic load with impairment of complex attention and memory in schizophrenia. Am J Psychiatry. 2004;161(1):116–124.
4.		Vinogradov S, Fisher M, Warm H, Holland C, Kirshner MA, Pollock BG. The cognitive cost of anticholinergic burden: decreased response to cognitive training in schizophrenia. Am J Psychiatry. 2009;166(9):1055–1062.
5.		Patterson TL, Goldman S, McKibbin CL, Hughs T, Jeste DV. UCSD Performance-Based Skills Assessment: development of a new measure of everyday functioning for severely mentally ill adults. Schizophr Bull. 2001;27(2):235–245.
6.		Schneider LC, Struening EL. SLOF: a behavioral rating scale for assessing the mentally ill. Soc Work Res Abstr. 1983;19(3):9–21.
7.		Mucci A, Rucci P, Rocca P, et al. The Specific Level of Functioning Scale: construct validity, internal consistency and factor structure in a large Italian sample of people with schizophrenia living in the community. Schizophr Res. 2014;159(1):144–150.
8.		Velligan DI, Fredrick M, Mintz J, et al. The reliability and validity of the MATRICS functional assessment battery. Schizophr Bull. 2014;40(5):1047–1052.
9.		Fray PJ, Robbins TW. CANTAB battery: proposed utility in neurotoxicology. Neurotoxicol Teratol. 1996;18(4):499–504.
10.		Keefe RS, Goldberg TE, Harvey PD, Gold JM, Poe MP, Coughenour L. The Brief Assessment of Cognition in Schizophrenia: reliability, sensitivity, and comparison with a standard neurocognitive battery. Schizophr Res. 2004;68(2–3):283–297.
11.		Randolph C, Tierney MC, Mohr E, Chase TN. The Repeatable Battery for the Assessment of Neuropsychological Status (RBANS): preliminary clinical validity. J Clin Exp Neuropsychol. 1998;20(3):310–319.
12.		Gold JM, Queern C, Iannone VN, Buchanan RW. Repeatable battery for the assessment of neuropsychological status as a screening test in schizophrenia I: sensitivity, reliability, and validity. Am J Psychiatry. 1999;156(12):1944–1950.
13.		Maruff P, Thomas E, Cysique L, et al. Validity of the CogState brief battery: relationship to standardized tests and sensitivity to cognitive impairment in mild traumatic brain injury, schizophrenia, and AIDS dementia complex. Arch Clin Neuropsychol. 2009;24(2):165–178.
14.		Silverstein SM, Hatashita-Wong M, Solak BA, et al. Effectiveness of a two-phase cognitive rehabilitation intervention for severely impaired schizophrenia patients. Psychol Med. 2005;35(6):829–837.
15.		Keefe RS, Poe M, Walker TM, Kang JW, Harvey PD. The Schizophrenia Cognition Rating Scale: an interview-based assessment and its relationship to cognition, real-world functioning, and functional capacity. Am J Psychiatry. 2006;163(3):426–432.
16.		Johnson I, Kebir O, Ben Azouz O, Dellagi L, Rabah Y, Tabbane K. The self-assessment scale of cognitive complaints in schizophrenia: a validation study in Tunisian population. BMC Psychiatry. 2009; 9:66.
17.		Stip E, Caron J, Renaud S, Pampoulova T, Lecomte Y. Exploring cognitive complaints in schizophrenia: the subjective scale to investigate cognition in schizophrenia. Compr Psychiatry. 2003;44(4):331–340.
18.		Chang WC, Chan TC, Chiu SS, et al. Self-perceived cognitive functioning and its relationship with objective performance in first-episode schizophrenia: the Subjective Cognitive Impairment Scale. Compr Psychiatry. 2015;56:42–50.
19.		Wilson BA. Cognitive rehabilitation: how it is and how it might be. J Int Neuropsychol Soc. 1997;3(5):487–496.
20.		US Department of Health and Human Services Food and Drug Administration Center for Drug Evaluation and Research (CDER). Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. Silver Spring, MD: CDER; 2009.
21.		US Department of Health and Human Services Food and Drug Administration Center for Drug Evaluation and Research (CDER). Guidance for Industry and FDA Staff: Qualification Process for Drug Development Tools. Silver Spring, MD: CDER; 2014.
22.		Pienkos ES. The Unmooring of the World: A Qualitative Investigation of Anomalous World Experiences in Schizophrenia [doctoral thesis]. New Brunswick, NJ: Rutgers University; 2014.
23.		Corbin J, Strauss A. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. 3rd ed. Thousand Oaks, CA: Sage; 2008.
24.		Glaser B, Strauss A. The Discovery of Grounded Theory: Strategies for Qualitative Research. New York: Aldine de Gruyter; 1967.
25.		Green J, Thorogood N. Qualitative Methods for Health Research. Thousand Oaks, CA: Sage Publications, Inc.; 2000.
26.		Reeve BB, Hays RD, Bjorner JB, et al. Psychometric evaluation and calibration of health-related quality of life item banks: plans for the Patient-Reported Outcomes Measurement Information System (PROMIS). Med Care. 2007;45(5 Suppl 1):S22-S31.
27.		Powell J, Clarke A. Information in mental health: qualitative study of mental health service users. Health Expect. 2006;9(4):359–365.