The Efficacy, Feasibility And Acceptability Of A Remotely Accessible Use Of CIRCuiTS, A Computerized Cognitive Remediation Therapy Program For Schizophrenia: A Pilot Study

Introduction

Cognitive impairment is considered a core feature of schizophrenia.^1–5 Deficits in several cognitive domains are present at the onset of the disorder and are relatively stable throughout the course of the disease. Cognitive impairment is also considered a prodromal factor of the disease, suggesting that early cognitive remediation intervention may improve the prognosis.⁶ Meta-analytic results demonstrate that, unlike available pharmacological treatments, cognitive remediation in people with schizophrenia has beneficial effects on cognition, which generalize to real-life functioning, especially when associated with other psychological interventions.^7,8 Recently, mechanisms underlying the positive effects of cognitive remediation have been explored.⁹ One of the factors proposed to mediate and facilitate the transfer of the therapeutic gains to real-life functioning is an improvement in metacognition.^10,11 This cognitive domain is defined as the ability to understand own cognitive skills (metacognitive knowledge) and to regulate the application of these skills to real-life tasks (metacognitive regulation or executive control).^12,13 Several studies have demonstrated deficits of metacognition in patients with psychotic disorders, and an association of these deficits with poor real-life functioning.^14,15 In light of these findings, a new cognitive remediation intervention focusing on the development of metacognitive skills has recently been developed: Computerized Interactive Remediation of Cognition – Training for Schizophrenia (CIRCuiTS). CIRCuiTS provides the possibility to administer the program remotely, in order to avoid dropouts and reduce program implementation cost. The patient can practice at home and the therapist can monitor the progress of the CIRCuiTS training sessions by accessing patients’ profile remotely. Acceptability and feasibility of the English version of CIRCuiTS were evaluated in previous investigations.^16,17 Encouraging findings from a randomized controlled trial (RCT) demonstrated that CIRCuiTS improves memory and functioning compared to treatment as usual and that the impact on functional outcome might be mediated by executive control.¹⁸ In this RCT study, the therapists administered CIRCuiTS remotely according to their clinical judgment, therefore no data are available on the acceptability, feasibility and efficacy of a standardized protocol to provide CIRCuiTS remotely.

For these reasons next steps in the development of CIRCuiTS include 1) the validation of a standardized protocol to administer CIRCuiTS remotely and 2) the demonstration of CIRCuiTS feasibility and acceptability in different cultural and linguistic settings in order to conduct further studies to test its efficacy using large-scale multinational randomized controlled trials.

In the light of this background, we 1) translated and adapted the program for the Italian context and 2) implemented a protocol to administer CIRCuiTS from home in order to assess its feasibility, acceptability and effectiveness.

Methods

Design

The first step of the study was the translation and adaptation of the CIRCuiTS Italian version. The translation was performed in collaboration with Spika, the company that built the English version of the program and provided the manual for translation (Circuits 2.0 Translation Manual). During the first phase of the translation procedure, a psychiatrist of our University Department received an accurate CIRCuiTS training at King’s College in London. After about 3 months of training, the psychiatrist, in collaboration with a psychologist and a psychiatric rehabilitation technician, adapted the translation at our University Department. Where possible it consists in a faithful translation, while in other cases it was necessary to adapt the content to the Italian context.

Successively participants were recruited and assessed at baseline (neuropsychological and clinical assessments) and then received up to 40 therapy sessions, three times a week, for about 1 hr. The training lasted for approximately 3 months and participants were reassessed at post-treatment on the same neuropsychological and clinical variables.

In the first 3 weeks, the therapist helped patients use the program to optimize the comprehension of the software. Subsequently, one session per week was independently handled by the patient from their home.

Participants

Participants were recruited from the outpatient unit for Schizophrenia spectrum disorders of our University Department of Mental Health. The inclusion criteria for the study were: 1) age between 18 and 55 years 2) confirmed DSM-IV diagnosis of schizophrenia 3) a stable clinical status, as evaluated with validate scales, that permitted reliable cognitive assessment (patients did not show acute psychotic symptoms that may have affected the neuropsychological testing); and 4) verified cognitive impairment assessed with a neuropsychological test battery. The exclusion criteria were: 1) brain and metabolic disorders known to cause neuropsychological impairments; 2) substance dependence within the last 6 months; and 3) IQ below 70.

This study was approved by the ethics committee of the University of Campania “Luigi Vanvitelli” Hospital and participants provided a written informed consent, after a comprehensive explanation of the study procedures. All study procedures were in accordance with the Declaration of Helsinki.

Materials

Neuropsychological Assessment

Neurocognitive functions were rated using the MATRICS Consensus Cognitive Battery (MCCB) which includes tests assessing different cognitive domains.^19,20 Assessed domains included processing speed, attention/vigilance, working memory, visual learning, verbal learning, reasoning and problem solving and social cognition. Standardized T-scores for each domain and for the neurocognitive composite index were used in the analyses. Cognitive flexibility was evaluated using the Wisconsin Card Sorting Test (WCST), computerized version.²¹ This task assesses abstract reasoning and cognitive flexibility. The WCST performance measures include categories achieved, perseverative errors and non-perseverative errors. In this study, numbers of categories achieved, perseverative errors and total errors were used to assess cognitive flexibility. For perseverative errors and total errors, higher scores indicate worse performance.

Functional Capacity Assessment

Functional capacity was evaluated by the short version of the University of California San Diego (UCSD) Performance-based Skills Assessment Brief (UPSA-B),²² a performance-based instrument that assesses “financial skills” (counting money and paying bills) and “communication skills” (to dial a telephone number for emergency or reschedule an appointment by telephone). The total score, ranging from 0 (worst functional capacity) to 100 (best functional capacity), was used in the analyses.

Self-Esteem Assessment

Self-esteem was evaluated using the Self-Esteem Rating Scale (SERS), a 40-item self-rating scale with scores in the range −120 to 120, with higher scores denoting higher self-esteem.²³

Psychopathological Assessment

The Positive And Negative Syndrome Scale (PANSS) was used to rate symptom severity. For this study, we used the five-factor solutions proposed by Wallwork et al,²⁴ this includes dimensions such as “disorganization” (sum of the scores on Conceptual Disorganization, Poor Attention and Difficulty in Abstraction), “positive symptoms” (sum of the scores on Delusions, Hallucinatory Behavior, Grandiosity, Unusual thought content) and “negative symptoms” (sum of the scores on Blunted affect, Emotional withdrawal, Poor rapport, Passive/apathetic social withdrawal, Lack of spontaneity and flow of conversation, Motor retardation).

Real-Life Functioning Assessment

Real-life functioning was assessed using the Specific Level of Functioning Scale (SLOF), a hybrid instrument that examines several aspects of functioning. This measure is based on key caregiver’s judgment on behavior and functioning of patients and consists of 43 items including six domains: physical efficiency, self-care, interpersonal relationships, social acceptability, community activities (e.g., shopping, using public transportation), and working abilities. According to Sabbag et al²⁵ the “physical efficiency” dimension was not used in this study both because ceiling effects was expected and because it did not assess everyday functioning. The Italian version of the scale was validated as part of the Italian Network for Research on Psychoses activities.²⁶ For this scale, the higher the total score, the better the overall functioning of the subject (total score ranges from 43=poorest functioning to 215=best functioning).

User Satisfaction

To evaluate the participants’ level of satisfaction with the treatment, the Client Satisfaction Questionnaire-8 (CSQ-8) was used. Scores for this measure range between 8 and 32, with a higher score indicating greater satisfaction.²⁷

Intervention

Computerised Interactive Remediation of Cognition Training for Schizophrenia “CIRCuiTS” is a computerized cognitive therapy program for people with a diagnosis of schizophrenia or schizoaffective disorder.^17,28 It is a psychological treatment aimed at improving cognitive functioning and metacognitive skills. It does not have the purpose of directly addressing the content of thought, beliefs or emotions but, through the use of non-emotional material, teaches basic thinking skills that could facilitate more complex social behaviors and life skills. CIRCuiTS uses a variety of techniques, such as errorless learning and scaffolding, which, according to empirical studies, contribute to improving cognitive performance.²⁸ CIRCuiTS aims to enhance cognitive abilities including long-term memory; working memory, attention, problem-solving, speed of processing, comprehension, flexibility of thought, and planning. Specifically, CIRCuiTS targets metacognitive difficulties by helping clients to improve their metacognitive knowledge (knowledge of their cognitive skills) and metacognitive regulation (the ability to monitor and control the application of cognitive skills). Metacognitive difficulties are explicitly targeted through specific software and therapy components. Metacognitive regulation is encouraged by asking the participant to develop a plan, to estimate the difficulty before beginning a task than monitor and finally review task performance. In particular, before each task, the participant rates the difficulty of the task, estimates how long the task will take and identifies strategies to complete the task. After the task, the participant sees his/her scores and rates how difficult he/she found the task and how useful he/she found the selected strategy. CIRCuiTS allows both to exercise in a safe environment to improve neurocognitive skills and to increase the awareness of own cognitive abilities and finally to structure useful strategies to transfer these skills from laboratory to the real world.

CIRCuiTS includes a maximum of 40 sessions, with a frequency of 3 sessions per week. Each session lasts about 1 hr; however, as participant concentration may be limited, due to attentional impairment, the session duration can be shorter for the initial sessions and gradually increased over sessions, to match the subject’s improvement in attention. Each session includes a series of tasks (usually about 4 to 8) that target different cognitive domains. Each task may concern verbal or nonverbal skills or both. The workflow can be modified, and tasks can be repeated or skipped to individualize therapy to the subjects’ needs.

CIRCuiTS provides: 1) Abstract tasks that use abstract material and are designed to improve some cognitive functions in an abstract context (for example, numbers or geometrical shapes); and 2) complex tasks designed to be more “ecological”, i.e., similar to activities that could be carried out in everyday life (e.g. planning a trip or a meal, reading a letter, or recalling a shopping list). There are up to 15 difficulty levels for each task; the program presents first the lowest and gradually progresses to the highest levels, depending on their performance CIRCuiTS automatically regulates the level of difficulty for individual users using a built-in algorithm. This is done to maintain performance positive feedback high. The program is based on the rationale that neurocognitive abilities learned in an abstract context can be gradually incorporated into real-life activities through the training of metacognitive skills and the development of new behavioral strategies. The ultimate goal is that the trained subjects gradually transfer the acquired cognitive skills to daily life.

Sessions are supported by a therapist whose role is to support the implementation of the program components and to support the clients' motivation.

Home Sessions

In our study, the CIRCuiTS program included an initial period (3 weeks) in which the therapist and patient met 3 times a week for 1 hr. After 3 weeks (period of time in which the patient could become able to use the program autonomously) the patient met the therapist in two weekly sessions and in one session the patient carried out CIRCuiTS independently from the home computer. The duration of the home session is similar to the session length received at the hospital. The therapist can monitor the client’s progress on the software by accessing the subject’s user profile remotely. The patient is required to note any difficulties encountered in performing the exercises and discuss them in the next meeting with the therapist. The ability to carry out tasks autonomously is designed both to facilitate access to training for patients who have greater difficulty in reaching the clinic, to promote autonomy and to reduce the program implementation cost. The rationale adopted to promote this method (1/3 session from home and not 2/3 sessions from home) is due to: 1) the importance of the therapist–patient relationship to promote new cognitive strategies and their generalization²⁹ 2) the importance of the therapist–patient relationship to promote and maintain the patient’s motivation.^29–31

Procedure

Potential participants were identified via the outpatient unit for Schizophrenia spectrum disorders of our University Department of Mental Health. If the inclusion criteria were met, and the participants signed the informed consent, a psychiatrist and a psychologist completed neuropsychological and clinical baseline evaluations. Two psychiatric rehabilitation technicians conducted the therapy for 4 patients each. Psychiatric rehabilitation technicians were trained by a psychiatrist from our research group, expert in cognitive remediation, who was trained at King’s College in London for 2 months and was involved in the Italian translation and adaptation of the program. Participants maintained their usual pharmacological treatment regime (e.g., antipsychotic medication) throughout the study. After the therapy sessions were completed, the same baseline evaluator administered the post-treatment neuropsychological evaluation and clinical assessment.

Statistical Analysis

The proportion of recruited service users who agreed to participate in the study, together with attendance and attrition rates, was recorded and represented the indicators of intervention feasibility.

Acceptability was assessed through the user satisfaction questionnaire. Measures of cognition, self-esteem, psychopathology, functional capacity and real-life functioning was assessed to evaluate the effectiveness of the program.

For the MCCB cognitive indices (speed of processing, attention/vigilance, working memory, verbal learning, visual learning, reasoning and problem solving, social cognition and neurocognitive composite score) T-scores were calculated by the MCCB Computer Scoring Program with correction of the raw scores by age and gender based on the normative Italian data³² Using this standardization method, all test scores, domain scores, and the composite scores are standardized to the same measurement scale with a mean of 50 and SD of 10.

The numbers of categories achieved, total errors and perseverative errors on the Wisconsin Card Sorting test were obtained by the E.M.S. software Test System v1.0.12 produced by the E.M.S. srl. in Bologna, Italy.³³ Scores of functional capacity, self-esteem and real-life functioning were obtained as the sum of their respective items.

As to psychopathological indices the PANSS scores for positive, negative and disorganization dimensions were calculated from the sum of the individual items included in the respective dimension by the consensus factor analysis by Wallwork et al.²⁴

Student’s t-test for the same sample was used to test pre-post treatment changes.

If the efficacy of the program will be confirmed, a correlation analysis will be conducted in order to assess which variables (cognition, self-esteem, symptomatology) are correlated with the improvement of functioning (functional capacity and real-life functioning). First, only variables with a significant post-intervention change will be analyzed. Second, it will be analyzed the amount of change of each variable between the pre and post (with a subtraction: post-pre). Finally, the changes in each variable will be correlated with changes in real-life functioning domains and functional capacity.

For all statistical tests, the level of significance was set at p<0.05. All the analyses have been conducted using STATISTICA version 10.

Effect size (Cohen’s d) for post- vs pre-training score changes was calculated as the mean difference between the pre-treatment and post-treatment scores, divided by the standard deviation, taking into account the association between pre and post-treatment scores.³⁴

Results

Ten participants affected by schizophrenia were found to be eligible for the study, eight (80%) consented to participate. Two out of 8 had never participated in any rehabilitation intervention. The remaining 6 had participated in different types of rehabilitation interventions (social skills training, other cognitive remediation interventions, psychoeducation). These other interventions were completed at least 18 months before starting the present study. The baseline characteristics and medication status of the sample are presented in Table 1. Participants attended an average of 37.5 sessions to complete the program (min 29, max 40). Of these, 30.3% were carried out from home (91 out of 300 total sessions). At baseline, participants performed on average 1 ½ to 3 ½ SD below the normative mean (Table 1), indicating moderate to large impairment on MCCB domains. The mean number of total errors and perseverative errors on WCST were in the range of moderate impairment (about 1 SD above the normative mean).

Table 1 Baseline Characteristics And Medication Status (N=8)

Feasibility

In total, 10 patients were assessed as eligible and were offered the therapy. Eight consented to participate (80%). Two patients refused to participate due to the distance between their home and the Department of Mental Health. All the remaining 8 subjects completed the intervention program. Figure 1 illustrates the recruitment process, therapy completion and dropouts.

Figure 1 Consort diagram.

Acceptability

The participants’ average score reported for CIRCuiTS on the CSQ was 29.3 indicating a high level of satisfaction according to Kelly et al 2018.³⁵

Effectiveness

Neurocognitive Indices

Compared to baseline, post-treatment scores were significantly improved for the MCCB Neurocognitive composite (t(7)=−4.35, p<0.01), as well as for the following domains: speed of processing (t(7)=−5.27, p<0.01), working memory (t(7)=−3.03, p<0.02), verbal learning (t(7)=−2.46, p<0.05) and visual learning (t(7)=−2.37, p<0.05) (Figure 2). Post-treatment total errors (t(7)=2.85, p<0.05) and perseverative errors (t(7)=2.53, p<0.05) on the WCST were also significantly reduced with respect to baseline, while the number of categories achieved did not show a statistically significant change (Figure 3).

Figure 2 Treatment effects on MCCB domains. Note: *Significant improvement post-treatment on MCCB domains (p<0.05).Abbreviations: MCCB, MATRICS consensus congnitive battery; SoP, speed of processing; A/V, attention/vigilance; WM, working memory; Verb L, verbal learning; Vis L, visual learning; Reas & PS, reasoning and problem solving; Soc Cog, social cognition; Neuricog Com, neurocognitive composite score.

Figure 3 Treatment effects on Wisconsin Card Sorting Test. Notes: *Significant improvement post-treatment (p<0.05). The y-axis shows the WCST-score.

The confidence interval and the effect size of these changes in neurocognitive scores are reported in Table 2. The MCCB neurocognitive pre and post-treatment scores of each subject are presented in Table 3. All but one subject (Case 7) improved by at least 0.5 to 1 SD in at least 2 cognitive domains. In four subjects there was a change in severity level (from moderate-severe to absent-mild) in 1 or more domains.

Table 2 Pre-Post Treatment Effect Size On Neurocognitive Domains

Table 3 T-Score For Each MCCB Domain At Baseline And Post-Treatment. Values Shown In Bold And Red Show Change In Severity Level (i.e., From Moderate-Severe Cognitive Impairment, 1.5 To 3.5 SD Below The Normative Mean, To Mild-Absent, From 0 To 1 SD Below The Normative Mean). Values In Bold And Black Show Improvements Of At Least 0.5–1 SD Vs Baseline Without Change In Severity Level

Functional Capacity

The UPSA-B total score showed a significant improvement at the end of the cognitive remediation program, with respect to baseline (t(7)=−7.65, p<0.01; effect size=2.72) (Table 3).

Self-Esteem

A significant improvement of the self-esteem, evaluated by SERS, was also found (t(7)=−3.1, p<0.02; effect size=1.13) (Table 4).

Table 4 Treatment Effects On Functional Capacity, Self-Esteem And Symptomatology

Psychopathological Indices

A significant improvement was observed on the PANSS disorganization dimension after the intervention (t(7)=3.52, p<0.01; effect size=1.88). No significant effect was found for the other PANSS dimensions, although there was a mild reduction of the negative symptom dimension (Table 4).

Real Life Functioning

After the end of the intervention, a significant improvement of real-life functioning, as assessed by the SLOF total score, was observed (t(7)=−3.54; p<0.01). Considering the change in each SLOF subscale, significant improvements of “self-care” (t(7)=−3.66; p<0.01), “interpersonal relationships” (t=−3.32; p<0.02), “social acceptability” (t=−3.64; p<0.01) and “working abilities” (t(7)=−4.10; p<0.01) were found (Table 5).

Table 5 Treatment Effects On Real-Life Functioning

Correlation Analysis Between Changes In Functioning, Functional Capacity, Cognition, Self-Esteem And Disorganization

The correlation analysis showed no correlation between the SLOF total or domain score improvement and any other significantly improved variable. For the correlation between UPSA-b total score and the other variables, only the MCCB composite showed a significant correlation (r=0.709, p<0.05, Table 6).

Table 6 Correlational Analysis Between Variables Showing Significant Improvement Post-Treatment

Discussion

This study aimed to implement the CIRCuiTS Italian version and to assess the acceptability, feasibility and efficacy of a standardized method to administer the program remotely. The results showed that the Italian version of the program administered according to our protocol was feasible and acceptable to service users. It suggests that the remote use may have important implications in terms of access to cognitive rehabilitation for patients with schizophrenia. Of the 10 eligible patients, 8 (80%) agreed to attend and all of them completed the treatment. It is difficult to compare this feasibility study to large-scale randomized trials present in the literature, but the absence of dropouts is an encouraging result. The two patients who refused the therapy attributed the decision to the cost of transportation from their home to the University Department, for the scheduled 2–3 weekly sessions.

Regarding acceptability, the participants evaluated the therapy as useful and satisfactory. The average CSQ score is “High” according to the table proposed by Kelly et al.³⁵ These results are in line with previous findings from a consumer-led survey on the experience of CIRCuiTS.^17,28

Our study is the first study assessing CIRCuiTS efficacy using the MCCB neurocognitive battery to analyze the results on cognitive domains. The MATRICS initiative was launched to foster the development of a cognitive consensus battery for use in clinical trials.^19,20 The MCCB battery has been translated into several languages, showing a high reliability and validity of the construct.^19,36–39 MCCB data show that CIRCuiTS participants improve significantly post-intervention on the neurocognitive composite score and on 4 on the 7 cognitive domains considered (Speed of Processing, Working Memory, Visual Learning and Verbal Learning). The rest of the cognitive domains (Attention/Vigilance, Reasoning & Problem Solving) did not show significant improvements but, with the exception of Social Cognition, there is a general improvement. It is very likely that the lack of statistical significance for Attention/Vigilance and Reasoning & Problem Solving is due to the better baseline performance of our patients on these compared to the other domains. The lack of changes in Social Cognition is most likely due to the absence of a targeted training for this cognitive domain in CIRCuiTS.

A general positive effect was found also for the cognitive flexibility measured by WCST perseverative errors and total errors. Our results are in line with data on the CRT effectiveness present in the literature. Indeed, CRT has shown a significant effect on the various neurocognitive domains as reported by several meta-analyses^7,8 and literature data on the effectiveness and feasibility of CIRCuiTS. In fact, in a recent study¹⁶ researchers reported a moderate effect of CIRCuiTS short version (the protocol included a total of 6 sessions) on global cognition and memory in a sample of inpatients with acute psychosis. In anotherCIRCuiTS feasibility study,¹⁷ on a small sample of young adults affected by schizophrenia, emerged a positive effect of CIRCuiTS on Attention, Problem Solving and Episodic Memory. Furthermore, in the recent randomized controlled trial (CIRCuiTS vs Treatment as Usual), the participants showed a significant improvement in visual memory, maintained at 3-month follow-up with the program. In addition, CIRCuiTS was found to protect from cognitive decline.¹⁸

Our study showed a generalization of the effects also to other variables: i.e. functional capacity, self-esteem, disorganization and real-life functioning.

Interestingly, our results showed no correlation between SLOF improvement and changes in any other variables improved post-treatment. Vice versa, MCCB composite score improvement correlated with the improvement of functional capacity. These results are in line with the hypothesis that the neurocognitive skills are directly associated with functional capacity, and functional capacity could be a bridge between cognition and real-life functioning.^5,40 In line with this model, CIRCuiTS could indirectly promote the improvement on the real-life functioning of the patient affected by schizophrenia.

Finally, these positive results suggest that the Italian translation of CIRCuiTS was faithful to the original and the autonomous use of the program (1 session out of 3) is feasible, acceptable and effective.

Limitations

There are a number of limitations to this pilot study. Our results are limited to the restricted number of subjects and there was no control condition. It would be necessary in the future to evaluate the effectiveness of the intervention by comparing in a randomized controlled trial the experimental group with a control group and a treatment as usual group.

A future study should also include a follow-up evaluation in order to assess the duration of treatment effects.

Conclusion

This is the first study in which the efficacy of a standardized protocol to administer the Italian version of CIRCuiTS remotely was assessed. The participants, after an accurate therapist’s training, were able to handle the program independently from home. The results showed that this protocol was feasible and well accepted by service users and confirmed the data obtained in other studies on the efficacy of the program. The autonomous use of CIRCuiTS (1/3 session) does not worsen the effectiveness of training but can reduce dropouts and implementation costs while increasing the possibility of access to treatment. Furthermore, in our pilot-study CIRCuiTS is effective also in patients with a long illness duration, who have previously been involved in other psychological treatments. A large-scale, controlled trial is needed to confirm the data of this preliminary study and to identify the patients’ profiles that are more likely to benefit from CIRCuiTS intervention.