Validation of the THINC-It Tool for Assessment of Cognitive Impairment in Patients with Bipolar Depression

Introduction

Bipolar depression (BD-D) is a major depressive episode of bipolar disorder that greatly distresses patients. Cognitive impairment is a critical feature of BD-D and one of the major causes of patient function impairment. Cognitive impairment is significantly heterogeneous in patients with BD-D. From the previous studies,^1,2 the proportion of BD-D patients with multi-domain cognitive impairment, selective cognitive impairment (attention/alertness and psychomotor speed), and without significant cognitive impairment ranged from 12% to 40%, 29% to 40%, and 32% to 48%, respectively. The most frequently affected cognitive domains include attention/alertness, memory, and executive function, regardless of whether the depression is an acute episode or stable.^3–5 Cognitive impairment is associated with decreased functional capacity and poor prognosis⁶ and poses significant harm to individuals with BD-D and a heavy burden to families and society. Therefore, screening tools need to be developed to identify cognitive impairment in patients with BD-D.

Regular screening can dynamically assess the clinical efficacy of the intervention on cognitive function and further improve treatment compliance.⁷ For these reasons, cognitive assessment has attracted increasing attention from multiple scientific disciplines. However, simple and effective standardized screening tools for assessing cognitive impairment are lacking in BD-D. Several subjective and objective cognitive assessment tools have been developed and assessed the amount of cognitive effort associated with the performance of a given task in clinical practice. Currently, the commonly used subjective cognitive assessment tools including the Perceived Deficits Questionnaire (PDQ-D)⁸ and the Cognitive Complaints in Bipolar Disorder Rating Assessment (COBRA),⁹ Along with the objective cognitive assessment tools (ie, the Wiscons in card sorting test, the Stroop Test, the Trail Making Test, the Go/NoGo and the N-Back Test), mainly assess the cognitive domains of attention/alertness, processing speed, executive function, memory, and language. The above single tests are simple to operate but only assess a certain dimension of cognitive impairment and need to be used in combination with multiple tests in the clinic. The cognitive battery tools, including the Cambridge Neuropsychological Test Automatic Battery (CANTAB), the MATRICS Consensus Cognitive Battery (MCCB), the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS), and The Cogstate Brief Battery (CBB), have been assessed multiple cognitive domains and are being extensively investigated in a number of preclinical studies for various diseases. Although they are easy to operate, they also have some disadvantages, like, eg, the RBANS tool is simple to conduct and takes a short time, but was initially designed for mild to severe dementia, not very suitable for people with mood disorders. The CBB, MCCB, and CANTAB provide sensitive and accurate test results, but they are complex, time-consuming, and expensive, making their application in clinical practice difficult. Therefore, developing a simple, free, accessible, and effective screening tool like THINC-it to assess cognitive function in patients with BD-D has become an essential issue for mental health practitioners.

The THINC-it tool¹⁰ is a free, simple, and easy-to-use cognitive screening tool, which has already been validated in patients with major depressive disorder by Dr. Mclntyre and Lam¹¹ and applied in a clinical setting. The THINC-it tool is the first cognitive tool to combine subjective and objective cognitive assessment. It contains five subscales, such as Codebreaker, Trials, Spotter, SymbolCheck and PDQ-5-D in the battery.¹² The entire assessment only takes approximately 10–15 minutes and can therefore be completed by patients while waiting for their doctors. Hannah¹³ considered the THINC-it tool advantageous in busy clinics with limited staff since it can reduce unnecessary expenditures on medical resources. There is evidence that the THINC-it tool can assess cognitive impairment in patients with major depression with high sensitivity^14,15 and applies to the Chinese population.¹⁶ Our previous study has reported on the clinical application of the tool in patients with BD-D in the Chinese population.¹⁷ However, its conclusions were not very convincing owing to only 58 cases that had been included in the data analysis. Thus, considering the inadequate sample sizes of previous studies, the primary aim of this study was that further validation of the THINC-it was achieved by recruiting more patients with bipolar depression and refining research designs and processes.

Method

Sample

The study participants comprised 120 patients with BD-D, and 100 healthy individuals (HC), all of Han Chinese origin (Table 1). The study procedures were conducted in the outpatient and inpatient psychiatry clinics of the Shanghai Mental Health Center and the Pudong New Area Mental Health Center, China. BD-D patients were diagnosed based on the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-V).¹⁸ On the other hand, the healthy controls did not meet the diagnosis of any mental disorder according to the DSM-V diagnostic criteria and did not have any history of neurological disorders, alcohol dependency, or family history of mental disorders among their first-degree relatives. All patients were recruited for participation in our former clinical trial (Clinical Trail Registry ID: NCT04471454), but the data used here were acquired at enrollment. Patients were required to meet the following criteria: (a) Han Chinese, aged between 18 and 65. (b) Only patients on Selective Serotonin Reuptake Inhibitor (SSRI) drugs, antipsychotics with olanzapine

Table 1 Comparison of General Information

/quetiapine, and mood stabilizer treatment for at least two weeks were included. (c) Patients with relatively stable symptoms after a period of treatment, with HAMD scores< 17. The exclusion criteria included: patients on treatment that seriously affects cognition, patients who had consumed alcohol 24 hours and Benzodiazepine 12 hours prior to the cognitive testing, and patients who had been treated with electroconvulsive therapy (ECT) in the 6 months preceding the study.

Clinical Assessment

We used five classical neurocognitive tests, including the THINC-it tool, and five standard cognitive tests, consisting of the Reaction Time (RTI), 1-back, the Digit Symbol Substitution Test (DSST), the Trail Making Test B (TMT-B), and PDQ-5-D (paper-and-pencil). The domains assessed including attention/alertness, processing speed, executive function, and working memory (Table 2). Seventeen items of the Hamilton Depression Rating Scale (HAMD-17) were used to assess the severity of depression with the delimitation score as follows: total score >24 for severe depression, >17 for mild or moderate depression, and <7 for no depression symptoms.

Table 2 Cognitive Function Dimensions and Measurement Indicators Reflected by Various Neuropsychological Tests

The young mania rating scale (YMRS) was used to assess symptoms of mania. This is an 11-item clinician-administered scale. Patients with a YMRS score≤5 were considered to have no manic episodes.

Procedure

The diagnosis of each patient was based on consensus by specialists in psychiatry according to the DSM-V criteria for BD-D. At the beginning of the study, sociodemographic data were collected from the study participants and psychopathology was rated using HAMD-17 and YMRS. After that, the participants were required to complete the THINC-it tool and five standard cognitive tests. All behavioral tests were conducted in a quiet environment. Forty-eight patients with BD-D were randomly selected and re-assessed after one week using the THINC-it tool to verify the test-retest reliability (Figure 1).

Figure 1 The flowchart of study participation. Abbreviations: BD-D, Bipolar depression; HAMD, Hamilton Depression Scale; YMRS, Young Mania Rating Scale.

The protocol for this research was approved by the Research Ethics Committee of the Clinical Research Center, Shanghai Mental Health Center, Shanghai Jiao Tong University School of Medicine (No:2020–03), the Shanghai Pudong New Area Mental Health Center and Tongji University School of Medicine (No: PDJWLL2019010). All subjects voluntarily signed a written informed consent before study inclusion, and the study was conducted according to the guidelines of the Helsinki Declaration.

Statistical Analysis

The sample size was estimated to be at least 5~10 times the number of variables using Kendall. The THINC-it tool was developed and presented results from the statistical outcomes of the tool. The raw scores of PDQ-5-D, Spotter, and Trails were positively correlated with the severity of cognitive impairment. To account for this reversal, all raw cores of PDQ-5-D, Spotter, and Trails must multiply it by −1 to switch the sign (ie, if it’s negative, make it positive, and vice versa). For calculating of the total THINC-it composite score, each of the THINC-it tasks was assigned a weight of 0.20. Accordingly, a higher score indicated better cognitive function.

SPSS 22.0 was used for statistical analysis. Data for the questionnaire items were transformed using an integrated z-score according to a method described by Mclntyre.¹⁹

Cronbach’s α coefficient was used to calculate the internal consistency reliability, while Pearson correlation analysis was used to calculate external consistency reliability. Exploratory factor analysis (EFA) was used to calculate the rationality of the Construct validity.

After the standard score (Z score) for each index of the THINC-it tool was transformed, discriminative validity was used to calculate the difference in cognitive impairment between the BD-D and the healthy control group using the rank-sum test and an analysis of covariance (ANCOVA) test.

Parallel validity using Pearson correlation analysis was used to analyze the correlation between the THINC-it tool and standardized test scores.

P-value <0.05 (two sides) was considered statistically significant.

Results

A total of 120 BD-D patients and 100 healthy controls were included in the analysis. There were significant differences in the level of education (t=−3.831, P<0.001), marital status (t=36.546, P<0.001), and occupation (χ²=46.756, P<0.001) between the healthy and BD-D groups. The level of education was lower in the BD-D group compared to the healthy control group (Table 1).

Internal Consistency

The five subtests of the THINC-it tool had a high internal consistency with a Cronbach’s α of 0.815. However, Cronbach’s alpha coefficient of the standard test was 0.688.

Test-Retest Reliability

Forty-eight patients with BD-D were followed up for one week for the THINC-it test-retest using the intra-group correlation coefficient (ICC). The results showed that the ICC ranged from 0.571 to 0.854 (P<0.001) (Table 3).

Table 3 Test-Retest Reliability of the THINC-It Tool

Construct Validity

Exploratory Factor Analysis (EFA) was performed in the current study to evaluate the construct validity. The results showed that the KMO value was 0.749, and Bartlett’s Sphericity test, χ² (10), was 198.257 (P<0.001). The factor loading coefficients of Spotter, Symbol Check, Codebreaker, and Trails on the common factor 1 were −0.724, 0.748, 0.824, and −0.717, respectively. The factor loading coefficient of PDQ-5-D on the common factor 2 was 0.957 (Table 4). The correlation coefficient of the two common factors was 0.125 (Table 5).

Table 4 The Factor Load Matrix After Rotation

Table 5  The Results of the Structural Analyses of the THINC-It Tool

Parallel Validity

The two RTI data were not saved, and only 218 subjects were included for statistical analysis. The correlation r values between the THINC-it tool and the standard tests ranged from 0.291 to 0.921 (all P < 0.001) (Table 6).

Table 6 Parallel Validity Analysis of the THINC-It Tool and Standard Test

Discriminant Validity

Cognitive function scores were derived by averaging the Z-scores of all tests. Rank-sum test analysis showed significant differences in the Z-scores of the THINC-it total score, Spotter, Codebreaker, Trails, and PDQ-5-D (Z=−3.861, −4.017, −3.026, −4.408, −4.218; P<0.05) between the two groups, After adjusting for occupation, marital status and level of education by an analysis of covariance (ANCOVA) test, We also found that the same significant differences in Z-scores of the THINC-it total score, Spotter, Codebreaker, Trails, and PDQ-5-D (F=27.821, 17.122, 9.293, 8.561, 18.716; P<0.05). In contrast, there was still no significant difference in Symbol Check between the two groups (Table 7 and Table 8).

Table 7 Comparison of z-Scores on THINC-It Tool Between BD-D and HC

Table 8 Comparison of z-Scores on THINC-It Tool Between BD-D and HC, Adjusted for Occupation, Marital Status and level of Education

Discussion

The THINC-it tool is simple, quick to conduct, and has broad potential application in the clinical setting. The THINC-it tool has been validated in Chinese populations and has been shown to be able to detect cognitive disorders in patients with MDD.¹⁶ Our study replicated and extended the study findings reported in Chinese populations and demonstrated that the THINC-it tool had good construct validity, test-retest reliability, discriminative validity, and an acceptable internal consistency among adults with BD-D.

The study showed that Cronbach’s α coefficient of the THINC-it tool had good internal consistency. However, Cronbach’s α coefficient of the standard test was 0.688, which was unsatisfactory. Hou¹⁶ found that the subjective and objective tests of the THINC-it tool had Cronbach’s α coefficients of 0.684 and 0.704, respectively, in patients with depression. These scores were not ideal, but they were acceptable Another study by Harrison²⁰ revealed that the THINC-it tool had a good internal consistency, the internal consistency (Cronbach’s α) of the four objective tests ranged from 0.7 to 0.93, and the subjective test (PDQ-5-D) was 0.76. Thus, the THINC-it tool is a screening tool with good internal stability. Furthermore, The above studies demonstrated that the internal consistency of the THINC-it tool was acceptable in different cultural backgrounds and diseases.

The retest reliability suggested poor reproducibility of Trails, a finding that was consistent with the results obtained by Hou¹⁶ and Zhang,¹⁷ but inconsistent with results obtained by Harrison²⁰ (r=0.74–0.81). The discrepancy in the results was attributed to two factors. One factor could be the difference in the participants used in the retest. Healthy controls were retested in the study by Harrison,²⁰ but only BD-D patients were retested in our study. Another factor could be differences in interest frequency. Three assessments were conducted between the two retests in the Harrison study, which involved retesting the same cognitive tests. Thus, the stability of the THINC-it tool improved with increasing familiarity and repetition of the tool under the influence of learning effects.

The results also showed that the objective tests (Spotter, Symbol Check, Codebreaker, Trails) of the THINC-it were distributed in the common factor 1, the subjective tests (PDQ-5-D) in the common factor 2, and the factor loading coefficient of the absolute value was > 0.4. This indicated that the objective test was strongly correlated with common factor 1, and the subjective test was strongly correlated with common factor 2. In addition, in both frameworks, the dimensions of the subjective and objective constructs were set reasonably, so the structure of the THINC-it was clear and reasonable, giving it great structural validity.

The study showed that the PDQ-5-D had the highest correlation with standard test instruments in the subjective tests. Meanwhile, in the objective tests, the correlation between Symbol Check and Spotter was very poor, which was consistent with findings from a previous study. However, unlike a previous study,¹⁷ our study discovered that the Trials had the highest correlation in four objective tests. These results were similar to those obtained by Harrison²⁰ but different from those obtained in a study by Mclntyre.¹¹ The discrepancy in the results could be attributed to the study participants’ age and level of education, who had different proficiency in the order of letters. Several studies have shown a low correlation between the Symbol Check and the standard tests in healthy subjects and patients,^16,20 which was consistent with findings from our study. The low correlation between the two tests may be because the operational requirements of the symbol check differed from those of the traditional tests, and subjects needed to shift their attention quickly between stimulus sequences and response options.

The Symbol Check is an important component of the THINC-it tool, reflecting the working memory in visual space.¹¹ Our study tested the theory that found a significant difference in the THINC-it tool’s composite scores between the patients and the healthy controls. However, the Symbol Check did not distinguish between BD-D patients and healthy controls irrespective of occupation, marital status, or level of education. In other words, there was no significant decrease in working memory in BD-D patients compared with healthy volunteers. The above conclusion was in accordance with the research conclusions in depressive patients.¹⁶ This may be because the working memory was assessed using a dual-task paradigm, and subjects were susceptible to distraction by multiple tasks, leading to no significant difference in the results. It is also possible that in the actual test, most subjects (including healthy controls) had difficulty understanding the operating rules and operated with an enormous difficulty factor, making it difficult to distinguish the low scores. In addition, bias could have been introduced by only considering the accuracy rate while ignoring the reaction time in the analysis of Symbol Check data. These findings were contrary to the conclusions drawn by some scholars.^21,22 It may be relevant to account for factors such as the disease status, assessment tools, medication, and other characteristics of the enrolled patients during statistical analysis.

Our study found that bipolar depression had subjective and objective cognitive impairment such as attention/alertness, information processing speed, and executive function. Galimberti et al²³ found the impairment of memory and executive functioning by the Montreal Cognitive Assessment (MoCA) in BD-D patients. Some reseachers had used CANTAB to find that BD-D patients had cognitive impairment in multiple domains, including executive control, visuospatial memory, verbal working memory, verbal learning, memory and spatial cognition.²⁴ In another study, using WCST and Wechsler memory scales, patients with BD-D performed worse in memory and executive function.²⁵ According to the MCCB, cognitive impairment is more prevalent in patients with BD depressive episodes, especially in multiple domains related to reasoning problem solving, information processing speed and visual learning.²⁶ Despite the above research results used different cognitive assessment tools, the domains of cognitive impairment were approximately the same as ours. It verified the applicability of THINC-it tool again. The above recognized cognitive tools provide an accurate assessment of objective cognition, however, they do not provide a complete assessment of subjective cognitive impairment. In addition, the existing cognitive assessment tools were time-consuming, complicated in operation, and costly, which brought difficulties to clinical practices. The THINC-it tool would not only potentially avoid the above mentioned problems, but also further assess subjective and objective cognitive function as comprehensively as possible. The tool worked on computers and tablets and had advantages such as easy, quick and economical. Therefore, it was expected to enable dynamic assessment, which would facilitate clinical diagnostics and personalized treatment plans in the future.

In conclusion, the present study pointed out the presence of specific cognitive deficits in patients with BD-D compared with HCs. Specifically, deficits were observed in general cognitive functions such as executive function, attention/alertness, processing speed, and subjective cognitive impairment. These findings differ marginally from previous studies.^23,27,28 These inconsistencies could be related to differences in tools used to measure cognitive function. In summary, this study demonstrates that the THINC-it tool can be used to screen for cognitive impairment in patients with bipolar depression. There were some limitations to this study. Firstly, we also did not control for factors such as disease status, medication status, psychotherapy, and other factors that could have affected cognitive performance. In addition, the sample size used for the 7-day follow-up was insufficient due to the outbreak of the COVID-19 pandemic. More research is needed into the effect of psychosocial, behavioral, and physiological factors that affect adherence.