Psychometric Properties of the Chinese Version of the Clinically Useful Depression Outcome Scale for Patients with Major Depressive Disorder

Introduction

Major depressive disorder (MDD) is a disabling mental disorder because it leads to impaired social function and decreased quality of life, as well as increased morbidity and mortality.^1,2 In addition to mood symptoms, individuals with MDD experience impairments in physical, occupational and social functioning.³ High prevalence and recurrence rates of MDD may lead to an increased financial burden on global healthcare system.^4–6 The World Health Organization (WHO) estimates that MDD will be the leading cause of the global burden of disease by 2030.⁷ The importance of treating patients until achieving a full remission was emphasized in many studies.^8,9 The residual symptoms in treatment associated with a much greater likelihood of relapse or recurrence of depression.¹⁰

The outcome of MDD is mainly based on the observations of clinicians and the self-report of patients. There are different tools to measure the levels of depression in the clinical practice. The 17-item Hamilton Rating Scale for Depression (HRSD),¹¹ and the Montgomery–Asberg depression rating scale (MADRS)¹² are commonly used to evaluate depressive symptoms and the effect of treatment in the field of depression. The current goal of antidepressant therapy is achievement of remission, which is defined solely on the basis of symptom severity scores as follows: ≤7 on the HRDS₁₇, ≤10 on the MADRS.¹³ However, the work of evaluating the patients’ symptoms with these instruments usually needs to be completed by trained psychiatrists and costs a large amount of time, subjective bias among different psychiatrists can affect the outcomes of these assessments.¹⁴ There are also many convenient self-report scales, such as the Patient Health Questionnaire-9 (PHQ-9),¹⁵ the Beck Depression Inventory (BDI),¹⁶ the hospital anxiety and depression scale (HADS).¹⁷ These self-management questionnaires are the most commonly used self-rating scales for measuring depression worldwide. But the scales mentioned above are designed to evaluate patients from the perspective of clinical symptoms, they cannot make global assessments of depressed. The Clinical Global Impression scale (CGI) is a classic instrument for making global assessments. The overall impression during an interview is a main element of psychiatric evaluation, but it is vague and difficult to operationalize.¹⁸ Therefore, The improved Clinical Global Impression Scale (iCGI) was designed to improve the validity of CGI severity and improvement scales in the field of depression.¹⁸ However, none of them is capable of assessing both symptoms and clinical outcomes at the same time, unlike the Clinically Useful Depression Outcome Scale (CUDOS).

The CUDOS was designed to be a brief, quickly scored instrument, within the MIDAS project (Rhode Island Methods to Improve Diagnostic Assessment and Service) to develop useful instruments for clinical practice.¹⁹ The CUDOS is a self-report scale, containing 18 items: 16 items concern the diagnostic criteria of MDD according to the DSM-IV, and the other two items concern the psychosocial damage caused by MDD and the quality of life of the patients.¹⁹ Every item in CUDOS uses 0 to 4 to represent the frequency of these symptoms in the past week (0=not at all true (0 days), 1=rarely true (1–2 days), 2=sometimes true (3–4 days), 3=often true (5–6 days), 4=almost always true (every day), with a total score of 0 to 72. The higher the score is, the more severe the depression is. The CUDOS can provide more complete information on the symptoms of MDD and improve work efficiency. It can also be subjected to rigorous scientific scrutiny and used by other researchers for scientific research.¹⁹

The CUDOS has been translated into various languages and has satisfactory reliability and validity in different regions and populations, such as Korea and Spain.^14,20 It has been reported that CUDOS has been successfully used to evaluate the depressive symptoms of patients with type 2 diabetes in Taiwan area.²¹ However, there is no such previous research on the reliability and validity of the Chinese version of the CUDOS in patients with MDD in China mainland. The purpose of this study was to assess the reliability and validity of the Chinese version of the CUDOS for the evaluation of depression in Chinese patients with MDD.

Methods

Design and Participants

This study was supported by the Guangdong Science and Technology Project (project NO: 2017A020215095) and was approved by the Clinical Research Ethics Committee of Guangdong Provincial People’s Hospital (No. GDREC2018156H (R1)). This study was conducted in accordance with the Declaration of Helsinki. Authorization for the use of CUDOS was approved by the original author Mark. Zimmerman. The patients with MDD were recruited from Guangdong Provincial People’s Hospital from October 2018 to October 2019. A total of 190 patients were recruited and met the following conditions: (1) all patients were provided written informed consent; (2) they were 18–60 years old; (3) they could read and understand all of the questions in the Chinese version of the CUDOS; and (4) they met the diagnostic criteria of MDD in the DSM-5 during last three months. The exclusion criteria were as follows: (1) suffering from bipolar disorder and other mental disorders; (2) having a history of alcohol or drug abuse in the past year; and (3) suffering from serious physical (medical) conditions. All patients were assessed while receiving their treatment as usual, which could be long-term ongoing or recently prescribed.

Translation of the CUDOS

The translation work started after obtaining the consent of the original author. The CUDOS was translated according to the guidelines of adaptation and validation of instruments in cross-cultural health care research.²² Firstly, the English version of the CUDOS was translated through the forward translation process by two bilingual and bicultural translators, all of whom were fluent in both Chinese and English. One of them is familiar with colloquial phrases, health care slang and jargon, idiomatic expressions, and emotional terms in common use in the Chinese. Secondly, the two translated versions of the instrument were compared, any ambiguities and questions were discussed and resolved by a discussion group. The final version was reviewed by a professional translator and a scholar of Chinese language studies and was agreed upon by the discussion group. Translation and back-translation of the CUDOS were repeated after state-of-the-art procedures in cross-cultural assessment^22,23 until the discussion group thought that the adapted Chinese version was suitable for Chinese patients with MDD.

Instruments

Clinically Useful Depression Outcome Scale (CUDOS)

The CUDOS is a self-report scale used to evaluate depressive symptoms and the functional aspects of depression.Depressive symptoms, the psychosocial damage caused by MDD and the quality of life were assessed by the Chinese version of the CUDOS.

Hamilton Rating Scale for Depression (HRSD)

The 17-item HRSD has been the most frequently used clinician-rated severity of depression scale in antidepressant efficacy trials.²⁴ The HRSD has demonstrated satisfactory internal consistency reliability and validity.²⁵

improved Clinical Global Impression Scale (iCGI)

The Clinical Global Impression-Severity Scale (CGI-S) is widely used in clinical practice, and the severity of patients’ depressive symptoms is evaluated by an 7-point Likert scale (from 1 = Normal to 7= Among the most extremely ill patients).²⁶ The iCGI is designed to address a broad spectrum of depressive disorders, and improves the response format for the Clinical Global Impression severity scale in depression.¹⁸

Procedure

All of the participants underwent a clinical assessment by a trained psychiatrist using the DSM-5 criteria and those meeting the diagnostic criteria for MDD according to the DSM-5 were referred to this study. After entering the study group, their general demographic data (the patient’s age, gender, occupation, education level, family history of mental illness, age of first onset, marital status, frequency of onset, etc.) were obtained. Then, the Chinese versions of the HRSD, iCGI and CUDOS were completed. The participants complete the instruments in a quiet room. Sixteen participants were randomly chosen to return to the hospital seven days after the initial test. We conducted an analysis with the entire sample, which included patients who met the full criteria for MDD as well as patients who were in partial and full remission.

Data Analyses

All analyses were performed using IBM SPSS version 25.0. The continuous variables are described by the mean ± standard deviation (SD), and the categorical variables were described by frequency and percentage. Internal consistency reliability was evaluated using Cronbach’s alpha. For clinical applications, much higher values of Cronbach’s alpha are needed.²⁷ Test-retest reliability was estimated with the intraclass correlation coefficient (ICC). The CUDOS dimensionality was assessed with principal component analysis (PCA). The Scree test, parallel analysis and interpretability of the simple structure were used to determine the number of components to be retained. A P-value less than 0.05 was considered to indicate a statistically significant difference between the data sets.

Reliability Testing

Reliability refers to the stability and consistency of the results measured by the scale. In this study, internal consistency reliability (Cronbach’s alpha, split-half reliability) and test-retest reliability coefficient of the total scores of the CUDOS were selected to test the stability of the scale.

Internal Consistency

The Cronbach’s alpha coefficient was used to evaluate the internal consistency of the total scores scale of the CUDOS. Cronbach’s alpha coefficient between 0.90 and 0.95 indicates that the internal consistency of the scale is desirable, between 0.80–0.90 indicates that the consistency of the scale is good, and between 0.70–0.80 indicates that the consistency of the scale is regarded as satisfactory.²⁷

Split-and-Half Reliability Coefficient

Split-and-Half Reliability refers to the half-and-half correlation coefficient between the first half part scores and the remaining part scores of the Clinically Useful Depression Outcome Scale.

Test-Retest Reliability Coefficient

Test-retest reliability is used to test the cross-time stability and consistency of the scale. Test-retest reliability of the CUDOS was evaluated with the intraclass correlation coefficient (ICC). ICC was interpreted according to the ranges of clinical significance proposed by Cicchetti (1994).²⁸ Test-retest reliability was examined in a sample of 16 participants. They repeating the test after an interval of 7 days.

Validity Testing

The construct validity was investigated using principal component analysis (PCA). The Kaiser–Meyer–Olkin measure and Bartlett’s spherical test was used to evaluate the data are suitable for PCA. In this study, Spearman correlation analysis was conducted to determine the correlation between the Chinese version of the CUDOS scores and the scores for the Chinese version of the HRSD. The ability of the CUDOS to discriminate between different levels of depression severity was investigated based on the rating of iCGI-S. An analysis of variance and post-hoc comparison of Tukey’s honestly significant difference (HSD) test was used to evaluate the known-groups construct validity.

Sensitivity and Specificity Testing

Sensitivity and specificity were evaluated by assessing the receiver operating characteristic (ROC) curve to obtain the optimal cutoff score when screening for major depression.²⁹ The Youden Index for any potential cut-off score is defined as the sum of the sensitivity and specificity (expressed as probabilities) of the scale at that point minus 1. The cut-off is selected as the point with the highest Youden Index.^30,31

Results

Characteristics of the Participants

A total of 203 patients with MDD were approached for this study and completed their interviews. After the analysis of the selection criteria, 13 patients had to be excluded either because they had not been diagnosed with MDD during the last three months (N=10) or because they could not complete the questionnaire successfully (N=3). Thus, the final valid sample for this study was made up of 190 patients. Their sociodemographic and clinical descriptive variables are shown in Table 1. The mean score on the 17-item Hamilton depression scale for the 190 depressed patients was 15.60 (SD=10.09). The mean score of the CUDOS was 28.01 (SD=18.47).

Table 1 Demographical Characteristics and Clinical Data of the Patients

Reliability Analysis

The Cronbach’s alpha coefficient of the total score of the CUDOS was 0.95 and the split-half reliability coefficient of the CUDOS was 0.92, respectively, suggesting excellent internal consistency reliability of the Chinese version of the CUDOS. Removal of any of the CUDOS items did not increase the Cronbach’s alpha coefficient (Table 2). The intraclass correlation coefficient (ICC) for the total score of the CUDOS was 0.77 (95% CI 0.449–0.911, P < 0.01), which shows excellent test-retest reliability.

Table 2 Internal Consistency and Corrected Item-Total Correlation of CUDOS Items

Validity Analysis

The results of the Kaiser–Meyer–Olkin measure of sampling adequacy were excellent (0.94), and Bartlett’s spherical test showed significant (χ²=2962.81, P<0.01), which indicates that the data are suitable for PCA. The ratio between the eigenvalues of the first and second components was 7.27 (10.26/1.41), which exceeds the critical value of 4, a criterion usually used as evidence of unidimensionality.³² This one-factor structure explained 56.99% of the variance. All factor loading values were positive, statistically significant (P<0.05), and ranged from being good to excellent.

The Pearson correlation coefficient between the HRSD and the CUDOS was 0.90 (P < 0.01). Table 3 displays the CUDOS mean scores’ distribution according to the depression severity levels as measured by the iCGI-S. The mean CUDOS scores of the different depression severity levels were 7.73±7.61, 27.29±9.28, 37.71±10.08, 45.64±7.21, 52.25±8.06, respectively. The five-groups analysis of variance was significant (F=173.74; df=4; P<0.01) and the differences among these groups were significant using Tukey’s HSD test.

Table 3 The CUDOS Mean Scores’ Distribution According to Depression Severity Levels as Measured by the iCGI-S

Sensitivity and Specificity Analysis

A receiver operating curve is a plot of a measure’s sensitivity versus one minus the specificity at each cutoff score.³³ When iCGI-S=1 was the “gold standard” definition of remission, the area under the ROC curve of CUDOS was 0.97 (95% CI: 0.950–0.990, P < 0.01, Figure 1). A score of 14/15 as the optimal cutoff point was suggested when screening for major depression. At this cutoff point, the sensitivity was 90.20% and the specificity was 93.60% in our sample (Table 4).

Table 4 Sensitivity, Specificity, and Youden Index at Various Cut-off Points of the CUDOS

Figure 1 The receiver operating characteristic (ROC) curve of the Chinese version of CUDOS for identifying patients in remission. Abbreviations: HDRS, The 17-item Hamilton Rating Scale for Depression; iCGI-S, The improved Clinical Global Impression-Severity Scale. Notes: When iCGI=1 was the “gold standard” definition of remission, the area under the ROC curve of CUDOS was 0.97 (95% CI: 0.950–0.990, P < 0.01); When HRSD≤7 was used as the criterion for remission, the area under the ROC curve of CUDOS was 0.97 (95% CI: 0.953–0.993, P < 0.01).

When we use the HRSD score less than or equal to seven as used as the criterion for non- depression,³⁴ the area under the ROC curve of CUDOS was 0.97 (95% CI: 0.953–0.993, P < 0.01, Figure 1). This high AUC shows that CUDOS can quite accurately distinguish between depressed and nondepressed patients. The optimal cutoff score to discriminate between depressed and nondepressed patients was 14/15, with a sensitivity of 95.40% and specificity of 88.10%.

Discussion

The results of this study showed that the Chinese version of the CUDOS is reliable, valid instrument identifying depression patients in remission in a Chinese clinical setting. It retains the same psychometric properties as the original version. The internal consistency of the CUDOS was 0.95 and the split-half reliability coefficient of the CUDOS was 0.92, which indicates the agreement among the individual components of the scale was very good. The test-retest reliability of the scale was 0.77, demonstrating an excellent temporal stability. Regarding to its internal structure, the factor analysis of the CUDOS identified a single-factor structure, which was consistent with the results of previous studies.^19,20 This findings implied that the single-factor model of this scale was stable in different culture populations.³⁵ The Pearson correlation coefficient between the HRSD and the CUDOS was 0.90, which showed that the CUDOS was more highly correlated with the HRSD than with measures of the other symptom domains. The known-groups construct validity of the Chinese version of the CUDOS seems to be excellent in discriminating between different depression severity levels patients. The area under the receiver operating characteristics curve of 0.97 indicates that the CUDOS has excellent properties for use as a screening instrument in the identification of MDD.

The cut-off score of the CUDOS for Chinese patients in this study was lower than the previously recommended value.³³ Another study reported that a score of 19/20 was the optimal cutoff score for identifying MDD, which is consistent with that of the original CUDOS cut-off score for identifying mild depression and remission.³⁶ The CUDOS has demonstrated satisfying validity and reliability to support its use for the assessment of the severity of depression symptoms in patients diagnosed with type 2 diabetes mellitus in routine clinical practice.²¹ Our study evaluated the Chinese version of the CUDOS is useful for measuring the symptoms and outcomes of MDD. Our findings suggested that the same cut-off score might not be appropriate for all settings. There are different cut-off values under different regions.²¹ Based on our findings, we recommend using a cut-off score of 14/15 for CUDOS when identifying patients in remission among MDD patients in China.

The HRSD has been one of the most widely used outcome measures in the clinical setting and antidepressant efficacy trials (AETs).³⁷ Self-applied scales such as the Beck Depression Inventory (BDI), and PHQ-9^7,38 have been successfully used in evaluating psychiatric outpatients as well as in clinical trials. Compared with these depression scales, CUDOS has advantages in evaluating not only psychosocial impairment and quality of life but also clinical symptoms of MDD. More than anything, patients reported that it took less time and reduced the burden on the evaluators.

Several limitations of the study should be noted. Firstly, the main limitation of this study lies in the potential sampling bias due to single-center cohort recruitment. Thus, the generalizability of CUDOS to patients with different socio-demographic or clinical characteristics needs to be determined in future works. Secondly, bipolar depression and other mental disorders were excluded in this study. Therefore, our findings are only applicable to assess the clinical outcome of patients with MDD. Further research should be conducted in different disease patterns to expand the application and confirm the validity. Thirdly, this study provides only cross-sectional data. It is not clear whether CUDOS remains sensitive over time to reflect the outcome of MDD.

Conclusion

In conclusion, the results of this study suggested that the Chinese version of the CUDOS is an effective and reliable self-administered questionnaire to measure depression outcomes in the Chinese population. The optimal cut-off scores of 14/15 are recommended for identifying patients in remission. The Chinese version of the CUDOS appears to be a useful measurement for evaluating the remission and the outcome of MDD in clinical practice.