Evaluation of the Paper and Smartphone Versions of the Quick Inventory of Depressive Symptomatology-Self-Report (QIDS-SR16) and the Patient Health Questionnaire-9 (PHQ-9) in Depressed Patients in China

Introduction

Depression is a recurring mental disorder that represents an important burden to the individuals, the society, the health care systems, and the economy.¹ The lifetime prevalence of depression in adults is 20% worldwide.¹ The incidence of depression is increasing year by year, and the onset of the disorder is occurring at an increasingly younger age.² At present, depression is one of the largest medical burdens in Chinese society.³

Many tools are available for the evaluation of depression. In 2011, the Quick Inventory of Depressive Symptomatology-Self-Report (QIDS-SR16) was introduced by Liu et al.^4,5 The QIDS-SR16 has been shown to have good reliability and validity for the screening of depressive disorders and the measurement of depressive symptoms, and it is widely used in China.⁶ The Patient Health Questionnaire-9 (PHQ-9) was introduced in China by Bian et al.⁷ in 2009. The PHQ-9 has high sensitivity and specificity for depressive disorders⁸ and has been found to have good reliability and validity in Chinese patients.⁹

Those questionnaires are usually filled by the patients in paper versions. In fact, the paper versions are the only validated versions of these questionnaires. Paper versions require giving the questionnaire to the patient and taking it back. It requires to have the patient in the office or to send the questionnaire by mail. Therefore, filling the questionnaire at any time and anywhere is not possible. Smartphone apps based on psychiatric measurement-based care (MBC) may alleviate these difficulties.¹⁰ Moreover, currently, mobile-based treatment is predominantly based on web-based interventions, but with the development of technology, smartphones now play an increasingly important role in mobile-based therapies and MBC. Nevertheless, little is known about the use of smartphone versions of the scales of depression in clinical work. Care is needed when migrating questionnaires to electronic formats to ensure that measurement equivalence with the original is demonstrated and that the measurement characteristics of the scale remain unchanged.¹¹ The available research on measurement equivalence mainly aimed to examine the factors influencing the validity of the results when migrating from paper to electronic formats, the operability of electronic versions, and questionnaires regarding somatic diseases and health conditions,^12–15 but direct comparisons between electronic and paper versions of psychometric questionnaires are rare. These studies showed that the demographic characteristics of the respondents and the management model of the app would affect the measurement results of the electronic versions. Some patients may encounter difficulties when operating a smartphone, and the different ways in which the entries and options are presented to the patient may cause problems. Individuals with smartphone anxiety may report more negative emotions when completing an emotional assessment on a smartphone. On the other hand, some studies reported that patients are more comfortable with the use of electronic versions and find them quicker to complete than paper versions.¹⁶ In addition, some patients are more likely to express their mood and be more relaxed when completing the electronic versions.

Studies showed that paper and electronic scale measurements are equivalent,^17,18 but only using handheld computers, and there is a lack of consistent research on depressive disorders. Thus, the aim of the present study was to assess the equivalence between the paper and the smartphone versions of QIDS-SR16 and PHQ-9 for patients with depressive disorders in psychiatric hospitals. This could allow for more efficient monitoring of changes in depressive symptoms in MBC.

Patients and Methods

Study Design and Patients

This prospective study was conducted from March 2016 to September 2018 by convenience sampling at the Outpatient Department of Beijing Anding Hospital. It was the part of a project testing the effect of MBC in the management of major depression. Beijing Anding Hospital hospital receives 1500 outpatient visits daily and serves approximately 21 million people. The study protocol was approved by the Clinical Research Ethics Committee of Beijing Anding Hospital. All patients provided written informed consent to participate in this study and agreed to the publication of the data. This study was conducted in accordance with the Declaration of Helsinki.

All patients who were diagnosed with major depressive disorders were assessed using the Chinese version of the MINI version 5.0 modules on major depression¹⁹ by a study investigator. The inclusion criteria were 1) met the criteria for major depressive disorder according to DSM-IV; 2) 21–65 years of age; 3) able to understand and fill in the questionnaires; and 4) no major physical diseases. The exclusion criteria were 1) previously diagnosed mania (light mania), bipolar disorder, schizophrenia, schizoaffective disorder, or other mental disorders; 2) depressive episodes with psychotic symptoms; 3) alcohol addiction or history of acute poisoning; or 4) questionnaires were not completed.

Study Procedure

In order to inform the research, a preliminary investigation was conducted before the formal investigation. All the scale raters were experienced psychiatrists, who received consistency training on the diagnosis of diseases, the determination of symptoms, the assessment of scales, and the use of the smartphone app. In this study, the paper and smartphone scales were completed in a randomized crossover design, with an interval of 1–2 h between tests, separated by other neuropsychological tests.

The app (URL for IOS: https://www.pgyer.com/lb2R; for Android: https://www.pgyer.com/iyFC; Department of Software Engineering, Beijing University of Technology, China; Beijing Anding Hospital has the copyright and ownership) was downloaded and installed onto the participants’ phones with the assistance of the researchers. The participants first completed a demographic characteristics form and were randomly divided by a central system into two groups. Group 1 completed paper versions of the depression scale QIDS-SR16 and PHQ-9 and then completed the smartphone versions 1 to 2 h later. Group 2 completed the same scales in the reverse order. Brief training on operating the app was given by the researchers. The participants completed scale measurements independently but were allowed to ask for assistance.

App Design

The app was designed to be as simple and easy to use as possible in order to reach a wide range of people with depression. All the contents in the smartphone versions are the same as those in the paper versions. Basic instructions were provided on the app homepage, and all patients were given time to read these prior to completing the scales. Only one item was shown on each page. If the item was not completely visible on one page, the participants could scroll down to see it all. Answers could be modified while answering each question, but could not be modified when completed. The program could not continue without a response being submitted to each item. Once submitted, data were analyzed automatically, and the results were displayed on the medical side of the app.

Measurements

The demographic data questionnaire included questions on the general background (eg, sex, age, education level, onset age, duration of illness, and family history of psychiatric disorders) of the patients, which were completed by the patients themselves.

The QIDS-SR16 is a 16-item, self-administered scale that measures the severity of depressive symptoms relating to nine symptom domains over the last 7 days.^4,5 The overall score of the scale ranges between 0 and 27, and a higher score indicates more severe depressive symptoms. It takes 5–7 minutes to complete. The scores may be graded as mild (score of 6–10), moderate (score of 11–15), severe (score of 16–20), and very severe (score of 21–27).⁵

The PHQ-9 is a multipurpose instrument that assists in the screening and measurement of depressive symptoms within the past 2 weeks.²⁰ It is a brief self-report tool, comprising nine items. Respondents rate the frequency of the symptoms on a 4-point rating scale: 0 (not at all), 1 (several days), 2 (more than half the days), and 3 (nearly every day). The overall scores range between 0 and 27. The scores may also be graded as mild (score of 5–9), moderate (score of 10–14), moderate-severe (score of 15–19), and severe (score 20–27).⁸ The PHQ-9 only takes 3–5 minutes to complete and is rapidly scored.

In 2009, Si et al.¹⁹ compared the reliability and validity of the Mini International Neuropsychiatric Interview (MINI) with that of the Composite International Diagnostic Interview (CIDI) and the Structured Clinical Interview for Disorders (SCID). The results showed that the MINI, which is a diagnostic tool for all mental disorders and not only depression, had very acceptable reliability and validity scores. The MINI is a brief structured clinical interview based on the DSM-IV criteria.¹⁹ It has similar reliability and validity properties compared with the CIDI and the SCID but can be administered in less time. It includes 130 questions.

Sample Size

The sample size was determined according to the ISPOR guidelines.¹¹ The study power was set at 95%, and the expected intraclass correlation coefficient (ICC) was 0.9 (α=0.05), resulting in a target sample size of 110 patients.

Statistics

SPSS 25.0 (IBM Corp., Armonk, NY, USA) was used for data analysis. Continuous variables are presented as means ± standard deviations (SD) or as medians and interquartile ranges (IQR) according to their distribution, as determined by the Kolmogorov–Smirnov test; comparisons between groups were performed with the Student’s t-test or Mann–Whitney U-test, as appropriate. Categorical variables are reported as frequencies with percentages and were compared with the chi-square test. Concordance of scale scores between the paper and smartphone versions was analyzed using a two-way fixed-effects ICC model, including intra-groups and inter-groups. High positive ICCs indicate that different versions’ measurements covary and that the mean and variability of the scores are similar. Two-sided P<0.05 was considered statistically significant.

Results

Sociodemographic Data

Out of the 112 patients who fulfilled the inclusion criteria, 110 gave informed consent and provided complete data. In Group 1, 54 patients completed the paper versions first. In Group 2, 56 patients completed the smartphone version first. The mean age was 30.6±7.3 years, and 40.9% of the patients were male (n=45). The majority of patients (74.6%) had a high level of education: university or higher (n=82) (Table 1). All patients had used smartphones previously. The majority of patients (70.9%, n=78) expressed a preference for the mobile app version. There were no significant differences in any of the patient characteristics between the two groups (P>0.05, Table 1).

Table 1 Sociodemographic Characteristics of the Study Population

Intra-Group Consistency Measurement

High concordances were noted between the paper and smartphone versions for the overall and individual item scores of the QIDS-SR16 scale (Table 2) and PHQ-9 scale (Table 3) in the two groups.

Table 2 Intra-Group ICC (95% CI) of the QIDS-SR16 Scale

Table 3 Intra-Group ICC (95% CI) of the PHQ-9 Scale

Inter-Version Consistency Measurement

High concordances were noted between the paper and smartphone versions of the QIDS-SR16 scale for the overall score (ICC=0.904, 95% CI 0.861–0.934) and individual item scores (ICC range: 0.769–0.923) (Table 4).

Table 4 Consistency Tests Between the Paper and Smartphone Versions of the QIDS-SR16

High concordances were noted between the paper and smartphone versions of the PHQ-9 scale for the overall scores (ICC=0.951, 95% CI 0.929–0.967) and individual item scores (range 0.779–0.914) (Table 5).

Table 5 Consistency Tests Between the Paper and Smartphone Versions of the PHQ-9

Discussion

Smartphone-based questionnaires have advantages compared with their paper versions, but there is a lack of consistent research on depressive disorder questionnaires. Therefore, the present study aimed to assess the equivalence between the paper and the smartphone versions of the QIDS-SR16 and PHQ-9 for patients with depressive disorders in psychiatric hospitals in China. This study demonstrated the equivalence in the measurement properties of the paper and smartphone versions of the PHQ-9 and QIDS-SR16 in depressed patients in China.

Selection of the Questionnaires

In the present study, the QIDS-SR16 and PHQ-9 questionnaires were selected because they are often used in the literature and because they are short, easy to administer, and self-rating, which allows their application as smartphone versions. In the Sequenced Treatment Alternatives to Relieve Depression (STAR*D) trial, both the QIDS-SR16 and PHQ-9 were used.²¹ In an MBC project on major depression in China, the psychometric properties of the QIDS-SR and PHQ-9 in depressed inpatients were examined, and it was found that they have similar and acceptable psychometric properties in most domains.²² Moreover, the QIDS-SR16 and PHQ-9 have different characteristics. PHQ-9 consists of only one page, and the questions are concise and easier to answer. The QIDS-SR16 consists of three pages, and only people with high education levels find it as easy to answer as the PHQ-9.²³ Therefore, those two questionnaires were selected for the present validation study.

Study Population

In comparison to epidemiological data from a previous study on psychiatric disorders in China,²⁴ the average age of the patients in this study was relatively young, the average duration of illness was relatively short, and the average education level was relatively high. The reasons for this may be that older patients are inexperienced in the use of smartphones or do not have smartphones, and thus were unwilling to take part in the study. In addition, the participants in this study were outpatients from a tertiary hospital at the Center of Science Technology and Culture of China in Beijing. These patients tend to have a short illness duration and high levels of education. Ali et al.¹⁶ reported that patients prefer iPad versions of scales of depression over paper versions. Similarly, in this study, the majority of patients (73.8%) also preferred the electronic version.

Consistency Between Electronic and Paper Versions

Previous studies generally showed good concordance between the electronic and paper versions of various questionnaires across a wide variety of diseases and conditions,^25–27 but those studies were not controlled clinical trials, they did not use smartphone electronic versions, and did not include scales used in depression. Nevertheless, the results of the present study for QIDS-SR16 and PHQ-9 are consistent between the paper and electronic versions, as supported by the previous studies mentioned above. For outpatients suffering from depression, this study showed that paper and electronic versions had a high concordance on the whole.

It is generally believed that ICC values lower than 0.70 indicate poor reliability, values higher than 0.75 indicate good reliability for group comparisons, while ICC should be between 0.85 to 0.95 for applications at the individual level.¹¹ In this study, all the items in the intra-group ICC analysis showed good reliability in both groups, further supporting the reliability of the ICC between two versions of questionnaires. Between the two versions, the ICCs for the overall scores were all higher than 0.90, ie, 0.904 for the QIDS-SR16 and 0.951 for the PHQ-9. For the individual items, the ICC for three items on the QIDS-SR16 and two items on the PHQ-9 were less than 0.85. The items showing a low level of concordance included “Sad mood”, “Self-outlook”, and “Agitation/Retardation” in the QIDS-SR16, and “Feeling down, depressed, or hopeless” and “Trouble concentrating” in the PHQ-9. These low ICCs may have arisen because there is no clear criterion for judging mood or emotional state. The ICC of the Appetite/weight item was the highest, at 0.923. This might be because both appetite and weight are mainly manifested as behavioral problems and easy to judge. Items that are difficult to judge, ie, those without clear criteria, may be more affected by a switch to a different version.

The PHQ-9 showed good consistency between the electronic and paper versions. This is supported by a previous study that showed that computerization of the PHQ-9 did not affect its psychometric properties.²⁸ Compared with PHQ-9, the descriptions in the responses in the QIDS-SR16 are complicated. Sung et al.²³ determined that the QIDS-SR16 score was able to distinguish minor from major depression, while the PHQ-9 could not. This may be because the QIDS-SR16 has more items in which symptoms must be assessed more precisely. This feature, however, may not necessarily be better for the electronization of the scale, and this may lead to difficulties for patients in distinguishing between the different possible answers to an item. Based on these features, the PHQ-9 may be a good choice for app quantitative treatment, as it is simple but effective. Currently, the PHQ-9 is being further simplified, and we expect the results of relevant studies to be tested in the MBC field.²⁹

In this study, the expected ICC was 0.9, but the items’ ICCs were mostly lower than 0.90. This is possible because the participants in this study had major depressive disorders, with decreased energy and attention, which may be required to complete the scale. Nevertheless, the ICCs of the overall score of both measurements were higher than 0.90, and the ICCs of items were mostly higher than 0.85, ie, the consistencies were within the acceptable range.

Limitations

The main limitations of this study pertain to sample heterogeneity and the lack of qualitative data. Only outpatients from one hospital were recruited, and therefore the findings may not be applicable to all areas of China. In addition, in the process of moving from a paper version to an electronic version, differences in patient groups, electronic equipment, or operation procedures may affect the measurement results.^12,30 Therefore, the population characteristics of the patients are important. In this study, there was no simultaneous qualitative interview to collect information from patients to better understand the impact of different populations on measurement results. The factors influencing population characteristics should be considered in future research. Furthermore, in this study, the application environment was controlled and stable, so the results may have poor predictability for real clinical circumstances. In the present study, the interval between the two questionnaires was 1–2 h. The determination of the interval time was based on previous research experience and the specific operating environment of the subject, to reduce the risk of carryover effect and risk of change in mindset as much as possible. There is currently no scientifically proven standard time for this time issue. This is a limitation, and therefore, we tried to minimize this bias by answering the questionnaires in the reverse order in these two groups in our study. Finally, clinimetrics is a new field focusing on the science of clinical measurements and is considered an emerging topic in the field of measurement-based care,^31–33 and this was not evaluated in the present study. We will explore this further in future studies.

Conclusion

In conclusion, this study demonstrated equivalence between the paper and smartphone versions of two scales of depression in depressed patients in China. This demonstrates that both the QIDS-SR16 and PHQ-9 scales are appropriate for use in both paper and electronic versions. These findings support the use of the electronic versions of the PHQ-9 and QIDS-SR16 via smartphone apps. This is particularly the case for the PHQ-9 because it is simple and easier for patients to complete, with good psychometric equivalence. Both the PHQ-9 and QIDS-SR16 can be flexibly selected and applied in clinical practice and/or scientific research, as required.