Reliability and validity of the Chinese version of Questionnaire – Children with Difficulties for Chinese children or adolescents with attention-deficit/hyperactivity disorder: a cross-sectional survey

Introduction

Attention-deficit/hyperactivity disorder (ADHD) is a common chronic neurobehavioral disorder characterized by excessive hyperactivity, impulsivity, inattention, or a combination of these, which affects ~6.26% of school-aged children in China.^1,2 ADHD is associated with a series of negative effects among children and adolescents, and puts a serious financial burden on families.² Children with ADHD usually face difficulties in various aspects of life – for example, family relationships, school life, friendships, and their daily behaviors during specific periods of the day, such as morning, during school, after school, evening, and night.^3–9 As the symptoms of ADHD are continuously recognized throughout the day and night time,^8,10 the assessment of a child’s functioning across various periods of the day is important from the perspective of the long-term prognosis of ADHD.¹⁰

There are several scales used in clinical practice to measure the symptoms and functional impairment of patients with ADHD in China and abroad. The Swanson, Nolan and Pelham IV (SNAP-IV) rating scale is a commonly used tool to assess ADHD symptoms.¹¹ The Weiss Functional Impairment Rating Scale-Parent (WFIRS-P) is another tool which is used to assess ADHD-related functional impairment in children and adolescents.¹² However, neither SNAP-IV nor WFIRS-P captures ADHD-related daily-life problems during specified periods of the day. No scale was available for this purpose in China.

In Japan, the Questionnaire – Children with Difficulties (QCD) is commonly used to evaluate parents’ perceptions of their child’s daily behaviors during specific periods of the day such as morning, school, after school, evening, and night time.⁸ The QCD has three key features: the ability to assess life function, the ability to enable the evaluation of life function at each period of the day, and convenience of use in daily clinical practice.⁸ The QCD is composed of practical and easy-to-understand questions inquiring about basic daily activities.¹³ One of the major advantages of the QCD is that it is more convenient to use than either SNAP-IV or WFIRS-P as it includes only 20 simple questions,⁸ which take less time to complete than SNAP-IV (consisting of 26 questions) and WFIRS-P (consisting of 50 questions).^11,12 Moreover, the QCD is more user-friendly in measuring problems in daily life during different time periods of the day, and offers necessary information for selecting appropriate drug therapy.⁸

The reliability and validity of the QCD in assessing daily-life problems of ADHD patients at various times of the day have not been evaluated in China. In order to use the QCD in future Chinese clinical studies, it is critical to evaluate the internal consistency and convergent validity of the QCD in Chinese children or adolescents with ADHD. Therefore, this study was designed to evaluate the reliability and validity of the QCD in Chinese children with ADHD. The major hypothesis of this study was that a satisfactory internal consistency of QCD and good correlations of QCD scores with SNAP-IV and WFIRS-P scores would be found. Moreover, this study answers the question whether the QCD can be recommended for use in daily clinical practice to identify the daily-life problems of ADHD patients during specified periods of the day in China.

Patients and methods

Study design and ethical considerations

This cross-sectional survey study was conducted at four study centers in China (Beijing, Shanghai, Nanjing, and Changsha) using paper questionnaires (study duration: November 2016 to March 2017). The investigators were apprised of all study procedures, and instructed to have face-to-face interviews with the children’s parents or guardians to collect data. Institutional ethics committee approval was obtained from each study center: Beijing Anding Hospital of Capital Medical University, Shanghai Mental Health Center, The Second Xiangya Hospital of Central South University, and Nanjing Brain Hospital Affiliated to Nanjing Medical University. Participants in the study were children and adolescents aged 6–18 years with ADHD and their parents/guardians. Written informed consent was taken from each parent on behalf of their children. The QCD as well as the SNAP-IV and WFIRS-P questionnaires were distributed to the parents or guardians of all enrolled patients, and they were asked to complete all questionnaires.

Participants and demographic characteristics of the patients

Outpatients aged 6–18 years who had been diagnosed with ADHD based on the Diagnostic and Statistical Manual of Mental Disorders, 5th Edition (DSM-5) criteria and who had visited psychiatry clinics were enrolled in this study. Patients with neurodegenerative disease, cerebral palsy, bipolar disorder, schizophrenia, psychotic disorder, and other severe psychiatric disorders were excluded.

The following demographic characteristics were recorded for each enrolled participant: age, gender, education level, comorbidity condition, ADHD diagnosis duration, ADHD current therapy, family history of ADHD, and ADHD subtype (Table 1).

Table 1 Patient demographic and baseline characteristics Notes: Because some patients received both behavioral and medication intervention, the cumulative percentage of behavior intervention, medication intervention, and no intervention does not equal 100%. Similarly, in medical intervention, the cumulative percentage does not equal 100% because a few patients received both atomoxetine and Chinese traditional medicine. Abbreviations: ADHD, attention-deficit/hyperactivity disorder; DMDD, destructive mood dysregulation disorders; N, total number of patients; n, number of patients in each category; ODD, oppositional defiant disorder.

Questionnaire – Children with Difficulties

The QCD comprises 20 questions with regard to ADHD-related difficulties occurring in the morning, during school, after school, in the evening, and overall difficulties over the entire day and night (Table 2). Each question is scored in four grades: 0 = completely disagree, 1 = somewhat (partially) agree, 2 = mostly agree, and 3 = completely agree. The average score for the QCD is defined as the average item-level score for all of the items on each assessment. The total score is the sum of all item-level scores. The subdomain score is the sum of all item-level scores within each domain, which is indicated by “items x-x.” Higher scores indicate higher life functioning and less difficulty in children’s daily activities that occur during specific periods of the day.⁸

Table 2 QCD scores for children and adolescents by their parents/guardians (N=200) Abbreviations: ADHD, attention-deficit/hyperactivity disorder; N, total number of patients; n, number of patients in each category; QCD, Questionnaire – Children with Difficulties.

SNAP-IV

The SNAP-IV (parents) is a commonly used tool to assess ADHD symptoms in China.¹¹ It consists of 26 items, and each item is scored on a four-point scale: 0 = not at all, 1 = just a little, 2 = quite a bit, and 3 = very much. The subdomains included in the scale are: inattention (items 1–9), hyperactivity/impulsivity (items 10–18), and oppositional criteria (items 19–26). The average score for the SNAP-IV is defined as the average item-level score for all of the items on each assessment. The total score is the sum of all item-level scores. The subdomain score is the sum of all item-level scores within each domain, which is indicated by “items x-x”. Higher SNAP-IV scores indicate worse ADHD symptoms, and vice versa.

WFIRS-P

The WFIRS-P assesses ADHD-related functional impairment in children and adolescents in China.¹² It consists of 50 items to be filled out by parents. The subdomains included in the scale are family, school, life skills, child’s self-concept, social activities, and risky activities. It is a four-choice Likert-type scale scored from 0 to 3 (no problem to high problem). The average score for the WFIRS-P is the average item-level score for the items with the response 0, 1, 2, or 3. Responses of “NA” were not included; thus, only items with the response 0, 1, 2, or 3 were included for total/average/subdomain score calculations. Higher WFIRS-P scores indicate less life functioning and more difficulty in children’s daily activities, and vice versa.¹²

Statistical analyses

A total of 200 ADHD outpatients, with an absolute minimum of 100 subjects, were planned to be enrolled in this study to ensure the item–subject ratio was ≥10. Descriptive statistics (frequency and percentage for categorical variables, and mean and SD for continuous variables) were presented. The internal reliability of the QCD scores (total and subscores) was examined by Cronbach’s alpha, which assessed the internal consistency of the questionnaire. Acceptable Cronbach’s alpha values range from 0.70 to 0.95,¹⁴ and a high Cronbach’s alpha value indicates good internal consistency of the items in the scale (≥0.9 = Excellent, ≥0.8 = Good, ≥0.7 = Acceptable, ≥0.6 = Questionable, ≥0.5 = Poor, and <0.5 = Unacceptable).¹⁵ Concurrent criteria validity of the QCD was examined by Spearman’s correlation of QCD scores (total scores and subscores) with those of the SNAP-IV and WFIRS-P, respectively. The threshold value of 0.3–0.7 for the absolute value of correlation coefficient is considered a satisfactory/good correlation.¹⁶ In order to explore the discrimination threshold of the QCD for determining ADHD severity, the receiver operating characteristic (ROC) analysis method was employed by using a binary classifier system (for SNAP-IV: a mean score of items 1–18 of <1.67 indicates insignificant ADHD symptom whereas ≥1.67 indicates significant ADHD symptom;¹⁷ for WFIRS-P: the impaired functions were <3 vs ≥3).¹⁸ The QCD cut-off points from the minimum to maximum values by every five points were tested. Sensitivity, specificity, and accuracy, as well as area under curve (AUC) were presented to measure the accuracy of a diagnostic tool. The two-tailed significance level P<0.05 was used. All data were analyzed with SAS 9.4 (SAS Inc., Cary, NC, USA).

Results

Patient disposition and characteristics

A total of 270 patients were screened for the study; of these, 70 patients were excluded due to not meeting eligibility criteria (Figure S1). A total of 200 eligible Chinese patients were analyzed in this study. The average (±SD) age of the enrolled patient was 10.4 (±2.66) years. The majority of patients were male (77.5%), and 49.0% had the combined ADHD subtype. The demographic and baseline characteristics of enrolled patients are summarized in Table 1.

Distribution of the QCD, SNAP-IV, and WFIRS-P scores

The questionnaires that were filled out completely were collected from the parents and guardians of 200 children. The mean (SD) of the total score of the QCD was 32.0 (9.77). The percentage of respondents who answered each question and the mean (SD) for each question or domain of QCD are shown in Table 2. The mean (SD) for total scores and subscores of SNAP-IV and WFIRS-P are shown in Table 3.

Table 3 SNAP-IV and WFIRS-P scores for children and adolescents by their parents/guardians Abbreviations: ADHD, attention-deficit/hyperactivity disorder; SNAP-IV, Swanson, Nolan and Pelham IV; N, total number of patients; WFIRS-P, Weiss Functional Impairment Rating Scale-Parent.

Internal reliability of the QCD

With regard to the internal consistency of the QCD, Cronbach’s alpha for the subscores ranged from 0.49 to 0.74, and the Cronbach’s alpha for the total score was 0.88 (Table 4). The highest internal consistency was noted for the early morning/before going to school subcategory of the QCD. Likewise, the lowest Cronbach’s alpha was noted for the overall behavior subcategory of the QCD. The Cronbach’s alpha value for the total QCD score indicates that the QCD is a reliable instrument to assess daily-life problems of ADHD patients during specified periods of the day in China.

Table 4 Internal consistency of the QCD (N=200) Abbreviations: N, total number of patients; QCD, Questionnaire – Children with difficulties.

Validity of the QCD

The correlation coefficients of the QCD total score with the SNAP-IV total score and WFIRS-P average score were −0.47 and −0.57, respectively (Table 5). Correlations for QCD with SNAP-IV and WFIRS-P were statistically significant (P<0.05). With regard to the convergent validity of the QCD, the correlation coefficients between the SNAP-IV score and the QCD subscores ranged from −0.15 to −0.47. The correlation coefficients between the WFIRS-P score and the QCD subscores ranged from −0.16 to −0.57. All correlations were significant at P<0.05. The lowest correlation was observed between “Night” on the QCD and “Hyperactivity/impulsivity” and “Oppositional criteria” on the SNAP-IV (Table 5). Furthermore, low correlations were observed between “inattention,” “hyperactivity/impulsivity score,” “oppositional criteria,” and “combination” of SNAP-IV and “after school” of QCD scores (Table 5).

Table 5 Correlation of the QCD with SNAP-IV and WFIRS-P scores (N=200) Note: All correlations were statistically significant at P<0.05. Abbreviations: N, total number of patients; QCD, Questionnaire – Children with difficulties; SNAP-IV, Swanson, Nolan and Pelham IV; WFIRS-P, Weiss Functional Impairment Rating Scale-Parent.

Sensitivity and specificity of the QCD

The AUC calculated from ROC analysis between the QCD and SNAP-IV was 0.70, indicating the accuracy of the QCD (Figure S2). A similar trend of accuracy was observed when the QCD was compared with the WFIRS-P, and the calculated AUC was 0.71. The sensitivity, specificity, and accuracy of the QCD against the SNAP-IV and WFIRS-P questionnaires are shown in Table S1.

Discussion

This is the first site-based study designed to test the reliability and validity of QCD in evaluating daily-life problems at various times of the day in Chinese children or adolescents with ADHD. In China, the SNAP-IV and WFIRS-P questionnaires have been most widely used to evaluate ADHD-related symptoms and function. However, these questionnaires do not inquire about behaviors during specific periods of the day.^11,12 Therefore, there was a need for a convenient, easy-to-use, reliable, and valid questionnaire which assesses daily-life problems at various times of the day among Chinese children or adolescents with ADHD.

The results of the present study indicated that the QCD had sufficient reliability and validity. The internal consistencies of both the total score and the subscores of the QCD were found to be satisfactory. Moreover, the convergent validity was found to be satisfactory for both the total score and the subscores of the QCD, with good correlation with the SNAP-IV and the WFIRS-P. Our study results are consistent with the Japanese study,⁸ which concluded that the QCD is a reliable and valid instrument for evaluating daily-life problems in Japanese children during different time periods of the day. The Cronbach’s alpha for the total score and subscores of QCD in our study was found to be similar to what was observed in the Japanese study; Cronbach’s alpha values for the total score was 0.88 in both studies.⁸ Cronbach’s alpha for subscores ranged from 0.49 to 0.74 in the Chinese study, and ranged from 0.57 to 0.78 in the Japanese study.⁸ The total score of the QCD significantly correlated with scores on the SNAP-IV and WFIRS-P, but subscores of the QCD exhibited a broad range of correlations in both Chinese and Japanese studies. In our study, the “night” subscore in the QCD displayed a low correlation with the SNAP-IV total score and the WFIRS-P average score. The SNAP-IV and WFIRS-P questionnaires do not include any questions related to sleeping problems or disorders, which could partly explain these low correlations in our study. The sensitivity and specificity of the QCD questionnaire were evaluated using ROC analysis. The AUC values from ROC analysis suggested a fair accuracy of QCD scores when compared with SNAP-IV and WFIRS-P.

In addition to its reliability and validity, the QCD can be completed by parents in ~10–15 minutes and is convenient to use in daily clinical practice in order to identify daily-life difficulties of ADHD patients during specific time periods of the day. The suggested ADHD severity discrimination threshold of the QCD total score was 30–35. Patients who had a QCD total score of ≤30 may have more impaired functions and, therefore, require immediate attention from their caregivers and further diagnostic tests. Treatment of ADHD requires the designing of a comprehensive and individualized plan for each patient and family. In the case of low subscores in the evening and at night, the clinician should pay more attention to the choice of medication for ADHD – for example, the presence of sleeping disorders or other comorbidities indicate the patient’s need for action during the day or at specific times of the day.

This study has some limitations that need to be considered. Due to the cross-sectional nature of the study design, the responsiveness of the QCD could not be examined by the current study. We have included only ADHD patients in our study, and the QCD scores of normal children or adolescents have not been studied. Future studies should compare these results with a normal community sample. Moreover, the subjects were recruited from only four study centers from tier 1 and 2 cities in China. Therefore, the present findings cannot be generalized for the overall Chinese population. However, the sample size in the present study was large enough to allow us to draw valuable conclusions. Furthermore, in the present cross-sectional survey study, there was only one interview planned for each patient. Because scores were collected on a single occasion, we could not examine test–retest reliability in this study. A follow-up study has been planned, which examines whether the QCD has the ability to distinguish subgroups of ADHD patients with regard to their demographic characteristics and severity of ADHD symptoms or ADHD-related functioning impairment. This will strengthen the evidence for the discriminant validity of the QCD.

Conclusion

Our study results found the QCD to be a reliable and valid instrument for evaluating daily-life problems of ADHD patients during specified periods of the day in China and recommend its use in daily clinical practice.

Acknowledgments

This study was funded and supported by Eli Lilly and Company. The authors would like to thank Rakesh Ojha, PhD, from Syneos Health for medical writing, and Antonia Baldo from Syneos Health for editorial support in the preparation of this manuscript. The authors would also like to thank Wan Qi Zhao from Eli Lilly and Company for project management, and Wendong Chen and Yi Chen from Changsha Normin Medical Technology Ltd., for assistance in the project operational processes, data cleanup, and project management.

Author contributions

Yi Zheng, Yasong Du, Lin Yan Su, and Xiao Yan Ke contributed to the study design, data collection and analysis, and critical revision of the manuscript. Yanlei Zhang, Zheng Yuan, and Yun Chen contributed to the study design, data management, analysis, interpretation, and critical revision of the manuscript. Qing Liu contributed to the data management, analysis, interpretation, and critical revision of the manuscript. All authors contributed toward data analysis, drafting and revising the paper and agree to be accountable for all aspects of the work.

Disclosure

Yanlei Zhang, Zheng Yuan, Yun Chen, and Qing Liu are employees of Eli Lilly and Company. The other authors report no conflicts of interest in this work.

---

References

1.		American Academy of Pediatrics. Clinical practice guideline: diagnosis and evaluation of the child with attention-deficit/hyperactivity disorder. Pediatrics. 2000;105(5):1158–1170.
2.		Wang T, Liu K, Li Z, et al. Prevalence of attention deficit/hyperactivity disorder among children and adolescents in China: a systematic review and meta-analysis. BMC Psychiatry. 2017;17(1):32.
3.		Biederman J, Faraone SV, Taylor A, Sienna M, Williamson S, Fine C. Diagnostic continuity between child and adolescent ADHD: findings from a longitudinal clinical sample. J Am Acad Child Adolesc Psychiatry. 1998;37(3):305–313.
4.		Iizuka C, Yamashita Y, Nagamitsu S, et al. Comparison of the strengths and difficulties questionnaire (SDQ) scores between children with high-functioning autism spectrum disorder (HFASD) and attention-deficit/hyperactivity disorder (AD/HD). Brain Dev. 2010;32(8):609–612.
5.		Pliszka SR. Treating ADHD and Comorbid Disorders: Psychosocial and Psychopharmacological Interventions. New York: The Guilford Press; 2009.
6.		Steele M, Jensen PS, Quinn DM. Remission versus response as the goal of therapy in ADHD: a new standard for the field? Clin Ther. 2006;28(11):1892–1908.
7.		Strine TW, Lesesne CA, Okoro CA, et al. Emotional and behavioral difficulties and impairments in everyday functioning among children with a history of attention-deficit/hyperactivity disorder. Prev Chronic Dis. 2006;3(2):A52.
8.		Usami M, Sasayama D, Sugiyama N, et al. The reliability and validity of the Questionnaire – Children with Difficulties (QCD). Child Adolesc Psychiatry Ment Health. 2013;7(1):11.
9.		Wilens TE, Biederman J, Spencer TJ. Attention deficit/hyperactivity disorder across the lifespan. Annu Rev Med. 2002;53:113–131.
10.		Usami M, Okada T, Sasayama D, et al. What time periods of the day are concerning for parents of children with attention deficit hyperactivity disorder? PLoS One. 2013;8(11):e79806.
11.		Zhou J-B, Guo L-T, Chen Y. Reliability and validity of the Chinese version of Swanson, Nolan and Pelham Version IV Rating Scale-Parent Form for attention-deficit/hyperactivity disorder. Chinese Mental Health Journal. 2013;27(6):424–428.
12.		Qian Y, Du Q-X, Qu S, Wang Y-F. Reliability and validity study of the Chinese version of Weiss Functional Impairment Rating Scale-Parent Form for school age children. Chinese Mental Health Journal. 2011;25(10):767–771.
13.		Wehmeier PM, Schacht A, Dittmann RW, Döpfner M. Global impression of perceived difficulties in children and adolescents with attention-deficit/hyperactivity disorder: reliability and validity of a new instrument assessing perceived difficulties from a patient, parent and physician perspective over the day. Child Adolesc Psychiatry Ment Health. 2008;2(1):10.
14.		Tavakol M, Dennick R. Making sense of Cronbach’s alpha. Int J Med Educ. 2011;2:53–55.
15.		Gliem JA, Gliem RR. Calculating, interpreting, and reporting Cronbach’s alpha reliability coefficient for Likert-type scales. Presented at: Midwest Research-to-Practice Conference in Adult, Continuing, and Community Education; October 8–10; 2003; Columbus, HO. Available from: https://scholarworks.iupui.edu/handle/1805/344. Accessed April 13, 2018.
16.		Ratner B. The correlation coefficient: its values range between +1/-1, or do they? J Target Meas Anal Market. 2009;17(2):139–142.
17.		Goulardins JB, Rigoli D, Piek JP, et al. The relationship between motor skills, ADHD symptoms, and childhood body weight. Res Dev Disabil. 2016;55:279–286.
18.		Dose C, Hautmann C, Doepfner M. Functional impairment in Children with externalizing behavior disorders: psychometric properties of the Weiss Functional Impairment Rating Scale – parent report in a German clinical sample. J Atten Disord. Epub 2016 Jul 28.

Supplementary materials

Figure S1 Flowchart of patient disposition. Abbreviations: ADHD, attention-deficit/hyperactivity disorder; BAH, Beijing Anding Hospital; DSM-5, Diagnostic and Statistical Manual of Mental Disorders, 5th Edition; n, number of patients in each category; NBH, Nanjing Brain Hospital; SMHC, Shanghai Mental Health Center; SXH, The Second Xiangya Hospital of Central South University.

Figure S2 Sensitivity and specificity of the QCD questionnaire (A, QCD vs SNAP-IV; B, QCD vs WFIRS-P). QCD cut-off points: QCD from minimum to maximum value by every five points. SNAP-IV cut-point: a mean score of items 1–18 of <1.67 indicates insignificant ADHD symptoms, whereas ≥1.67 indicates significant ADHD symptoms. WFIRS-P cut-off point: the impaired functions <3 vs ≥3. Abbreviations: AUC, area under the curve; QCD, Questionnaire – Children with difficulties; SNAP-IV, Swanson, Nolan and Pelham IV; WFIRS-P, Weiss Functional Impairment Rating Scale-Parent.

Table S1 Sensitivity, specificity, and accuracy of the QCD questionnaire against SNAP-IV and WFIRS-P (N=200) Note: Only the cut-off points with sensitivity ≥0.5 and accuracy ≥0.6 are presented. Abbreviations: N, total number of patients; QCD, Questionnaire – Children with difficulties; SNAP-IV, Swanson, Nolan and Pelham IV; WFIRS-P, Weiss Functional Impairment Rating Scale-Parent.