Validity and Reliability Study of the Chinese Traditional Version of the Devereux Early Childhood Assessment for Toddlers DECA T

Introduction

Social-emotional development is defined as a child’s developing ability to (a) experience, manage and express the full range of different kinds of emotions; (b) develop satisfying and close relationships with adults and other children; and (c) explore their environment actively and learn.^1,2 Successful engagement with parents, teachers, and peers promotes the development of many skills essential for emotional regulation, problem solving, empathy, cognition, and language.^3–5 Given that learning is closely associated with socially mediated interaction in preschool children, unrecognized or undervalued social and emotional problems not only affect overall development but also impact on their readiness for kindergarten or elementary school, and potentially cause serious lifelong developmental issues, premature mortality, and adult socioeconomic status.^1,6 However, early identification of and service provision to infants and toddlers with social-emotional/behavioral problems have lagged far behind advances in early identification and service provision for delays in cognition, language, and motor development. An estimated 10% to 15% of 1- and 2-year-old children experience significant social-emotional problems.⁷ Moreover, research indicates that infant–toddler social-emotional/behavioral problems are not transient and highlight the need for early identification in multidomain and the effectiveness of early intervention.⁸ Early identification of children at risk of social-emotional problems is critical to provide evidence-based intervention and optimize social-emotional development as well as educational success.^9–14

Psychopathology in young children is often conceptualized as falling along the three broad domains of internalizing, externalizing, and dysregulation.¹⁵ In early childhood, internalizing problems include difficulties with anxiety, depression/withdrawal, fears, and shyness/inhibition. Externalizing problems include aggression, overactivity, impulsivity, and inattention. Recently, extreme problems in the regulation of state, affect, and sensory processing have been addressed in the regulatory disorders of the Diagnostic Classification System for 0–3-year-olds.¹⁶

There are some parent-reported psychometric assessments for infant and toddler social-emotional development and problems,¹ including the Ages and Stages Questionnaires: Social-Emotional-2 (ASQ:SE-2),¹⁷ Baby Pediatric Symptom Checklist (BPSC),¹⁸ Preschool Pediatric Symptom Checklist (PPSC),¹⁹ Brief Infant–Toddler Social and Emotional Assessment (BITSEA),²⁰ Child Behavior Checklist (CBCL/1.5–5),²¹ Devereux Early Childhood Assessment for Infants and Toddlers (DECA-I/T),²² Early Childhood Screening Assessment (ECSA),²³ Greenspan Social-Emotional Growth Chart (SEGC),²⁴ Infant–Toddler Social and Emotional Assessment (ITSEA),²⁵ Merrill–Palmer-Revised Scales of Development (M-P-R)—Social-Emotional domain,²⁶ and Social-Emotional Assessment/Evaluation Measure (SEAM™).²⁷ Although the Child Behavior Checklist (CBCL/1.5–5)²¹ and the Infant–Toddler Social and Emotional Assessment (ITSEA)²⁵ questionnaires have good reliability and validity, they have too many items for use as screening tools. Of the other two brief screeners in existence, the 35-item Eyberg Child Behavior Inventory²⁸ focuses on conduct problems in children aged 2 to 11 years and has demonstrated acceptable reliability and validity, while the 40-item Toddler Behavior Screening Inventory (TBSI)²⁹ addresses common behavior problems in 1- to 3-year-olds. However, they do not address both problems and competencies.

Few instruments focus on the social-emotional competence specific to the toddler period. The Devereux Early Childhood Assessment for Toddlers (DECA-T), which is one of the few standardized, norm-referenced behavior rating scales related young children’s mental health, resilience, and social-emotional development, was developed with the aim of assessing toddlers aged between 18 and 36 months.²² It is an assessment tool based on parent or childcare staff ratings after a 4-week observation period. It helps users to identify young children who may be experiencing social and emotional challenges, leading to interventions to reduce associated behavioral problems. The DECA-T contains 36 items measuring three protective factors, including Attachment/Relationships (A/R), Initiative (IN), and Self-Regulation (SR). The DECA-T has a Total Protective Factors scale (TPF), which is a composite across the scales for each assessment. Therefore, the scale provides an overall indication of the strength of young children’s protective factors.

There are two different approaches for evaluation of psychometric properties of rating scales: classical test theory and item response theory.^30–33 The limitations of classical test theory include the summed score problem and sample-dependent statistics, which may result in different psychometric properties when based on different samples.^34,35 Item response theory analysis provides item-level statistics that are not influenced by differences between samples. Item response theory assumes that scale items can be ordered along the levels of a latent trait, with item “difficulty” demonstrating where items are difficult (rare) or less difficult (common). Mokken scale analysis is a non-parametric item response theory derived from Guttman scaling.^36,37 On a Guttman scale, a single response can be used to predict responses to all items on the scale. Being a non-parametric analytical method, Mokken scale analysis is robust according to the underlying distribution of the data, thus avoiding the methodological limitations of previous studies.³⁶ To the best of our knowledge, no study has used Mokken scale analysis to examine the properties of the DECA-T. Therefore, the aim of this study was to evaluate the psychometric and clinimetric properties of the Chinese version of the DECA-T (C-DECA-T) in a population of toddlers aged 18 to 36 months, including those enlisted in public well-baby clinics (control group) and those from at-risk groups who were recruited from child psychiatric clinics (subject group). Homogeneity of the scale, item sensitivity, item severity, reliability, internal consistency, sensitivity, and specificity in discriminating an at-risk group for social-emotional problems and a healthy group were tested.

Methods

Participants and Setting

The study was carried out in Chang Gung Memorial Hospital at Linkou and YuNing Psychiatric Clinics. A co-author and colleagues (child psychiatrists and child psychologists) conducted translation (English to Chinese) and back-translation of the DECA-T, which was approved by the original publisher (Kaplan Press). The community-based sample initially consisted of 81 children aged 18 to 36 months who visited the hospital for conditions other than psychiatric problems. The clinical-based sample consisted of 57 children aged 18 to 36 months who visited the hospital seeking help for psychiatric problems. We excluded six subjects in the community-based sample and seven in the clinical-based sample due to one of the parents failing to complete the questionnaires.

The C-DECA-T was distributed to parents of the subjects and the healthy control group. Sample exclusion criteria included children with severe chronic physical illness. Mental health staff were trained to be familiar with the instruments and the procedures before the study. Both parents of each case were given a brief description of the study and written informed consent was obtained. Mothers and fathers rated the C-DECA-T separately to detect inter-rater reliability. Other information collected by self-rating instruments included demographic data of parents and children, birth weight, birth complication, physical condition, questions about parents’ mental health (the Chinese Health Questionnaire: CHQ-12), and the Child Behavior Checklist (CBCL/1.5–5). For test–retest reliability, recruited parents rated the C-DECA-T again in the following 14 days.

Ethics

The study data did not indicate any identifying personal information. Institutional Review Board approval was obtained from Chang Gung Memorial Hospital.

Instruments

Chinese (Traditional) Version of the Devereux Early Childhood Assessment for Toddlers (C-DECA-T)

The DECA-T is for toddlers ages 18 to 36 months and contains 36 items (on a five-point Likert scale from 0=never to 4=very frequently) measuring the social-emotional competence within the past 4 weeks, and includes three protective factors: Attachment/Relationship (A/R), Initiative (IN), and Self-Regulation (SR). The Attachment/Relationship (A/R) scale (18 items) assesses the relationship between the infant or toddler and significant adults such as family or teachers. The Initiative (IN) scale (11 items) assesses the toddler’s ability to use independent thought and action to achieve his/her needs. The “Self-Regulation” (SR) scale (7 items) assesses the ability to gain control of and manage emotions and sustain focus and attention. The DECA-T has a Total Protective Factors (TPF) scale, which is a composite across the scales for each assessment. The TPF scale provides an overall indication of the strength of the young child’s protective factors. We converted the raw data of three subscales and TPF into T scores following the original standardization table for the calculation of T scores and national percentiles for children of all ages (18–36 months). The internal reliability of the original DECA-T, rated by parents, was 0.94 in TPF, 0.87 in A/R, 0.92 in IN, and 0.79 in SR.⁶

Chinese Version of Child Behavior Checklist for Age 1.5–5, Parent Form (CBCL/1.5–5)

Children’s behavior and emotional problems were assessed by the CBCL/1.5–5 parent form.²¹ The CBCL/1.5–5 includes a set of 99 items with three response options (0=not true; 1=somewhat or sometimes true; 2=very true or often true), plus one open-ended item to add problems not listed on the form. Raw scores were analyzed for DSM-oriented scales as well as for original syndrome scales.³⁸ The DSM-oriented scales were designated as affective disorder, anxiety disorder, somatic disorder, attention-deficit/hyperactivity disorder, oppositional defiant disorder, and conduct disorder. The sum of the 100 items was counted as the total problems score, and 67 of them were scored specifically to seven narrowband behavioral syndromes, including emotionally reactive (9 items), anxious/depressed (8 items), somatic complaints (11 items), withdrawn (8 items), sleep problems (7 items), attention problems (5 items), and aggressive behavior (19 items). In addition, two broadband behavioral syndromes were derived, with the former four syndromes constituting an internalizing problem scale and the latter two syndromes constituting an externalizing problem scale. The Chinese version of the CBCL/1.5–5 was found to have good to excellent levels of reliability (internal consistency, test–retest reliability, and inter-parent agreement) and an acceptable model fit for the seven-syndrome factors when used in preschool children in Taiwan.³⁹

Chinese Health Questionnaire (CHQ-12)

Parents' depression and anxiety were assessed by the CHQ-12, which has 12 items rated on a four-point Likert scale, with a higher score indicating more severe psychiatric symptoms. The CHQ-12 was a short form of CHQ (60 items) which was derived from a Chinese translation of the General Health Questionnaire,⁴⁰ with the addition of specially designed, culturally relevant items. The Cronbach’s alpha in the community and clinical settings were 0.84 and 0.83, respectively. Persons with CHQ-12 scores of 4 and over are considered to have minor psychiatric disorders.^41,42

Statistics and Data Analysis

First, we tested C-DECA-T properties using a classical theory test approach, including analyzing the reliability (test–retest reliability, internal consistency, and inter-rater reliability) and validity (concurrent, discriminative, and sensitivity/specificity). We conducted sensitivity and specificity analysis and receiver operating characteristic (ROC) analysis to find the cut-off value among the at-risk group and the healthy control group. Concurrent validity was hypothesized to be expressed in significantly negative Pearson correlations between the C-DECA-T TPF and the main CBCL/1.5–5 broad-band syndromes (internalizing, externalizing, and total behavior problems), as well as parent’s depression and anxiety (CHQ-12). Conversely, small-to-medium negative correlations were expected to be found between the C-DECA-T TPF and the CBCL/1.5–5 and CHQ-12. All data management and statistical analyses were performed using SPSS version 18 software (SPSS Inc., Chicago, IL, USA).

Second, we evaluated C-DECA-T clinimetric properties using an item response theory approach. The C-DECA-T was evaluated by principal component analysis (PCA) first, followed by Mokken scale analysis. Mokken scale analysis was conducted using the Mokken package in R version 3.3.0 (R Foundation for Statistical Computing, Vienna, Austria).⁴³ 

Principal Component Analysis (PCA)

PCA was conducted to classify the 36 items of the C-DECA-T using the psych package in R version 3.3.0 (R Foundation for Statistical Computing, Vienna, Austria). The Kaiser–Meyer–Olkin measure of sampling adequacy was computed to determine the adequacy of sample size. We extracted two unrotated components of the C-DECA-T (PC1 and PC2). PC1 referred to the general factor of the C-DECA-T, and PC2 referred to the dual factor of the C-DECA-T.

Mokken Scale Analysis (MSA)

Mokken scale analysis, a nonparametric form of item response therapy derived from Guttman scaling,^36,37 was developed to avoid the limitations of classical test theory including the summed score problem and sample-dependent statistics, which may result in different clinimetric properties when based on different samples. We checked the four assumptions of Mokken scale analysis, unidimensionality, local independence, monotonicity, and non-intersection. First, the assumption of unidimensionality was checked using Loevinger’s scalability coefficients H.^43,44 Loevinger’s scalability coefficients comprises three indexes: item-pair (H_ij), item (H_i), and scale (H_s) scalability coefficients. A scale is considered weak if 0.3≤H≤0.4, moderate if 0.4≤H≤0.5, and strong if H≥0.5. Second, we used conditional association to identify local dependence. Third, we checked serious violations of monotonicity for each item with a cutoff crit statistic >40. Fourth, we checked serious violations of non-intersection for each item with a cutoff crit statistic >80. For those items which met the four assumptions, we then checked invariant item ordering (IIO) of those items with method manifest invariant item ordering (MIIO). The Ligtvoet coefficient H^T was used to evaluate IIO. The IIO of a scale is considered weak if 0.3≤H^T≤0.4, moderate if 0.4≤H^T≤0.5, and strong if H^T ≥0.5.^36,43

Results

Data Description

Table 1 presents the subjects’ demographic data and parents’ education level. There was no statistically significant difference between the subject group and control group regarding age, body weight, height, and parents’ education level. In the subject group, there was a higher boy versus girl ratio than the control group. Table 1 also shows the total score and subscales of C-DECA-T in both groups. The results conveyed that the TPF and subscales – A/R, IN and SR in the subjects’ group, rated either by father or mother – were significantly lower than those in the control group.

Table 1 Demographic Data and C-DECA-T T Scores

Reliability

Table 2 displays excellent internal consistency in Total Protective Factors, subscale of Attachment/Relationship (Cronbach’s alpha, 0.94 and 0.92, respectively), good internal consistency in the subscale of Initiative (Cronbach’s alpha, 0.84) and acceptable internal consistency in the subscale of Self-Regulation (Cronbach’s alpha, 0.74). The test–retest reliability was found to be good in the TPF (r=0.80) and acceptable in the subscales of A/R, IN, SR (r=0.77, 0.77, 0.72, respectively). Based on the Cicchetti and Sparrow guideline,⁴⁵ inter-rater reliability between father and mother for the TPF (ICC, 0.46), the subscales of A/R (ICC, 0.41) and IN (ICC, 0.52) were fair and the subscale of SR was good (ICC, 0.66).

Table 2 Reliability Statistics of the C-DECA-T

Principle Component Analysis of the C-DECA-T

The overall measure of the sampling adequacy of the C-DECA-T was 0.93. All Kaiser–RMeyer–Olkin values for individual items of the C-DECA-T were between 0.87 and 0.97. Using principle component analysis, we extracted two unrotated components of the C-DECA-T (PC1 and PC2). As shown in Table 3, all items had positive loadings on the PC1. Such results suggested that the 36 items of the C-DECA-T measured the same latent trait and supported the composite scale of the C-DECA-T, the TPF scale. Regarding the PC2, 18 items had positive loadings on PC2 (3, 8–10, 12, 16, 19, 23, 26–35) and another 18-items had negative loadings (1, 2, 4–7, 11, 13–15, 17, 18, 20–22, 24, 25, and 36). Compared with the original construct of the C-DECA-T, the items with positive loadings on the PC2 corresponded to the IN scale and the SR scale except two items (items 8 and 21).

Table 3 Mokken Scale Analysis of the C-DECA-T

Scalability of the C-DECA-T Using Mokken Scale Analysis

Table 3 also shows the scalability, monotonicity, and intersection of the C-DECA-T assessed using Mokken scale analysis. The result shows the C-DECA-T has a weak unidimensional scale (H_s=0.35), which referred the C-DECA-T as a multi-dimensional scale, supporting its construct validity. The C-DECA-T was divided into three subscales (H_s ranged from 0.35 to 0.47). The AR has a moderate reliable Mokken scale (H_s=0.47); the other two subscales, IN (H_s=0.38) and SR (H_s=0.35) are weak reliable Mokken scales (Table 3).

Validity, Sensitivity, and Specificity

The concurrent validity of the C-DECA-T was investigated by analysis of association between the C-DECA-T and the Chinese CBCL/1.5–5. The results showed the C-DECA-T was negatively correlated with both CBCL/1.5–5 broadband syndromes, including internalizing scale, externalizing scale, and total problem scale. The TPF was negatively correlated with all DSM-oriented scales of the CBCL/1.5–5 including depression, anxiety, autism spectrum disorder (ASD), attention-deficit hyperactivity disorder (ADHD), and oppositional defiant disorder (ODD). Moreover, the TPF scale was negatively associated with five narrowband syndromes, including emotional reactive, anxiety/depressed, withdrawn, attention, and aggressive behavior problems (Table 4).

Table 4 Pearson’s Correlations Between the Subscales of the C-DECA-T and Demographic Date, Child Behaviour Problems, and Parental Emotional Distress

The discriminative validity of the C-DECA-T was carried out by comparing clinical subjects and control group C-DECA-T scores. The difference between the clinical subjects and control group C-DECA-T scores was significant with a medium effect size (Cohen’s d=0.67).

The area under curve (AUC) of the C-DECA-T in prediction of social-emotional competence was 0.69 (95% confidence interval 0.59–0.79) (p < 0.000). The optimal cutoff of the TPF score of the C-DECA-T was a T score of 40.1 (T40.1), corresponding to the 19th percentile. The T40.1 had a sensitivity of 95% and a specificity of 68% (Figure 1).

Figure 1 ROC curve of C-DECA-T.

Discussion

To the best of our knowledge, this is the first study to examine the clinimetric properties of the DECA-T in a Chinese ethnic population. Overall, the results showed that the C-DECA-T has a satisfactory psychometric performance with high internal consistency, inter-rater reliability, test–retest reliability, concurrent and discriminative validity. Principal component analysis showed the C-DECA-T has sampling adequacy. Mokken scale analysis showed the 36-items of the C- DECA-T formed a weak unidimensional scale (H_s=0.35), supporting C-DECA-T is a multi-dimensional scale.

Similar to the English version, the C-DECA-T is a reliable tool for measuring toddler social-emotional competence. The Cronbach’s alpha and test–retest reliability of the C-DECA-T were 0.94 and 0.80, respectively. The inter-rater reliability between mother and father in the C-DECA-T was fair (ICC = 0.46). The validity of the C-DECA-T was also supported by the significant negative correlation with total problems scale of the CBCL/1.5–5. Furthermore, the C-DECA-T was found to be negatively associated with internal and external behavior problems, narrowband syndromes (emotional reactive, anxious/depressed, social withdrawal, attention problems, aggressive behaviors), and DSM-IV orientated scales (depression, anxiety, ASD, ADHD, ODD).

Mokken Scale Analysis Approach

Since this is the first study evaluating DECA-T clinimetric properties, we cannot compare the results with the literature. The Mokken scale analysis of the C-DECA-T revealed the items 4, 6, 7, 11, 15, 17, 21–23, 29, 30, 34, and 35 were removed before the backward selection. Items 3, 13, and 31 were removed in the backward selection using method manifest invariant item ordering (MIIO). There are 20 items of the C-DECA-T having the property of invariant ordering (the coefficient H^T was 0.43 for the remaining 20 items), from the most prevalent (easiest) to the least prevalent (most difficult): 2, 36, 18, 14, 5, 20, 25, 8, 27, 1, 24, 19, 9, 16, 26, 32, 12, 28, 10, and 33. If the most prevalent item (easiest), such as item 2 (“show affection for a familiar adult”) is rated negative, it means most items might be rated as negative as well. In contrast, if the least prevalent item (most difficult), such as item 33 (“calm herself/himself”) is rated positive, it means most items might be rated as positive as well. The two items could provide the highest amount of information for clinical practice in the assessment of the social-emotional strength and needs in toddlers.

Test–Retest Reliability and Inter-Rater reliability

The test–retest reliability of the C-DECA-T TPF (r=0.80) was found to be acceptable, though slightly lower than the original DECA-T, in the subscales of A/R, IN and SR (r=0.77, 0.77, 0.72, respectively). One reason for this might be due to the test–retest interval in the original DECA-T being 24 to 72 hrs compared to 14 days in the current study, thus allowing less time for symptoms to change.

The result showed that the correspondence (intraclass correlation coefficients; ICCs) between maternal and paternal rating in the C-DECA-T was fair in the TPF scale and subscales of A/R and IN, and good in the subscale of SR. In a meta-analysis study, Duhig et al suggested higher interparental score correspondence in externalized behaviors than the internalized behavior problems in children aged between 3 and 19 years.⁴⁶ Regarding the effect of child age on interparental rating correspondence, the meta-analysis showed lower interparental correspondence for children in early (aged 3 to 5 years old) and middle (aged 6 to 12 years old) childhood than for adolescents.⁴⁶ The observed differences between maternal and paternal ratings highlight the importance of gathering reports from both parents when assessing early social-emotional behavior problems. The inter-rater reliability coefficients for the original DECA-T score were 0.70 in TPF, 0.62 in A/R, 0.64 in IN, and 0.72 in SR. The interrater reliability coefficients were lower than the original DECA-T (Table 2); thus, cultural differences in the differences between father’s and mother’s reporting of toddlers social-emotional behavior/competences might be worthy of future study.

Parental Depression and Toddlers’ Social-Emotional Competence in the C-DECA-T

Toddlers rated with lower social-emotional competence in the C-DECA-T would likely include those with problems that may be precursors to psychopathology and whose parents have distorted perceptions of child functioning, as may occur with parental depression.⁴⁷ In the present study, maternal depression/anxiety was significantly negatively correlated with toddlers’ social-emotional competence on three subscales, while paternal depression/anxiety was also negatively correlated with social-emotional competence and toddler’s social initiative and emotion self-regulations. Numerous studies have shown maternal depression was associated with poor mother–infant interaction,⁴⁸ reduced infant social responsiveness and engagement,^2,49 increased risk of infant insecurity and preschool attachment,⁵⁰ leading to disturbances in the acquisition of competences, such as prosocial peer relations and mastery motivation,⁵¹ and may predict later behavioral problems in school-age boys but not girls.⁵² However, the effect of depression in fathers during the early years of a child's life has received little attention. One prospective population study found depression in fathers during the postnatal period was associated with adverse emotional and behavioral outcomes in children aged 3.5 years and an increased risk of conduct problems in boys after controlling for maternal postnatal depression and later paternal depression.⁵³ The mechanism between parental depression/anxiety and toddlers’ social-emotional problems should be investigated in future studies.

Strengths and Limitations

The strength of our study is that it involved at-risk children (e.g., presenting symptoms of pervasive developmental disorders or of disruptive behavior disorders). We applied the Mokken scale analysis to assess the construct validity and dimensionality of the C-DECA-T. Although this investigation provided valuable information on the reliability, validity, and usefulness of a new child social-emotional resilience screening measurement, important study limitations should be pointed out. One limitation of the present study is that information, such as parenting skill and parent–child interaction, which could affect children’s social-emotional development, were not collected in the study. Another limitation of this study was that it was conducted in a highly urbanized setting; therefore, a high number of parents had university-level education, which may limit the generalizability of the findings from this study. Moreover, the sample size for this study was too limited to conduct factor analysis for the C-DECA-T. A larger sample will be required for future study. Despite these limitations, this is the first study examining the clinimetric proprieties of the C-DECA-T and applying clinimetric analyses to a tool measuring social-emotional strength and weakness in toddlers. Therefore, this short screening scale is suitable for use in surveys with two-phase designs, in which a quick screening test is used to select a subset of respondents who are likely to have social-emotional problems requiring more intensive psychiatric diagnostic assessment. Although the inter-rater reliability between father and mother was fair in TPF and acceptable in the subscales, the correspondence and discrepancy between maternal and paternal views toward toddlers’ social-emotional problems/competence presents a potential area for future research. Future validation of the C-DECA-T in a larger population with greater variability in socio-economic status would increase the generalizability of our findings and reinforce our conclusions.