Validation of the Chinese Version of the Autism Diagnostic Interview-Revised in Autism Spectrum Disorder

Introduction

Autism spectrum disorder (ASD), characterized by socio-communication deficits, and repetitive restricted behaviors and interests, is a common childhood neurodevelopmental disorder, 1% in the child population in Taiwan,¹ that substantially impacts families and society.² Language ability and cognitive function are two important predictors for the prognosis of ASD. Pooled estimates of the recent meta-analysis revealed around half of people with ASD have poor outcomes regarding employment, social interaction, and independent living.³ The etiologies of ASD are complex and include genetic influence⁴ and environmental risk factors.⁵ Early identification of ASD is important since intervention can begin earlier and may improve short to mid-term outcomes.⁶ The previous study proved the diagnosis of typical autism at age two was stable when followed up at age nine.⁷ The American Academy of Pediatrics suggests that early signs of ASD including little or no eye contact, limited response to parent’s smile or facial expressions, no pointing behaviors, and so on.⁸

Diagnosis of ASD requires information of the developmental history and clinical observation through nonstructural or structural instruments. The current gold standard instruments for diagnosing ASD are Autism Diagnostic Observation Schedule (ADOS)⁸ and Autism Diagnostic Interview-Revised (ADI-R).⁹ The ADOS is a standardized, semi-structured diagnostic instrument to assess the social interaction, communication, play and imagination for individuals with possible diagnosis of ASD.⁸ The ADI-R was published by Lord et al in 1994.¹⁰ The ADI-R is a standardized, semi-structured diagnostic instrument for ASD and is administered by trained interviewers to parents/caregivers of individuals with possible diagnosis of ASD.¹¹ The ADI-R can be administered for children and adults, as long as their mental age is above 18 months. The ADI-R has been translated to 19 languages¹² and used in non-English countries such as Brazil,¹³ Greek,¹⁴ Japan,¹⁵ Korea,¹⁶ Taiwan¹⁷ and so on. The case numbers of the validation studies varied from study to study. Most studies showed the ADI-R is a reliable and valid instrument. Some studies suggested the sensitivity of ADI-R might decrease when applied to toddler and preschool children.^15,18,19 Cultural effects, including the cultural perception of the ASD symptoms, have also been discussed.²⁰ To achieve the universal diagnosis and to across the linguistic and cultural boundaries, different language versions of the instrument would be translated. Before the application of the translated version of the instrument to different populations for clinical or research use, the investigation of the psychometric properties of the translated version of the instrument is very important. Though the Chinese version of the ADI-R was approved by WPS in 2007¹⁷ and has been used for the research purpose, the psychometric properties had not been surveyed thoroughly and published. This study aims to investigate the discriminant validity, internal consistency, test–retest reliability, factor analysis, receiver operating characteristic (ROC) analysis, sensitivity, specificity, Youden index, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio (PLR) and negative likelihood ratio (NLR) of the Chinese version of the ADI-R in Taiwan.

Methods

Participants

The present study recruited total 74 participants with ASD and 33 control participants without ASD. The participants with ASD were clinically-based and enrolled from YuNing Clinic, a major child and adolescent Psychiatric clinic of Northern Taiwan. All participants with ASD (aged from 3 years 4 months to 32 years old) were diagnosed as autistic disorder by two experienced child psychiatrists according to DSM-IV criteria.²¹ Fifty-eight cases were comorbid with intellectual disability. Thirty ASD participants had no verbal or minimal verbal ability. The control participants (aged from 3 years 8 months to 41 years old) were community-based, and none had ever been diagnosed as ASD according to information provided by parents and medical records. Individuals with obvious motor, hearing, and visual impairment were excluded. The corresponding author (Yu-Yu Wu), a senior board-certified child psychiatrist, is an internationally independent trainer for both ADOS-2⁸ and ADI-R.¹⁰

This study was approved by the Institutional Review Board of Antai Medical Care Cooperation, Antai Tian-Sheng Memorial Hospital (TSMH IRB No./Protocol No: 15–075-A2) and conducted in accordance with the Declaration of Helsinki of the World Medical Association.²² Written informed consent was obtained from all participants.

Procedures

There was a Traditional Mandarin Chinese version of the ADI-R translated from English to Chinese by professional translators and with back-translation to English approval by the original authors. Then, Taiwan’s autism research team, a group of child psychiatrists and psychologists led by senior author (Susan Shur-Fen Gau), organized a task force to correct medical terminology of the translated version of the ADI-R. The Traditional Chinese version of the ADI-R was approved by WPS in 2007¹⁷ and has been used to assist in research settings^23–43 in Taiwan for individuals with ASD. In 2017, Psychological Publishing Co., Ltd. got permission from WPS to publish the ADI-R including testing the reliability of the instrument used in Chinese culture. Since the ADI-R had been translated to Traditional Chinese with the original authors proved back-translation, only the new 2003 algorithm scoring sheet needed to be translated (was done for this study by Wu) and with back-translation proved by the original authors. In translating, some examples mentioned in the ADI-R involved language and cultural differences, but most of the examples were kept without modification since they conveyed the meaning of questions well. The example regarding language difference included #38 “‘Plin’ for a free-falling piece of paper or fabric, or ‘Mashuda’ for triangles” (“Plin” and “Mashuda” were reserved without translated to Chinese). The examples involving culture difference included #47 “such as mowing the lawn with some toy vehicle” (most Taiwan families do not have lawn outside the house), #61 “social games such as “Here we go around the Mulberry Bush” or “Ring-A-Ring-O-Roses” (modified to “sing and play” and “throw away the handkerchief” in the Chinese version), #64 “Ring-A-Ring-O’-Roses” (English was reserved without translated to Chinese) and #74 “Salt and peppers are on the table” (most Taiwan families do not put salt and peppers on the table). All interviews of ADI-R were performed by two clinical psychologists who were research reliable on the instruments with the senior child psychiatrist who is an independent trainer for both ADOS-2 and ADIR. Both clinical psychologists are female and have 15 years of research experience in using ADI-R.

Instrument

The Autism Diagnostic Interview-Revised (ADI-R)

The ADI-R is a standardized and semi-structured interview instrument. The ADI-R consists of 93 items and assesses the patient comprehensively. The content of the evaluation encompasses family backgrounds, early development, development milestones, regression and loss of learned skills, and three categories of core symptom domains including social interaction, language and communication, and restricted and repetitive behaviors and interests. Additionally, the ADI-R evaluates self-harm, aggression, disease onset age, and special talents. Forty-two items of the ADI-R can be scored and calculated according to the diagnostic algorithm to achieve the final diagnosis. The diagnostic algorithm includes four domains and follows ICD-10⁴⁴ and DSM-IV²¹ criteria. Domain A describes social interaction symptoms; Domain B describes language and communication symptoms; Domain C describes restricted and repetitive behaviors and interests; and Domain D marks the age onset of abnormal development reported by parents/caregivers. Domains A, B and C each contain four subscales, eg, A1, A2, A3, A4, etc. In individuals with verbal ability, all four subscales of Domain B can be assessed (Domain BV), while in individuals without verbal ability, only non-verbal communication items of B1 and B4 can be evaluated (Domain BNV). The original scores of all items were transformed according to the manual, and we used the transformed scores in the following statistical analysis.

Required Sample Size Estimation

Sample size was estimated by software G*Power version 3.1 (Heinrich-Heine-Universität Düsseldorf, Düsseldorf, Germany). The effect size (Cohen’s d) was set at 0.8, type I error (alpha) 0.05, power 0.80, statistical method is student’s independent t-test (ASD vs control groups), and the allocation ratio of ASD:control is 2:1. The estimated minimum required sample sizes of each group were: 39 ASD and 19 control.

Statistical Analysis

Continuous variables were reported with mean ± standard deviation (SD) and were compared using Student’s independent t-test or Mann–Whitney U-test (if normality was not assumed). Categorical variables were presented as number and percentage and were compared using Chi-square test. The reliability of all items and three core symptom domains were analyzed using Cronbach’s α to present internal consistency, the statistics corrected item-domain correlation and Cronbach’s α if item deleted were also reported. EFA was used to investigate the belongings between items and factors. The extraction method of EFA was principal component analysis; the rotation method was varimax with Kaiser normalization. The allocation of the item to factor was according to the maximum factor loading, and the absolute loading value should be larger than 0.40. The factor with the most items of a domain would be assigned to the domain. Different domains belonging to the same factor was allowed. All analyses were done using IBM SPSS Version 25 (SPSS Statistics V25, IBM Corporation, Somers, New York). The statistical significance level for all the tests was set at a p-value < 0.05, two-tailed.

A summary table was established to indicate the item analysis results, including mean (difficulty), corrected item-domain correlation, Cronbach’s α if item deleted, discrimination between ASD and control (p-value), and EFA results (factor loading). Mean <0.50 or >1.50, absolute correlation coefficient <0.70 (large correlation level), increasing Cronbach’s α if an item was deleted, non-significance of discrimination between ASD and control, and wrongly allocation of item were the definitions of the unfavorable index of item analyses.

A ROC curve analysis was used to investigate the diagnostic efficacy of each domain to ASD. The area under the ROC curves (AUCs), sensitivity, specificity, Youden index, PLR, NLR, and the cut-off points suggested by comparatively maximum Youden index were reported.

Results

Participant’s Characteristics

There were a total 107 participants included in this study, 74 participants with ASD and 33 controls without ASD. The average age of the ASD group was 14.06±6.14 years and male-to-female gender ratio was 3.93:1 (59:15). The average age of the control group was 15.92±10.94 years and male-to-female gender ratio was 0.94:1 (16:15). The mean age of all participants was 14.63±7.93 years, and the male-to-female gender ratio was 2.34:1 (75:32). No statistical significance was found in age between ASD and control groups (p=0.772); however, there were comparatively more male participants in the ASD group than the control group (79.73% vs 48.48%, p=0.001).

Discriminant Validity

We performed the comparison of all the items between ASD and control groups (Supplementary Table 1). Almost all items showed significant differences between groups and represented good discriminant validity. However, item #36 and #38 of Domain BV failed to discriminate between ASD and control participants (all p>0.05).

Internal Consistency and Test–Retest Reliability

High reliability was observed in the overall all-item score (0.973), Domain A (0.979), and Domain B (0.898). A fairly good reliability was noted in Domain C (0.776). There were 10 ASD participants who underwent a re-test to confirm the test–retest reliability of the ADI-R in this study. The within-domain Pearson’s correlation coefficients were 0.97 (Domain A), 0.99 (Domain BV), 0.98 (Domain BNV), and 0.95 (Domain C). All results indicate a good test–retest reliability of the ADI-R.

Exploratory Factor Analysis (EFA)

EFA was used to investigate the item allocation of domains. As indicated in Supplementary Figure 1, the suggestive number of factors by elbow point was 3. In further analysis, the number of factors was set at 3 to observe the theoretical item allocation. Table 1 shows the results of allocation, the maximum factor loadings of items were colored. As indicated, the items in Domain A and non-verbal communication items in Domain B were all allocated in the same factor. Verbal communication items in Domain B developed as an isolated factor, and item #36 and #38 fell out of this domain. Domain C seemed to be allocated to the same factor with Domain A and non-verbal communication items in Domain B. Item #68 and #39 fell out of this domain.

Table 1 Factor Loadings for the Algorithm Items (N=107)

Item Analysis Summary

Supplementary Table 2 summarized the results of item analyses, including mean (difficulty), corrected item-domain correlation, Cronbach’s α if item-deleted, discrimination between ASD and control (p-value), and EFA results (factor loading). Each unfavorable result would be marked with a “X”. As counted in Supplementary Table 2, item #36, #38, #68, and #39 seemed to have comparatively poor statistical properties (number of “X” ≥ 3) in this scale. The overall results of statistical properties were poorer in verbal communication items in Domain B and Domain C than Domain A and non-verbal communication items in Domain B.

Domain BV Results Within Participants with Verbal Ability

Thirty ASD participants in our study had minimal or no verbal ability. Since verbal communication items in Domain B were restricted to participants with verbal ability, the related items were re-analyzed with only verbal participants. As indicated in Table 2, after excluding non-verbal participants, the comparisons of all verbal communication items in Domain B reached statistical significance. However, the reliability results of item 36 and 38 remained lower. The revised Cronbach’s α of Domain BV was improved (0.939) after excluding non-verbal participants.

Table 2 Domain BV Results Within Participants with Verbal Ability

The ROC Results and Diagnostic Efficacy of Each Domain to ASD

Figure 1 shows the box plot with all points between groups in the domain. As indicated, distributions of domain scores between two groups had little overlap. On average, participants in the control group had lower scores than those in the ASD group. However, there were still some participants overlapped in scores between groups. ROC analysis was used to further investigate the cut-off points of each domain. Table 3 and Figure 2 demonstrate all ROC functions and diagnostic parameters of each domain to ASD. For a good index, it is expected to have higher AUC (>0.8), sensitivity (>0.8), specificity (>0.8), Youden index (approaching 1), PPV approaching 1, NPV approaching 1, PLR (>10), and smaller NLR (<1). As indicated, all domains have extremely good statistical properties in these parameters. Compared to the cut-off points defined by original questionnaire tool (A=10, BV=8, BNV=7, C=3), the results in this study were stricter in domain A, and looser in Domain BV and BNV in diagnosing ASD. If we define ASD as the scores over the cut-off points in at least one of the four domains (Domain A, B, C, D), the sensitivity is 100%, specificity is 93.94%, PPV is 97.37% and NPV is 100%. If we define ASD as the scores over the cut-off points in all four domains (Domain A, B, C, D), the sensitivity is 97.30%, the specificity is 100%, PPV is 100%, and NPV is 94.29%.

Table 3 The ROC Results and Corresponding Diagnostic Parameters of Suggested Cut-Offs Within Domains

Figure 1 “Box and whisker plots with all points” between groups in Domain A, Domain BV, Domain BNV, and Domain C. The participants without verbal ability were excluded from Domain BV.

Figure 2 The ROC functions in Domain A, Domain BV, Domain BNV, and Domain C in ASD group (control group as reference).

Post-Hoc Estimation of Statistical Power

Comparing between ASD and control groups, the Cohen’s d of the total score of Domain A, BV, BNV, and C were 8.67, 4.34, 5.03, and 4.32 respectively. These extremely high effect sizes estimated the same statistical power: 1.00.

Discussion

The present study investigated the psychometric properties of the Chinese version of the ADI-R. The participants included cases with ASD and controls without ASD, and their ages ranged from preschool to adulthood. The internal consistency evaluated by Cronbach’s α showed good to excellent results, and test–retest reliability was excellent. Moreover, we measured several types of validity. The discriminant validity indicated that most items performed well at differentiating ASD from non-ASD participants. The sensitivity, specificity, PPV, and NPV were all high for the clinical diagnosis of ASD. The factor analysis referred to a three-factor structure.

Discriminant Validity

The original English version of the ADI-R published by Lord et al presented good discriminant ability for social items and non-verbal communication items.¹⁰ However, while this might be limited to the small samples, most of the verbal communication items did not show significant results even after non-verbal ASD cases were excluded. The repetitive behaviors items showed least discriminant ability. About half of the items had significant results, while half of the items showed marginal and non-significant results. The original authors explained that these items were kept because previous studies had proved the diagnostic usefulness of these items in elder population. We had assessed each item of the instrument to evaluate the discriminant validity, and nearly all items can successfully differentiate cases with ASD from the controls except for #36 and #38. The item #36 (inappropriate questions or statements) and #38 (neologisms/idiosyncratic language) are regarding language use. After excluding the cases with no verbal ability, the discriminant validity of item #36 and #38 became significant. Therefore, all items of the Chinese version of the ADI-R can effectively differentiate cases with ASD from controls without ASD.

Reliability

Three core symptom domains of the Chinese version of the ADI-R showed good Cronbach’s α results respectively (0.979, 0.898, 0.776) and good results for overall items (0.973). The results represented fair internal consistency and reliability of the Chinese version of the ADI-R. Our results were similar to the original English version of the ADI-R, and the Cronbach’s α results for social and communication items were higher (0.95, 0.84) and the Cronbach’s α of the repetitive behavior items was lower (0.69).¹⁰ Moreover, the present study adopted the test–retest method to measure external consistency. The participants were assessed by the same interviewer on two separate occasions. A total of 10 participants received a test–retest evaluation. Similar to the reliability results of the original English version,¹⁰ our study provides strong evidence to support the high test–retest reliability and high consistency over time of the Chinese version of the ADI-R.

Factor Structure

The original English version of the ADI-R was designed as three-dimensional structure corresponding to the three categories of core symptom of ASD. Tadevosyan et al collected the ADI and the ADI-R data from 292 people with autism, measured the factor structure by principal components analysis and reported six domains including spoken language, social intent, compulsions, developmental milestones, savant skills and sensory aversions.⁴⁵ Some studies focused on the ADI-R algorithm items. According to Lecavalier et al, the EFA was performed to assess the ADI-R data of 226 children with ASD. The best solution was three-factor structures, similar to the original design of the ADI-R; however, the non-verbal communicated items were more related to the social domain.⁴⁶ Twelve subscales of the ADI-R had been surveyed and the EFA analysis supported bi-factor models and divided into socio-communication and stereotype behavior domains. The confirmatory factor analyses favored two or three-factor structures.⁴⁷ In present study, since the range of the scores of algorithm items (0, 1, 2) is narrow, the amount of estimated variance would be restricted. Nearly all of the social and non-verbal communication items and some of the repetitive behaviors items were scored 1 or 2. Similar to the findings reported by Lecavalier et al,⁴⁶ our study showed all non-verbal communication items were assigned to the same factor with social items and had high factor loadings. Verbal communication items were allocated to another two factors with moderate to high factor loadings. Most repetitive behaviors items were associated with the social items. The above discussion indicated most studies favored bi- or three-factor structure of the ADI-R. The item #39 and #68 were assigned to a different factor from the other repetitive behavior items. The item #39 (verbal rituals) evaluates the repetitive and ritual use of language. All 30 ASD participants were scored as “not applicable” for item #39. The real repetitive behaviors in those non-verbal ASD participants might not have been evaluated correctly in this item, and may explain why item #39 showed inconsistent factor assignment from other repetitive behaviors items. We noted 45 ASD participants were scored as “0” (untransformed score) for item #68 (circumscribed interests), and it means no circumscribed interests is ever found. The number of the ASD participants scored as “0” for item #68 was much higher than for the other repetitive behavior items, for example, only 26 ASD participants were scored as “0” for item #67 (unusual preoccupations). High percentage (78.3%) of ASD participants in present study was comorbid with intellectual disability. Lower cognitive function may limit the development of more complex restricted interests. The discrepancy in the results of scoring may account for why the item #68 was assigned to a different factor from other repetitive behavior items.

Validity

The validity of diagnostic accuracy is measured by sensitivity and specificity. The ROC analysis was performed to estimate the discriminant ability of the diagnostic test. The larger the area under the ROC curve is, the better the discriminant ability. All domains measured in the present study showed good to excellent ROC results, sensitivity, specificity and other diagnostic parameters. For the correct diagnosis of ASD, the present study showed the sensitivity of 97.30%, the specificity of 100%, PPV of 100%, and NPV of 94.29%. The results indicated that the Chinese version of the ADI-R can effectively differentiate individuals with ASD from those without ASD. The original English version of the ADI-R validation study showed the sensitivity is 96% and the specificity is 92%.¹⁰ The ADI-R was then translated to several other languages in non-English-speaking countries and validation studies were published. The interpretation of validation studies from different countries or areas might take the variable sample size, characteristics of cases and controls, and cultural effects into consideration.^{13–16,20,48} Smaller samples from mainland China (n=50), Brazil (n=40) and Greek (n=77) showed adequate sensitivity (88–100%),^13,14,48 but the specificity of Greek study was lower (69%). The Brazil study assessed participants aged from 7 to 18 years old and reported perfect sensitivity and specificity (100%, 100%). The recent validation study from Korea recruited total 1271 participants aged from 24 months to 34 years old, and they also obtained good validity data as the sensitivity is 86.06% and the specificity is 99.55%.¹⁶ Regarding the Japanese version of ADI-R, the author recruited 317 participants aged from 2 to 19 years old, and the sensitivity is 92% and the specificity is 89%.¹⁵ The subgroup analysis found much lower sensitivity (55%) in young children aged younger than 5 years old (n=89). Consistent findings from previous studies indicated the ADI-R performed in toddlers and preschool children might show decreased sensitivity.^18,19 One American study evaluated toddlers aged from 16 to 30 months old (n=45) and reported lower sensitivity and specificity (52.8%, 66.7%) for ADI (combined data of ADI-R and ADI Toddler Form).¹⁹ Previous systematic review reported the sensitivity is 91% in participants over 3 years old, while the sensitivity reduced to 82% in participants aged less than 3 years old.¹⁸ Overall, we showed that the Chinese version of the ADI-R demonstrated high validity, similar to the original English version. For clinicians in Taiwan, this could be a trustworthy tool for future ASD diagnosis. The Chinese version could also be modified and adapted for use in other Chinese-speaking regions and communities in the world.

Cultural Effect

On the premise that previous studies had proved that the diagnosis of ASD could be established according to the standard diagnostic criteria across-culturally,⁴⁹ some cultural effects might exist. In present study, we had some findings that related to cultural effect and language difference. First, the cut-off point (16) estimated for social domain (Table 3) was much higher than the cut-off point (10) suggested by the original version of the ADI-R. Moreover, much fewer ASD participants (n=12) with verbal ability were reported to have neologism/idiosyncratic language (#38). More studies to replicate and verify if these findings were related to cultural effect and language difference in the future are needed.

Further studies with larger sample sizes, to investigate the application of the Chinese version of the ADI-R in toddlers and preschool children and to compare the ASD samples with control samples with other developmental disorders are also important in the future.

Limitations and Strengths

There are limitations to this study. First, the participants of our study were from Northern Taiwan. Considering the cultural difference and regional language variation, our findings may not be generalized to other Chinese-speaking countries or regions. Second, the present study did not measure inter-rater reliability, and future studies may consider comparisons of diagnostic results from different interviewers. Third, the small sample size was the major limitation of this study, and it also caused potential selection bias of included participants. A wide range of age and incomparable gender ratio between groups might also confound results. To expand the sample size of the control group or matched cases is needed in future work. Moreover, because the sample size was limited, we did not perform subgroup analyses of the psychometric properties of different age levels. Nevertheless, some features of our study constitute its strengths. We explored the psychometric properties of the Chinese version of the ADI-R from many aspects including discriminant validity, test–retest reliability, ROC curve, sensitivity, specificity, and several diagnostic parameters. Our results indicated satisfactory performance of the Chinese version of the ADI-R. Moreover, we provided factor structure analysis, and the results supported three factor models.

Conclusion

Our investigation showed satisfactory psychometric properties of the Chinese version of the ADI-R. It proved that the Chinese version of the ADI-R is a reliable and valid instrument in differentiating people with ASD from people without ASD.