Application of Attention-Deficit/Hyperactivity Disorder Diagnostic Tools: Strengths and Weaknesses of the Korean ADHD Rating Scale and Continuous Performance Test

Introduction

Attention-deficit/hyperactivity disorder (ADHD) is a neuropsychiatric condition common in children and adolescents. Children and adolescents with ADHD often experience significant impairment in academic functioning, as well as in social and interpersonal situations.^1,2 ADHD is frequently associated with comorbid disorders such as learning disorders, anxiety disorders, mood disorders, and disruptive behavior disorders, and up to 60% of school-aged children with ADHD are symptomatic into adulthood.^3–5 Therefore, screening and diagnosis of ADHD are expected to aid in the improvement of a child’s impaired function and contribute to a better quality of life.

In general, multiple measurement tools such as parent-teacher interviews, behavioral rating scales, direct observations, and clinical examinations are used to evaluate patients with ADHD.⁶ Importantly, the ADHD Rating Scale (ARS),⁷ Conner’s Rating Scale,⁸ Attention Deficit Disorder with Hyperactivity (ADD-H) Comprehensive Teacher Rating Scale (ACTeRS),⁹ and Child Behavior Checklist (CBCL)¹⁰ are widely used behavioral rating scales with limited clinical relevance, but high value in the quantitative assessment of neurobehavioral problems in children with ADHD. Clinical examinations are used to measure neuropsychological deficits in children with ADHD, as many studies conducted over the past 30 years support the view that ADHD is a heterogeneous disorder associated with multiple neurological deficits.¹¹ For example, independent oscillatory patterns determine performance fluctuations in children with attention-deficit/hyperactivity disorder.¹² Further, sleep problems are a common feature in ADHD which can lead to various cognitive deficits,¹³ and these neurobehavioral deficits are well documented. Executive functions, which are neurocognitive processes that maintain the appropriate problem-solving skills necessary to attain a future goal, are also affected in ADHD.¹⁴ Some of the neuropsychological tests that measure various aspects of executive functioning in ADHD include: Continuous performance test (CPT) commission errors [Response inhibition]; CPT errors and hit reaction time variability [Vigilance]; the Wisconsin card sorting test [Set shifting]; the Trail-making test, Part B [Set shifting]; the Tower of Hanoi/London test [Planning ability]; the Porteus Maze test [Planning ability]; the Rey-Osterreith complex figure test [Planning/organization]; the Working memory sentence span test [Working verbal memory]; the Digits backward test [Working verbal memory]; the Self-ordered pointing test [Spatial working memory]; and the CANTAB spatial working memory test [Spatial working memory].¹⁵ These tests can be used to reveal important brain mechanisms underpinning deviant cognitive traits in ADHD.

We chose to focus on the ARS and the CPT, which are frequently used in clinical settings. The ARS is a behavioral assessment questionnaire developed by DuPaul in 1991 based on the Diagnosis and Statistical Manual of Mental Disorders (DSM). The Korean translation of the questionnaire, the Korean ADHD Rating Scale (K-ARS), has been established as reliable and valid.¹⁶ The ARS has proven to be a useful tool for the screening of ADHD⁷ and is particularly useful in distinguishing different clinical presentations that reflect the two-dimensional structure of ADHD symptoms (predominantly inattentive presentation, predominantly hyperactive/impulsive presentation, and combined presentation).^17,18 It is also known to have excellent sensitivity to drug response.¹⁷ Such behavioral rating scales are relatively easy to use and obtain information regarding the child’s behavior in different situations from various observers. It should be noted, however, that ratings can be inconsistent across observers or environments.¹⁹ The Integrated Visual and Auditory Plus (IVA+Plus), a kind of CPT, is a test designed to aid in the diagnosis of ADHD. As the test requires split attention/cross-modal divided attention, it significantly increases the individual’s working memory load during the test.^20,21 Studies have shown divided attention to be related to the function of the prefrontal region of the brain.^22–24 Considering ADHD impairs an individual’s ability to control and organize, which is associated with the prefrontal cortex, and abnormalities in prefrontal lobe function are frequently observed in the functional brain imaging of patients with ADHD,^25–27 IVA+Plus can provide useful clinical information related to prefrontal dysfunction in ADHD. Additionally, according to a recent study, reaction time variability is related to default mode interference.²⁸ Studies showed that ADHD is related to inadequate control of the default mode network (DMN).^29,30 They suggested that ADHD’s inattention may be due to inadequate suppression of DMN activities and is therefore associated with slower and more variable responses.³¹

In this study, we compared three groups (attention-deficit/hyperactivity disorder [ADHD], ADHD-not otherwise specified [NOS], and normal control [NC]) using the K-ARS and the IVA+Plus. A few authors have posited that DSM-IV ADHD diagnosis thresholds (ie, 6 of 9 symptoms per symptom domain) were appropriate for young children but not effective for identifying adolescents and adults experiencing ADHD-related impairments.^32,33 For this reason, prior to DSM-5, many researchers and clinicians used the ADHD Not Otherwise Specified diagnosis. This was subsequently revised in the DSM-5, in which the minimum number of symptoms in either symptom domain, required for older adolescents and adults to be diagnosed with ADHD, was reduced from six to five.³⁴ A previous study has shown that patients with NOS can be distinguished from NCs using quantitative electroencephalography, resulting in the early detection of borderline cases and thereby reducing the social difficulties and functional impairments related to the condition.³⁵ Another previous study reported clinical improvement after atomoxetine was administered to patients with later onset of NOS or insufficient symptoms to fully diagnose ADHD.³⁶ Despite these findings, NOS has not been a major focus; it has fewer clinical symptoms than ADHD, and few studies have examined its diagnosis or treatment. Recently, however, the prevalence of NOS was reported to be higher than that of ADHD,³⁷ and another study indicated that NOS may eventually develop into ADHD.³⁸ To reduce treatment costs and improve these patients’ quality of life, it is increasingly important to distinguish patients with NOS from NCs. Therefore, in this study, we administered the representative diagnostic tools for evaluating ADHD to individuals in a clinical group, a sub-threshold group, and a control group. First, we wanted to determine the clinical and neurobehavioral abilities of each assessment tool. Then, we assessed whether these tools can be helpful in discriminating the subthreshold group.

Patients and Methods

Subjects

Individuals who visited our child and adolescent psychiatric clinic at Daegu Catholic University Hospital from 2018 to 2020 were considered for inclusion in this study. Inclusion criteria were: diagnosed with ADHD; and between 7 and 12 years of age. ADHD diagnosis was based on the Korean version of the Diagnostic Interview Schedule for Children Version IV (DISC-IV), which is a structured interview tool, and these diagnoses were confirmed by more than one child-and-adolescent psychiatrist. Children with brain damage, a neurological disorder, a genetic disorder, substance dependence, epilepsy, or any other mental disorder reported during the collection of personal history or anamnesis were excluded. Children who exhibited an IQ of 70 or lower according to the Korean–Wechsler Intelligence Scale for Children (Fourth Edition) or who were receiving drug treatment were also excluded. Normal controls were recruited through the use of posters. Posters were attached to child and adolescent outpatient clinics, student counseling centers, and schools. Those who read the explanation and voluntarily wished to participate were enrolled in this study. The exclusion diagnosis of ADHD was made using the DISC-IV. Based on the inclusion and exclusion criteria, 74 participants (58 males and 16 females) were finally enrolled. These participants were classified into three groups – ADHD, NOS, or NC – to analyze the diagnostic discriminative ability of the K-ARS and IVA+Plus. Participants were categorized into the NOS group based on suggestions made by the developers of the DISC (P. Fisher).^39,40 In other words, while these subjects did not meet full criteria for ADHD, they had at least half (3 to 5) of the symptoms present in either ADHD dimension (inattentive or hyperactive/impulsive); presented the symptoms in two or more settings (ADHD Criterion C); and had either a moderate rating of impairment in at least two of the six domains measured or a severe rating in at least one domain (Criterion D). This definition is in line with the vague definition of ADHD-NOS in DSM-IV.⁴¹ The K-ARS was completed by the parents of the participants distantly. DISC-IV was performed by pediatric psychiatrists, using face-to-face interviews. The IVA+Plus was administered to the participants by a technician over the course of 15 mins, during school days.

Detailed information regarding the study was provided to the children and their parents or guardians. Written consent for the children’s participation and for medical use of the test results was obtained from all of the participants’ parents or guardians. In addition, all of the children participated voluntarily and provided written informed consent for participation. This study was approved by the Institutional Review Board (IRB) of Daegu Catholic University Medical Center (DCUMC IRB approval No. CR-18-096) and was performed in accordance with the Declaration of Helsinki (World Medical Association: Ethical Principles for Medical Research Involving Human Subjects, 1964).

Korean Version of Diagnostic Interview Schedule for Children Version IV

The DISC-IV is a structured diagnostic tool developed for use in epidemiological studies of children and adolescents. Through the use of the DISC-IV, we were able to determine the existence of nine symptoms of attention/concentration issues and nine symptoms of hyperactivity–impulsivity during the preceding 6 months. When symptoms were noted, detailed questions, such as whether the symptoms were observed at home or school, were asked. The DISC-IV was revised in 2000 by the US National Institute of Mental Health. The present study used the Korean version of the DISC-IV, which was translated into Korean in 2007 and subsequently underwent reliability and validity verification.⁴²

Korean Parent and Teacher ADHD Rating Scale (K-ARS)

The K-ARS is used to evaluate ADHD symptoms in school-age children. It consists of 18 items based on the DSM-IV ADHD diagnostic criteria. Each item is rated on a 4-point Likert scale from 0 (never) to 3 (almost every day) according to the frequency of the problem behavior. There are nine items in each category; the sum of the scores of odd-numbered items represents inattentive symptoms, while the sum of the scores of even-numbered items reflects hyperactive-impulsive symptoms. The total score ranges from 0 to 54. The K-ARS is completed by the parents of children with suspected ADHD, and it has been standardized as an ADHD screening tool for Korean children.^19,43

Continuous Performance Test (CPT)

The CPT is used to assess the functioning level of the attention/arousal system. We used the Intermediate Visual and Auditory Continuous Performance Test (IVA CPT; BrainTrain, Inc., Richmond, VA, USA, www.braintrain.com) to obtain behavioral measures of attention.⁴⁴

Two stimuli (ie, numbers “1” and “2”) are presented pseudorandomly, alternating between visual and auditory modalities over 500 times. The examinee is expected to respond only to the target (“1”) and not to the non-target (“2”). The basic IVA+Plus provides a number of scores: six global composites (Full-Scale Response Control Quotient, Auditory Response Control Quotient, Visual Response Control Quotient, Full-Scale Attention Quotient, Auditory Attention Quotient, and Visual Attention Quotient) and 22 other scales measuring four broad categories (Response Control, Attention, Attribute, and Symptomatic).⁴⁵ Primary scales make up the Quotient scales.⁴⁶ In the case of Response Control Quotient, these include Prudence [assesses impulsivity/response inhibition; combines three types of commission errors], Consistency [assesses the ability to stay on task; involves reliability and variability of response times], and Stamina [assesses sustained attention; compares the mean reaction time of correct responses during the first 200 responses to that during the last 200 responses].⁴⁵ Primary scales for the Attention Quotients include Vigilance [Measures attention; combines two different types of omission errors], Focus [Assesses variability of response speed], and Speed [Provides information on the average reaction time for all correct responses].⁴⁵ Attribute and Symptomatic scores provide information on test-taking characteristics. Overall, the test provides us with a very large number of possible scores. In this study, we used Response Control Quotients and Attention Quotients as an IVA+Plus hyperactivity-impulsiveness scale and an IVA+Plus Inattention scale, respectively, for analysis.

IVA+Plus is an interesting tool that combines the capabilities of different CPTs.⁴⁵ First, it provides variable interstimulus intervals designed to capture both impulse control and vigilance deficits, requiring the subjects to respond almost continuously, while at other times, to respond infrequently. Second, given that auditory and visual CPT can detect attention deficit in other groups, stimuli provided in both auditory and visual modalities can increase sensitivity. It can also be useful for evaluating populations that are expected to have difficulty in cross-modal switching, such as schizophrenic patients.⁴⁷ In addition, it has the ability to provide screening for simple reaction time deficits, including pre- and post-reaction times for evaluation of fatigue, as well as containing a number of validity indicators. Additionally, only a single non-target (“2”) is presented, making identification simpler than other CPTs.⁴⁸ It can also be useful for detecting symptoms of malingering.⁴⁹

Statistical Analysis

All of the values are expressed as the mean and standard deviation (SD). First, an analysis of covariance (ANCOVA) was conducted on the K-ARS and IVA+Plus results for the ADHD, NOS, and NC groups to adjust for the effects of covariate variables (sex and age). Statistical significance was defined as p < 0.05. The Bonferroni comparisons were conducted as post hoc analyses after ANCOVA. Second, the diagnostic performance of a test or its accuracy in discriminating patients with ADHD or NOS from NCs was evaluated using a receiver operating characteristic (ROC) curve analysis. In a ROC curve analysis, an instrument’s effectiveness is assessed by evaluating the accuracy with which it discriminates between two groups. Accuracy is measured by the area under the curve (AUC). All data were analyzed using IBM SPSS Statistics software for Windows, version 25.0 (IBM Co., Armonk, NY, USA).

Results

Demographic and Clinical Characteristics

Participants in this study included 58 boys (8.9±1.21 years of age) and 16 girls (8.7±1.18 years of age), and there was no significant difference in age between the sexes. The mean K-ARS score was 29.47±12.57 for boys and 23.19±14.40 for girls, with the average of both boys and girls higher than 20. There was no statistically significant difference in scores between the sexes (t=1.714, p=0.091). The three groups classified based on DISC-IV results were the ADHD group (n=29, 24 boys, 5 girls), NOS group (n=33, 26 boys, 7 girls) and NC group (n=12, 8 boys, 4 girls) (Table 1). The mean age for each group was as follows: NC group (8.5±0.88 years of age), NOS group (9.2±1.30 years of age), and ADHD group (8.6±1.11 years of age). The subtypes of the ADHD group included: ADHD predominantly Inattentive subtype (n=14, 48.3%), ADHD predominantly hyperactive/impulsive subtype (n=8, 27.6%), and ADHD combined subtype (n=7, 24.1%) (Table 1).

Table 1 Demographic Characteristics of the Participants

Comparison of K-ARS and IVA+Plus Scores

After controlling for sex and age using ANCOVA, the average K-ARS and IVA+Plus scores of the three groups were compared (Figure 1). The K-ARS score (F=21.75, p<0.001), IVA+Plus hyperactivity–impulsiveness scale score (F=4.01, p=0.022) and inattention scale score (F=3.80, p=0.027) differed significantly among the three groups. The K-ARS score was significantly lower in the NC group (11.58±8.81) than in the NOS (27.36±10.71) (p<0.001) and ADHD (35.79±10.49) (p<0.001) groups. The NOS group (27.36±10.71) also had a significantly lower score than the ADHD group (35.79±10.49) (p=0.004). For the IVA+Plus, both the hyperactivity–impulsiveness and inattention scale scores were significantly lower in the ADHD group (77.03±32.77 and 77.24±32.60, respectively) than in the NC group (101.67±8.04 and 101.08±15.51, respectively) (p=0.008 and 0.009, respectively). However, no significant differences were observed between the NC and NOS groups or between the NOS and ADHD groups.

Figure 1 The result of comparisons between ADHD, NOS, and NC group using K-ARS and IVA+Plus.

ROC Curve Analyses for the K-ARS and IVA+Plus

To evaluate the usefulness of the K-ARS and IVA+Plus, we performed an ROC curve analysis (Figure 2). To evaluate the screening ability of each tool, four combinations were analyzed: ADHD vs NC, NOS vs NC, ADHD+NOS vs NC, and ADHD vs NOS+NC.

Figure 2 Receiver operating curves for the K-ARS and IVA+Plus in each group.

In the ROC curve analysis for the K-ARS, the AUCs were 0.960 (ADHD vs NC), 0.885 (NOS vs NC), 0.920 (ADHD+NOS vs NC), and 0.779 (ADHD vs NOS+NC) (Table 1). In the ROC curve analysis for the IVA+Plus hyperactivity–impulsiveness scale, the AUCs were 0.740 (ADHD vs NC), 0.643 (NOS vs NC), 0.688 (ADHD+NOS vs NC), 0.626 (ADHD vs NOS+NC); for the IVA+Plus inattention scale, the AUCs were 0.731 (ADHD vs NC), 0.658 (NOS vs NC), 0.692 (ADHD+NOS vs NC), and 0.625 (ADHD+NOS vs NC) (Table 2).

Table 2 AUC in K-ARS and IVA+Plus Among Each Group

Discussion

In this study, the ADHD, NOS, and NC groups, defined using the DISC-IV, comprised children and adolescents who volunteered to participate in ADHD research in a hospital and who were tested using both the K-ARS and IVA+Plus. Our goal was to evaluate the strengths and weaknesses of each tool and to determine their discrimination ability. The demographic characteristics of the subjects in each group did not differ significantly according to age or sex. There was a male-to-female ratio of 3:1, similar to that reported in previous studies.⁵⁰

The ANCOVA revealed statistically significant differences in K-ARS scores in the comparisons of NC-NOS, NC-ADHD, and NOS-ADHD; however, a significant difference in IVA+Plus was observed only in the comparison of NC-ADHD. This suggests that the K-ARS is more discriminatory than the IVA+Plus in the classification of NOS. Similar patterns appeared in the ROC analyses: The K-ARS was excellent in distinguishing the ADHD group from the NC group (AUC=0.960) and the ADHD+NOS group from the NC group (AUC=0.920). Its ability to discriminate the NOS group from the NC group (AUC=0.885) was also good, and its ability to discriminate the ADHD group from the NOS+NC group was fair (AUC=0.779) (Figure 2). The AUC for distinguishing NC vs ADHD+NOS (AUC=0.920) was greater than the AUC for distinguishing ADHD vs NOS+NC (AUC=0.779). Although at a subthreshold level, the NOS group showed characteristics closer to those of the ADHD group than to those of the NC group. These results indicate the K-ARS may better discriminate between children with ADHD symptoms and NCs. In fact, the K-ARS is recognized as an economical and efficient tool for screening ADHD and is widely used in research and in the clinical field.¹⁸ This tool is a self-rating scale for parents or teachers. It provides normative data that can be used to objectively evaluate the observations of children’s behavior compared to the behavior of their peers of the same age and gender. In addition, the CPT, which is now widely used as an evaluation tool in clinical settings such as hospitals, fails to meet the sensitivity and specificity requirements suitable for diagnosis.⁵¹ As such, the most useful method to diagnose ADHD is via an interview or self-report scale completed by parents or teachers.⁵² It should be noted, however, that the K-ARS can be rated inconsistently depending on who evaluates the child’s behavior or the scenes he or she observes. In fact, differences have been reported in the K-ARS scores of teachers and parents who evaluated the same child, and studies have also shown that teachers tend to give significantly higher scores than parents in the inattention scale.¹⁹ The K-ARS also takes a dichotomous and categorical approach, which facilitates diagnosis using the DSM system, but this can make it difficult to assess the child as a whole.⁵³ It is thought that diagnostic aids that are objective and capable of continuous measurement may compensate for the shortcomings of the K-ARS. Our ROC analysis of the IVA+Plus revealed generally poor discrimination power, but it showed fair discrimination (AUC= 0.740 and 0.731 for the hyperactivity–impulsiveness and inattention scales, respectively) between the NC and ADHD groups (Table 1). This result confirms the idea that the IVA+Plus is better suited as an aid for confirming ADHD than for screening it. Indeed, previous studies on the CPT have shown mixed results regarding its usefulness for the differential diagnosis of ADHD. Since the introduction of the CPT, many researchers have used it to test for neuro-psychological performance deficits among populations with mental illness, and even more frequently among populations with ADHD.⁵⁴ Early research results have indicated that children with ADHD exhibited performance deficits on the CPT,⁵⁵ and several studies have demonstrated performance deficits measured by omission- and commission-error rates in children with ADHD compared to NCs.⁵⁶ However, not all studies showed differences between those groups,^57–60 and controversy remains as to whether the CPT can differentiate ADHD from other psychopathologies.⁶¹ Another study noted that, despite the strengths of the CPT to measure cognitive deficiencies, the Conners CPT is not an effective diagnostic tool for ADHD.⁵⁸ The author added that CPT should be used in combination with other tests rather than by itself. The study, which tested whether variables of the CPT correspond to ADHD behavior topography, revealed that each variable seemed to be related to the whole constellation rather than to a specific domain of ADHD. This is contrary to a prior assumption that commission error indicates impulsivity and omission error indicates inattention; it suggests that an overall interpretation is needed rather than a focus on the separate results of the CPT. Currently, the CPT is included in the neuropsychological battery for ADHD, but it should be used with the above points in mind.

Moreover, the discrimination power of the CPT was low when the NOS group was included. The discriminative power of the K-ARS was good when the NC, ADHD, and NOS groups were included. In particular, the K-ARS could distinguish NOS from NC pretty well (AUC = 0.885). In Korea, 9% of children have subthreshold attention-deficit and hyperactivity symptoms,⁶² and studies have shown that these children also develop the behavioral problems and emotional disorders commonly seen with ADHD.⁶³ K-ARS is thought to be useful in distinguishing between clinical-level and borderline ADHD.

This study had some limitations. First, the samples between the three groups were inconsistent and the groups were not matched for age and gender, so we corrected this inconsistency by conducting the analysis using covariate variables. Additionally, the sample sizes investigated here were quite small, especially considering that the ARS is more appropriate for use in large epidemiological studies. Similarly, results from CPT mandate using a larger and homogenous sample size, thus it was difficult to obtain statistically meaningful results. Second, it is possible that the children in the NC group who participated in the ADHD study through public recruitment tended to be somewhat more distracted and careless than normal children in the community. Despite the above limitations, this study is expected to provide clinically useful information for the diagnosis of ADHD, especially when classifying individuals into ADHD, NOS, and NC groups.

Conclusion

In this study, the K-ARS and IVA+Plus, which are commonly used tools for the evaluation of ADHD, were compared and analyzed, and the utility and limitations of each tool were examined. We found the K-ARS had the advantage of being able to discriminate not only the psychopathological group but also the subthreshold group. We found that the ability of the IVA+Plus to distinguish the ADHD group from the NC group was good, and it allowed for the evaluation of cognitive deficiencies in the patient group. It is more objective than the K-ARS. However, when using the CPT to evaluate ADHD, it should be used as part of an overall battery rather than on its own with a focus on each independent variable. Clinicians should ensure they understand the characteristics of each tool and apply them appropriately when diagnosing ADHD.