Do Teachers Confirm Parent’s Ratings of ADHD DSM-IV Criteria? A Study of a Mexican Population

Plain Language Summary

In the present study, we observed that parents and teachers agree at a minimum level when rating the DSM-IV ADHD criteria in Mexican school-aged children. Agreement increased slightly to moderate levels only when rating children in the first school grades, but it remained low when analyzing boys and girls separately, and when considering individual ADHD symptoms. Inattention symptoms were reported more frequently by teachers. The discrepancies here reported may reflect how behaviors are expressed by children and perceived by adults in ways that depend on the characteristics of the setting.

Introduction

Attention deficit hyperactivity disorder (ADHD) is one of the most common neurodevelopmental disorders in childhood with a prevalence of around 5%,¹ though variation is large, from 1% to 20%.^2,3 The core of ADHD diagnoses is the frequency and range of 18 behaviors related to inattention (9), hyperactivity (6), and impulsivity (3) that affect a child’s development and functioning. To be diagnosed with this disorder, the child must display at least six inattention or hyperactivity-impulsivity behaviors for at least 6 months at a higher frequency than expected for her/his developmental level.⁴ Usually, parents or main caregivers complete behavioral rating scales to quantify the frequency of these behaviors. Less often, teachers are asked to complete these scales according to the behaviors that the child manifests in the school setting.⁵ The other four diagnostic criteria of ADHD derive from Criterion A. Criterion B was modified between the fourth and fifth editions of the DSM, as the former indicates that symptoms should be present before age 7⁶ but DSM-5 extends this age to 12 years.⁴ Criterion C stipulates that those behaviors must be observed in more than one setting (eg, home, school, work, with friends or relatives, or in other activities), while Criterion D requires clear evidence that symptoms interfere with, or reduce the quality of an individual’s social, academic, or occupational functioning. Criterion E, finally, stipulates that the symptoms cannot be explained by any other mental disorder.⁴

Criterion C is somewhat unclear. Normally, confirming ADHD behaviors across settings cannot be reflected accurately without consulting at least two informants, one for each context.⁷ This procedure, however, is not always followed.⁵ When it has been performed in this way, several studies have shown low agreement when comparing the reports elaborated by parents vis-à-vis those of teachers.^8–10 In fact, the DSM-5⁴ points out that (i) concordance between informants seems to vary, and (ii) the variation in symptoms usually observed depends on the context analyzed. In reality, several elements may enhance or diminish the expression of certain behaviors. Motivational models have shown that under-stimulating environments exacerbate children’s expression of ADHD symptoms, such as disinhibition, whereas in more compelling and motivating contexts the differences observed between ADHD cases and controls diminish or even disappear.^11–13 Moreover, the norms of each context can influence the perception of behaviors. For instance, in the school setting, where students are frequently asked to stay quiet, energetic behaviors may be reported as impairing; whereas in sports or other settings involving physical activities, those same behaviors may be more aligned with the rules and may even be considered functional.¹¹

Furthermore, concordance varies according to ADHD presentation. Interestingly, predominantly hyperactive/impulsive presentations (ADHD-HI) achieve higher concordance than the predominantly inattentive form called ADHD-I.^9,14,15 This suggests that hyperactivity/impulsivity traits are more likely to be perceived by both informants (parents and teachers) than those associated with inattention. Nonetheless, moderate correlations between ratings by parents and teachers of the symptoms of both presentations (inattention and hyperactivity/impulsivity) were recently reported in a meta-analysis.¹⁰ If environmental behavioral demands vary, then ADHD children could show greater difficulties in adapting their behavior accordingly. For example, activities that require cognitive effort and behavioral control are more numerous in the school than the home, a fact that leads to an increase in manifestations of ADHD behaviors in the former setting. In addition, the frequency of ADHD behaviors tends to decrease with age,^16–20 so a child’s age could also mediate agreement between raters in distinct contexts, in the sense that the higher the frequency of symptoms, the higher the agreement.

To tackle the issue of inter-rater agreement, some studies determine the agreement between parents’ and teachers’ by contrasting the final result from each informant’s report, that is, the presence or absence of ADHD,¹⁵ or the total scores from the inattention and hyperactive-impulsive dimensions.^9,14,21 Other approaches have combined the responses to individual items. The AND-rule counts a behavior as present if both informants positively endorse it.²² This makes it possible to detect the behaviors that are more likely to be identified in both the home and school settings (when both parents’ and teachers’ questionnaires are used). The OR-rule, in contrast, considers a behavior as present if either parent or teacher indicates it.¹⁵ This rule is usually applied when low agreement is expected.²³

We consider that the data from two informants (a parent and a teacher) are essential not only for identifying but also for qualifying ADHD in clinical practice. In this sense, determining the level of agreement between raters could provide additional, valuable information about children’s behavior –in both the family and school settings– on the presence/absence of ADHD and on the reports of individual symptoms. The aim of the present study, therefore, was to identify the agreement rate between parents’ and teachers’ reports of the DSM-IV ADHD criteria in a large sample of Mexican school-age children from public elementary schools. As suggested by the literature for other cultures, we hypothesized that the study will find low agreement between parents and teachers in Mexican school-age children. We also hypothesized that certain characteristics of the children, such as age or sex, could have an effect on the level of agreement. Specifically, the degree of agreement between these two informants may vary as a function of elementary school level of the children with greater agreement being found in the lower grades since hyperactivity symptoms are more noticeable in younger children. While we expected that both informants would observe more ADHD symptoms in boys than girls, we predicted that no differences in agreement between informants in relation to the sex of the children would appear. Moreover, when applying the AND/OR rules to rate individual symptoms, the OR-rule led us to expect a higher frequency of reports of hyperactive symptoms in children, while the AND-rule inclined us to predict stronger agreement between raters for hyperactivity symptoms, since these can be more disruptive and noticeable than those associated with inattention. Finally, previous studies have reported that parents classify more children with ADHD than teachers,^{9,15,21,24,25} but a study with a population more similar to ours²⁶ reported an opposite trend. Under these circumstances, we have no support for predicting a trend for the Mexican children evaluated in the present study. Thus, our findings from Mexican parents and teachers on issues concerning the frequency of ADHD behaviors in children should enhance our knowledge in the field of transcultural psychology.

Materials and Methods

Sample

This study derives from a broader research project conducted in public elementary schools in the city of Guadalajara. With the data obtained, we first determined the prevalence of ADHD in children according to parents’ reports.²⁷ This second analysis was designed to determine parent/teacher agreement. We followed the standard procedure for working in public elementary schools in Mexico. First, the General Direction of Elementary Schools (in Spanish, DGEP) approved conducting the research in four educational districts of the Metropolitan Area of Guadalajara and put us in contact with the corresponding supervisors who, in turn, provided access to 16 schools. This procedure ensures the participation of principals and teachers in the study after they receive an oral explanation of the research.²⁸

A total of 202 elementary school groups were visited. We delivered two letters to 6639 children: the parental consent form and the parents ADHD questionnaire. Only those children who returned both letters were included in the sample. A total of 4399 parents responded to our request. On the basis of their reports, we identified 544 children as having the number of symptoms required for ADHD diagnosis.²⁹ We then randomly identified 544 children from the same sample and classrooms as being free of ADHD symptoms, according to their parents’ reports. With the permission granted by district supervisors and principals, the teachers of these 1088 children were asked to fill out a similar rating scale for each child. To avoid any answer bias, parents and teachers were told that we were studying children’s behavioral characteristics. The teachers were not informed of the ratings given by the parents or the resulting status, that is, with or without ADHD.

From the total of 1088 questionnaires delivered to the teachers, we recovered 789 completed forms (72%). Of these, 385 (48.8%) corresponded to children identified by their parents as having at least 6 ADHD symptoms of one presentation. The remaining 404 (51.2%) referred to children who were classified as being free of these symptoms according to parents’ reports. Girls (335) represented 42.5% of the sample. Mean age was 8.4 (±1.7) years. Data analysis considered three groups classified by school level: Group 1 (grades 1–2), Group 2 (grades 3–4), and Group 3 (grades 5–6) (see Table 1). No differences were found in the analysis of the distribution of children by school grade or sex (χ²= 3.4, p=0.18). For practical reasons, in the following sections we refer to the children who met the ADHD criteria in the questionnaires as “ADHD cases”, although no formal, clinical diagnoses were performed.

Table 1 Sample Distribution by Sex and Grade

The Ethics Committee of the Institute of Neurosciences of the University of Guadalajara approved this study (registration no. ET052010-82) in accordance with the principles of the Helsinki Declaration and the General Health Law of Mexico for health-related research with human beings.

Materials

The parents and teachers were instructed to answer a questionnaire that included the 18 symptoms listed in Criterion A of the DSM-IV criteria for ADHD diagnoses⁶; thus, they rated 9 inattention behaviors (Criteria A1), 6 for hyperactivity, and 3 for impulsivity (Criteria A2), according to how often they had observed each behavior in their child over the preceding six months (“never” =0; “sometimes”=1; “often”= 2; “very often”= 3). In the design of the rating scale, we alternated questions from the two dimensions (eg,1a, 2a, 1b, 2b …). The presence of Criteria B, C, and D was assessed only by parents, by a single item for each one, as follows: B) did you observe the inattentive or hyperactive-impulsive symptoms rated as “often” and “very often” before your child was 7 years old?; C) were these behaviors present in two or more settings? (eg home, school, with friends, etc.); and D) do these behaviors substantially interfere with, or reduce the quality of, your child’s social, academic, or occupational functioning?

Data Analysis

Analyses were conducted in R.³⁰ For this study, we used the current classification proposed by both DSM-IV⁶ and DSM-5⁴ (ie, more than six symptoms of inattention or hyperactivity/impulsivity to identify a potential case of ADHD). At present, this is the most common classification employed. To complement data exploration, we calculated the dimensional scores(the sum of the scores given to each symptom, from 0 to 3). Based on the former, we determined the internal consistency (Cronbach’s Alpha) of parents’ and teachers’ questionnaires. We also explored the differences between raters’ scores, taking into account the children’s sex and school grade. General linear models (GLM) were elaborated using informant (parent/teacher) as within-subject factor and school grade and sex as between-subjects factors.

To determine parent-teacher agreement, we used the categorial approach, that is, the number of children that met, or failed to meet, Criterion A in the reports by parents and teachers reports. In what follows, we will refer to these children as ADHD or non-ADHD. Inter-rater reliability was determined using Cohen’s kappa (κ) coefficient, which reports values from 0 to 1.³¹ We determined Cohen’s κ for the entire sample and, subsequently, separately for boys and girls, for the school grade groups, and for the 18 ADHD symptoms.

In a more qualitative analysis, we explored how many children were reported as “often showing” each behavioral symptom (scored as “often”=2 or “very often”=3), in four different scenarios and considering: a) only parents’ reports; b) only teachers’ questionnaires; c) when both informants positively endorsed the presence of the symptom (AND-rule); and d) when at least one informant positively endorsed the presence of the symptom (OR-rule).¹⁶ We calculated the frequencies separately for ADHD and non-ADHD children to explore which symptoms were identified more frequently among those children deemed more likely to present ADHD.

Results

Cronbach’s alpha showed good internal reliability on inattention symptoms scored by parents (α= 0.92, 95% I [91, 93] and teachers (α = 0.96, 95% I [95, 96], as well as for hyperactivity-impulsivity symptoms (parents α = 0.91, 95% I [91, 92], teachers (α = 0.96, 95% I [95, 96].

Mixed GLM conducted with the total dimensional scores showed a main moderate effect of rater (F(1783)=125, p<0.001, η_p²=0.13), and low effects of school grade (F(2783)=2.411, p=0.09, partial squared eta=0.006) and sex (Sex= F(1783)=68.19, p<0.001, η_p²=0.08) (see Table 2). Significant interactions of informant/school grade were found (F(2783)=3.55, p=0.03, η_p²=0.009) since teachers tend to rate the G2 children lower, while parents rated them higher. Informant/sex interaction (F(2783)=5.39, p=0.02, η_p²=0.007) indicated that, compared to parents, teachers tended to identify less ADHD behaviors in girls, however these two interactions also reported small effect sizes (Figure 1).

Table 2 Comparisons of Rater’s Results

Figure 1 Interactions between informant, school grade group, and sex.

Parent/Teacher Agreement

As mentioned above, of the 789 children included in the original sample, 385 (48.8%) met Criterion A in their parents’ reports, and only 234 (29.6%) satisfied Criterion tA in their teachers’ reports. Teachers coincided with parents in 174 of the cases for the presence of ADHD (Figure 2). Higher agreement was evident when considering the non-ADHD cases. The inter-rater reliability coefficient indicated low agreement (κ=0.30, 95% CI [0.24, 0.36]) for the whole sample.

Figure 2 Agree-plot. Number of children identified as meeting (ADHD) and not meeting (non-ADHD) criterion A for ADHD diagnosis (DSM-54), according to parents’ and teachers’ report. Black squares represent the number of cases reported in agreement, white areas represent discordant cases.

Regarding sex, agreement remained low when analyzing the questionnaires on boys (κ=0.26, 95% CI [0.17–0.34]) and girls (κ=0.31, 95% CI [0.22–0.41]) separately.

As Table 3 shows, we observed variations in agreement when considering school grade. Moderate agreement between raters was observed in G1 (grades 1–2), but low agreement for G2 (grades 3–4) and G3 (grades 5–6). Teachers returned fewer questionnaires for the G2 children (67% vs 77% in G1 and 76% in G3). In proportion to the parents’ classification, teachers returned slightly more questionnaires of non-ADHD (74%) than ADHD (71%) cases. This difference was more evident in G1 (non-ADHD: 82%, ADHD: 71%; see Table 2). In general, teachers tended to report fewer ADHD cases in all three groups compared to the parents’ reports.

Table 3 Number of Questionnaires Delivered to and Recovered from Teachers, Classification According to Parent’s and Teachers’ Reports, and Inter-Rater Agreement, by School Grade

Parent/Teacher Ratings and Agreement Across Symptoms

Low inter-rater reliability was observed for all three symptom dimensions, that is, inattention (κ ranging from 0.18 to 0.34), hyperactivity (0.14–0.31), and impulsivity (0.16–0.19) (see Figure 3). When exploring the number of children that exhibited each symptom according to parents only, teachers only, both informants (AND-rule), and at least one (OR-rule), we noted that one inattention symptom was most often reported among the ADHD children: “(h) is often easily distracted by extraneous stimuli”⁴ (p. 59). In contrast, among the children that did not meet DSM Criterion A (non-ADHD), the hyperactivity symptom “(f) often talks excessively”⁴ (p. 59) was the one most often reported.

Figure 3 Analysis by symptoms in children meeting (ADHD) and not meeting (non-ADHD) criterion A for ADHD diagnosis (DSM-54). Columns show: (A) Inter-rater reliability coefficient (κ) for the entire sample, and (B) the frequency (in proportions) of children reported as having each symptom, by parents only (P), by teachers only (T), by both raters (AND), or by at least one (OR). In each column, the highest frequency (max) is in dark blue, and the lowest (min) in white.

It is noteworthy that teachers tended to report higher frequencies of inattention symptoms than parents, and that opposite patterns were observed among the impulsivity symptoms, as symptom (g) was the least frequent for teachers, whereas for parents the lowest was symptom (i).

Discussion

The aim of the present study was to determine the concordance between parents’ and teachers’ reports regarding DSM-IV ADHD criterion A (the list of symptoms) in Mexican school-age children from public elementary schools. The high values of internal consistency observed reveal that the answers to the questions in the instrument employed are closely related, suggesting that it is suitable for use with this study population. In general, we observed low concordance between raters. The analysis of consistency shows that reports from parents are related as a set, as are the teachers’ reports, independently, but when contrasting the two sets of reports, almost no concordance appeared. Only a moderate agreement was observed for the younger children in school grades 1 and 2, but for all other grades, agreement remained low. Both groups of informants tended to give higher scores to boys than girls, but the cases identified differed. The analyses by individual symptoms also showed low levels of inter-rater agreement, and only the inattention symptoms showed high frequencies (AND/or rules) among the children classified as ADHD.

In accordance with the literature, we expected to detect low inter-rater agreement.^4,8–10,21 Given that parents identified more children meeting Criterion A than teachers, we explored whether agreement changed when a more stringent approach was applied to parents’ ratings. In this case, we hypothesized that upon applying the full DSM criteria to parents’ reports, fewer cases would be identified by parents and that concordance with teachers would likely increase. The number of cases did, indeed, decrease (from 385 to 283 children) but the inter-rater reliability coefficient remained low (κ= 0.29, 95% C.I. [0.22, 0.36]). Thus, the low agreement cannot be explained only because parents are more prone than teachers to identify a child as meeting ADHD criteria. Our results thus suggest, that other factors must be involved. It is well known that certain sociocultural factors not only increase the risk of developing ADHD but also affect the likelihood of this conditions being recognized, diagnosed, and treated.³²

Two sources were identified in this regard. On the one hand, the specific features of a setting might either inhibit or enhance the expression of ADHD-related behaviors. Several factors have been identified as having the potential to modulate ADHD behaviors. These include frequent rewards for appropriate behavior,³³ close supervision, interaction in one-to-one situations, contexts with a clear structure,⁴ motivating factors in the environment (such as the novelty of the setting and, especially, interesting activities), the use of electronic devices,⁴ and even cognitive demand.^10,34 On the other, informant subjectivity impacts the scoring of these behaviors; for example, levels of permissiveness and tolerance for certain conducts can vary substantially from one person to another.

Thus, the context supplies the norms according to which a behavior may be measured and, later, labeled as dysfunctional;¹¹ but it also provides elements that can enhance or diminish certain behavioral difficulties. Motivational models highlight the role of context in the expression of ADHD by proposing that instead of a static attention deficit in all contexts, ADHD individuals have difficulties to appropriately regulate or adapt their behavior in conditions, or on tasks, that are not intrinsically rewarding or demanding.¹¹ These models are based on different responses to reward and to reward delay observed in ADHD individuals.^35,36 In summary, since behavioral manifestations depend on interaction with the environment, and the degree or form of stimulation, as well as the requirements imposed, they may not be identical in two different settings, or in relation to two different people. It is not surprising, then, that (i) a child may behave differently according to the setting, and/or (ii) her/his behavior could be interpreted distinctly at school versus the home. Studying Mexican family and school contexts in which these kinds of behaviors are observed would likely contribute to understanding the low agreement found in the present study.

At first sight, one might think that agreement between informants would make diagnoses more reliable, but this does not seem to be the case. Differences between raters should not necessarily be considered as a source of error since they may simply reflect the range of behaviors manifested by children in different settings.²³ Diagnoses are performed by clinicians, so it is to be expected that a comprehensive clinical assessment may or may not match an informant’s subjective point of view. Discrepancies arising from moderate correlations among multiple informants may contain meaningful information.³⁷ Hence, what a teacher observes in the classroom is not necessarily opposed to, but might complement, parents’ observations in the home or other contexts. Indeed, these differences between raters could help clinicians identify and understand the factors that enhance or diminish the frequency of ADHD behaviors.

Regarding the sex of the children in this study, both groups of informants reported that more boys than girls met the criteria for ADHD, a finding consistently reported in the literature, and one which indicates that this neurodevelopmental disorder is more common in the former.^4,38 Interestingly, no effect of sex on inter-rater agreement was evident.

According to our data, school grade did influence inter-rater agreement, which was observed in 55% of the ADHD cases among the first- and second-graders (G1) compared to only 45% in the ADHD cases for the sample as a whole. In addition, G2 obtained the lowest agreement, mainly in the ADHD group. However, we recovered fewer questionnaires from the G2 teachers, and the G1 teachers tended to return more questionnaires for non-ADHD children. Clearly, these factors could bias the results of our agreement analyses, though it is important to note that, when considering the non-ADHD cases, agreement remained stable across all three school grade groups (84–85%).

We expected higher inter-rater agreement for the younger children in light of reports that ADHD behaviors tend to decrease with age.^{17,19,20,29,39} However, higher agreement in the first years of school could also reflect a greater level of communication or exchange of information between parents and teachers.⁴⁰ In the first school grades, children and their parents are in the process of adapting to the elementary school system, so this could lead to closer contact with teachers, while in the middle years of elementary school there is no additional reason to maintain such close communication. This fact could also be related to the lower agreement levels observed in grades 3 and 4.

It is important to point out that in all three school grade groups, parents reported more cases that met the ADHD criteria than teachers. This finding agrees with previous studies.^{9,15,21,24–26} Yet, in a context more similar to ours,²⁶ teachers reported a greater number of ADHD cases than parents, suggesting that higher behavioral demand in schools might enhance the noticeability of ADHD traits,¹⁰ though this was not the case in our study. Variation in the frequencies of ADHD behaviors reported by parents and teachers (parents>teachers or teachers>parents) could also be explained by cultural practices in the family and school contexts,^15,41 including the amount of time spent with the child.

The analyses of individual symptoms showed low inter-rater reliability for most items. Contrary to our expectations, however, inattention symptoms showed higher frequencies obtained through the AND-rule – that is, when both informants agreed – among the children identified as meeting ADHD Criterion A. For the subsample of children that did not meet the ADHD criteria, both parents and teachers most often identified the hyperactivity symptom “(f) often talks excessively”⁴ (p.60). This behavior might be observed naturally by parents and teachers in the open population, but is neither exclusive to, nor indicative of, ADHD. Meanwhile, the inattentive symptom “(h) is often easily distracted by extraneous stimuli”⁴ (p. 59) was the one reported most frequently by parents and teachers among the ADHD children. It was also the symptom with the highest number of coincidences observed upon applying the AND/OR rules. This inattentive symptom, then, might be a sensitive trait for ADHD.

Previous studies have suggested that hyperactivity symptoms are more prominent (noticeable) and, therefore, more easily detected by adults.^9,14 Positive predictive values (PPV) (ie, the probability of a symptom being reported as “present” by the teacher given a previous positive report by a mother) has been reported¹⁵ for the same inattention symptom that we found to be most often reported among the ADHD cases; that is, (h). Thus, this symptom seems to be especially relevant for ADHD in the school setting. Regarding hyperactivity, the aforementioned study¹⁵ found that the symptom with the highest PPV was “often fidgets with hands or feet or squirms in seat”. However, symptom “(f) often talks excessively”⁴ (p.60) was the symptom with the highest frequencies for children that did not meet the ADHD criteria. It is also possible that perception of hyperactivity symptoms is dependent on cultural practices. Talking excessively might be perceived as a common but noticeable behavior in our culture, not necessarily related with behavioral disorders, but perceptions could differ in other cultures. In some cultures for instance, hyperactive behaviors are endorsed by parents with a positive meaning of masculinity in the case of boys, and thus, not rated as problematic.⁴² Cross-cultural analyses of symptoms would allow us to study this matter in much greater detail. Finally, impulsivity traits were perceived less frequently by both parents and teachers. This finding was expected, given that the behaviors listed are potentially related to rules established in the family or school context. Since very few studies have reported agreement on symptoms between informants, additional research in this field is required. Future projects should include analyses of individual symptoms in order to reach decisive conclusions.

Limitations

In future studies, it will be important to specify whether the informant was the child’s mother or father, since some research has proposed that mothers tend to rate their children as having more problems than fathers^15,41 and that fathers, but not mothers, tend to be more tolerant of disruptive behaviors performed by boys than girls.⁴³ We were unable to account for such factors as teachers’ experience, parents’ age or school level, and socioeconomic status, so we cannot rule out the possible that factors of this kind could explain discrepancies between raters. In addition, a common feature of studies of this nature is the difference in the number of questionnaires filled out by each group of informants. Obviously, parents were only asked to answer their own child’s questionnaire, while teachers might have had to fill out as many as 10 surveys, though we spread our sample out through 79 school groups. This discrepancy in the number of questionnaires received from each type of informant could raise concerns regarding the non-independence of observations.

Conclusion

The present study found low agreement between parents and teachers. The sex of the children did not appear to affect agreement. The analysis across school groups showed the highest agreement for the younger children. The analysis by individual symptoms suggests that inattentive symptoms related to external stimuli distractibility are sensitive indicators of ADHD in both the home and school settings, whereas hyperactivity and impulsivity symptoms were found to have lower diagnostic value in this Mexican context.

Based on these results, we would emphasize the importance of compiling complete clinical histories as exhaustively as possible in a clinical setting. Our data suggest that (i) interpretations of data from two informants cut several ways, and (ii) clinicians should consider that discrepancies between raters can be indicators of how behaviors are expressed in different settings, of what each informant is perceiving, and of which conditions or contingencies might reinforce or diminish the expression of such behaviors.²³ Finally, since behaviors are qualified and quantified through the “cultural-eyes” of the informant, and we detected that some “ADHD traits” are often observed in children in Mexican culture (vgr “often talks excessively”), the diagnostic value of each behavior (symptom) may vary across cultures and, hence, must be taken into account accordingly.