Pragmatic abilities in children with neurodevelopmental disorders: development of Pragmatic Abilities Questionnaire based on the Rasch rating scale model

Introduction

Pragmatic abilities comprise one of the underlying components of social communication^1–4 that can be defined as a set of abilities used for the proper utilization and interpretation of language in different communicative situations.⁵ Some of these abilities can be impaired in children with Autism Spectrum Disorder (ASD),^6–11 Intellectual Disability (ID),^12,13 Attention-Deficit/Hyperactivity Disorder (ADHD),^14–16 Learning Disabilities (LD),^17–22 and Specific Language Impairment (SLI).^23–25 With the introduction of a new diagnostic category to the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5),²⁶ namely Social (Pragmatic) Communication Disorder (SPCD), and the proposals for adding Pragmatic Language Impairment (PLI) to the International classification of Diseases-11 (ICD-11),²⁷ the emphasis on pragmatic abilities and methods for assessing these abilities in children with pragmatic impairment has been growing.

Some checklists are available for examining the various aspects of pragmatics within the theoretical framework from which the instrument was developed.⁴ Some of these checklists focus only on one aspect rather than all the aspects of pragmatics comprehensively; examples include the Discourse Skills Checklist²⁸ and the Parent-Report Rating Scale.²⁹ The Pragmatic Protocol,³⁰ on the other hand, has considerable merit as a checklist and covers various aspects of pragmatics, although its results are mainly dichotomous and do not reveal a range of behaviors, and also, the tool has not yet been validated.^31,32 In addition, some of the available tools lack adequate reliability; for example, the Targeted Observation of Pragmatics in Children’s Conversations (TOPICC).^33,34

The Children’s Communication Checklist-2 (CCC-2)³⁵ is probably the most widely-used questionnaire in the linguistics field.⁴ Although It is norm-referenced and well-validated, there are some accepted limitations on its diagnostic accuracy and also some risk of inconsistent parent reports.^32,34 While some studies have found that pragmatic impairment exists separately from other language impairments,^26,36 this checklist is specifically designed to check for pragmatic language problems in children with an identified SLI⁴ and includes scales for language functions (speech, syntax and semantics).

Several points have to be considered in the assessment of pragmatic abilities. First, pragmatics are influenced by culture, and cultural differences must therefore be considered in the assessment of pragmatic language.^3,4,34,37 To identify cultural differences in pragmatics, there is also a need to avoid developing instruments that are not ecologically valid,³⁸ although no culture-based instruments exist for measuring pragmatic abilities in Farsi-speaking children. Second, there is a need for valid and reliable instruments for use in children with pragmatic impairment.^31,39 Third, a good measurement instrument must include a range of perspectives (interviews with family members, teachers and specialists involved with the child) and contexts (observations in daily and clinical environments) in order to gather information and get a comprehensive view of children’s pragmatic abilities.^4,32 And finally, because pragmatic abilities include the comprehension and use of speech acts and communicative intentions, the organization and management of discourse, adapting communication to the social context, understanding nonliteral language, inference of the speaker’s intended meaning and nonverbal behaviors,^4,5,32,40 pragmatic assessment measures have to consider the various aspects of pragmatics more comprehensively rather than focusing on one or two components.

Current study

Given the points noted for the development of a new instrument to measure pragmatic abilities, four main sources were used in this study, including a literature review of the theoretical basis of pragmatic abilities, the available pragmatic measures, interviews with experts and the mothers of children with neurodevelopmental disorders about pragmatic abilities and the observation of their children, and finally the Rasch model. Rasch analysis is based on the probability theory and is a powerful tool for evaluating construct validity by which researchers can evaluate the validity and reliability of a measurement more thoroughly compared to the traditional analysis.^41,42 This paper reports on the development of a new instrument for measuring pragmatic abilities for application in children with neurodevelopmental disorders aged five to nine years, especially those with normal structural language skills (such as children with SPCD).

Methods

This study was carried out in three stages:

Stage 1

Stage 1 consisted of a qualitative study. The aim of this stage was to generate items using literature review, the currently-available measures, interviews with experts and mothers and the observation of the mothers’ interaction with their children.

Participants

The interviews and observations were conducted by the first author. The interview framework was semi-structured. Fourteen experts, including speech and language pathologists (3), linguists (2), teachers of children with special needs (2), occupational therapist (2), child psychologists (2), a child psychiatrist (1), a school counselor (1) and a rehabilitation counselor (1), who had five years or more of professional work experience, plus 15 mothers of children with ASD (4), ADHD (8), SLI (2) and SPCD (1) with a mean age of 76±12.07 months were selected through convenience sampling and interviewed and their conversational interaction with the children were observed at home and in the examination room of a rehabilitation center based on the Pragmatic Protocol.³⁰ The duration of the interviews was 30–60 min and the duration of the observations 15 min. A team consisting of the authors as well as a linguist (1), a child psychologist (1) and a speech and language pathologist (1) transcribed the recorded interviews and analyzed the interview transcripts based on thematic content analysis. Ultimately, 119 items were identified.

Stage 2

The aim of stage 2 was investigating the content validity of the items, which is the extent to which an instrument measures the intended concept and is supported by evidence from qualitative studies which demonstrate the items and domains of the instrument are appropriate and comprehensive in relation to its intended concept, study population and application.⁴³ Without evidence of content adequacy, an instrument may include construct-irrelevant variance or underrepresent the construct^,44 In this stage, a questionnaire consisting of 119 items was prepared and the discussed experts rated the items.

Participants

The questionnaire was evaluated by ten experts, including speech and language pathologists (2), linguists (2), a teacher of children with special needs (1), an occupational therapist (1), a child psychologist (1), a child psychiatrist (1), a school counselor (1) and a rehabilitation counselor (1) with five years or more of professional work experience. Judgment was based on a 5-point Likert scale in which 1 indicated a “very irrelevant item” and 5 a “very relevant item.” Finally, after the professionals gave their feedback on the questionnaire, the items were reduced to 80.

Stage 3

Stage 3 involved the drafting and analysis of the psychometric properties of the main questionnaire.

Participants

Convenience sampling was used in stage 3 to select 185 normally-developing children and 120 children with neurodevelopmental disorder over a 12-month period (September 2017–2018). The sample size was adequate for supporting the planned Rasch analysis,⁴² because Person (sample, test) reliability is independent of sample size in this analysis. That is, reliability depends chiefly on sample ability variance (wider ability range = higher person reliability), and If the additional person sample is more central than the original sample, then the person separation will reduce.⁴⁵ Sample size depends on many items in every study; for example, the number of items, the location of the items along a trait, the overlap of the items along a trait, the distribution of the respondents along a trait, the number of respondents, the targeting of the items to the persons along a trait and the goal of the instrument.⁴² This study was conducted in accordance with the Declaration of Helsinki. Also the Ethic Committee of Tehran University of Social Welfare and Rehabilitation Sciences approved the study proposal, and written informed consent was obtained from all the parents.

A total of 185 normally-developing children (96 boys and 89 girls) were recruited from six preschools and schools. The children were aged 60–108 months (boys: M=83.98, SD=13.99; girls: M=85.37, SD=15.06) and had no history of special educational placement. They were all in the appropriate grade for age and their parents reported that they had no developmental concerns. A total of 120 children with neurodevelopmental disorders (ASD, SLI, ADHD and SPCD) were recruited from 11 rehabilitation centers. These children were also aged 60–108 months (ASD: M=80.04, SD=15.06; ADHD: M=89.72, SD=11.91; ADD: M=79.20, SD=15.91; SLI: M=68.71, SD=5.76; and SPCD: M=86.15, SD=15.64). They had been diagnosed by a multidisciplinary team consisting of a child psychiatrist, a child psychologist and a speech-language pathologist based on the review of all the tests, observations, medical and family histories and extensive group discussions using the DSM-5 guidelines.

For the children with ASD (n=24), the Persian version of tests including the Gilliam Autism Rating Scale (GARS),⁴⁶ the Stanford–Binet intelligence scales- 5th edition (SB5)⁴⁷ and the Test Of Language Development (TOLD-P3)⁴⁸ were used. The children with ASD were excluded if their IQ scores were more than 2 SD below the normal mean.

For the Children with SLI (n=21), the GARS, SB5, TOLD-P3 and the Persian version of the Diagnostic Evaluation of Articulation and Phonology (P-DEAP)⁴⁹ were used. These children were identified as cases of specific language impairment when they met the following criteria:⁵⁰ Spoke Persian as their primary language, had no history of intellectual disabilities, autism, neurological problems or sensory neural hearing loss, had a performance IQ greater than 85 on the non-verbal subtests of the SB5, and a performance more than 1.25 SD below the mean on the TOLD-P3.

For the children with ADHD (n=49), the SB5, TOLD-P3 and the Persian version of the Swanson, Nolan and Pelham Rating scale (SNAP-IV)^51,52 were used. A total of 24 of these children met the criteria for the predominantly inattentive type (attention deficiency disorder) and 25 had the combined type. They did not have sensory neural hearing loss and were excluded if they had IQ scores greater than 2 SD below the normal mean on the SB5.

For the Children with SPCD (n=26), the GARS, SB5, TOLD-P3, P-DEAP and the Persian version of Children’s Communication Checklist Version 2⁵³ were used. These children did not have sensory neural hearing loss, intellectual disability or apraxia. The children with SPCD were excluded if they had a TOLD score 1 SD below the normal mean.

Table 1 presents the data on the IQ and language scores of the children.

Table 1 The characteristics of children with neurodevelopmental disorders

Instrument

The Pragmatic Abilities Questionnaire (PAQ) was developed with 119 items. Thirty items were removed after the review of the tool’s content analysis by ten experts. The remaining 80 items were administered in written format to assess the pragmatic abilities of the examined children. The response categories for all the items included “never”, “very rarely”, “rarely”, “occasionally”, “very frequently” and “always”. The six-point scale (0–5) used for scoring the items was reversed for the negatively-worded items.

Data analytic plan

Stage 1

Data analysis started with thematic content analysis in stage 1, which involves extracting themes from the interview transcripts and attempting to verify, confirm and qualify them by searching through the data and repeating the process for identifying further themes and categories.^54,55

Stage 2

Content validity refers to the extent to which the items on a test are fairly representative of the entire domain which the test seeks to measure, and a standard method for assessing this measure is to seek the views of experts with expertise in the content of the test. Good items have high scores, indicating the high agreement among raters.⁵⁶ Content validity was established in this stage and the items were rated.

Stage 3

The analyses performed to assess the psychometric properties of the PAQ were based on the Rasch rating scale model. Rasch developed a measurement model for responses to dichotomous items that shows the procedures for constructing measurement instruments and documenting the measurement properties of these instruments.^57,58 This model also incorporates a method for ordering persons based on their abilities and also ordering the items based on their difficulty. The probability of correct responses increases with the level of the item’s difficulty and the individual’s ability.⁵⁹

Unidimensionality

The Rasch model is based on the assumption that effective measurement involves the examination of only one attribute at a time, which is referred to as unidimensionality.^42,59 There are two ways to test for unidimensionality; first, examining the mean-square infit and outfit statistics, and second, conducting a Principal Component Analysis (PCA) of the standardized residuals.⁵⁸

Construct validity

Construct validity is shown by fit,⁶⁰ which consists of Information-weighted fit (Infit) Mean Square (MnSq), Outlier-sensitive fit mean score (Outfit) MnSq and Z-score-standardized (Zstd) fit statistics. The items were considered a misfit for the model and deemed not to belong to the constructs when MnSq was >1.4 and were associated Zstd > positive 2.0.⁵⁹

Reliability

This model contains two reliability indices, including person reliability and item reliability, and also two separation indices, including person separation and item separation.⁴² The item and person reliability was estimated as >0.7, suggesting a good reliability. The item and person separation indices were also estimated as >2, suggesting a good separation. A low person separation implies that the instrument may not be sensitive enough to distinguish between high and low pragmatic abilities. A low item separation implies that the person sample is not large enough to confirm the item difficulty hierarchy or the construct validity of the instrument, and a low person separation with a relevant person sample implies that the instrument may not be sensitive enough to distinguish between high and low performers.⁴⁵

The person-item map (Wright Map)

The person-item map (also called the Wright map) is an accurate graphical representation of the relationship between measures of persons and items⁴² that is used to explain the locations of the respondents and the item thresholds on the construct. It can be calculated based on the difficulty of the test items and expressed using the same linear scale used for expressing the person measure.^57,61

Systematic steps based on psychometric inquiry were used to obtain an appropriate and valid subset of items. The items with poor psychometric properties were excluded based on the item Infit and Outfit and item-measure (point biserial [PTBIS]) correlations. Another Rasch analysis was performed after eliminating the misfitting items and persons. This process was repeated until the entire remaining respondents and items fit. Winsteps (version 3.64) was run to evaluate the psychometric properties and hierarchies of item difficulty placed on the person-item map.

Results

Stage 1

The thematic content analysis, observations, literature review and study of available pragmatic measures led to the generation of 119 items.

Stage 2

Content validity was established according to the experts’ understanding of the construct.⁶² The items of the questionnaire were rated by ten professionals based on their relevance to pragmatic abilities using a rating scale from 1 to 5. The median of the items was calculated and 39 items were omitted due to their low median (4<). Instances of the eliminated items include “He/she talks to himself/herself”, “He/she uses many long pauses, repetitions and revisions when talking”, “He/she excuses himself/herself when interrupting the speech of another”, “He/she may hurt other children” and “He/she is silent when he/she should respond to questions”. To estimate the rater reliability, the Intraclass Correlation Coefficient (ICC) was calculated and reported as 0.78 (P<0.01), indicating the high inter-rater agreement about the PAQ items.

Stage 3

After the original items of the PAQ were reduced to 80 based on the experts’ views, further item reduction was performed through RSM measures. This process reduced the questionnaire items from 80 to 40. The items were considered poor representations of the constructs of the PAQ when the Mean Square (MnSq) was above 1.4 and were associated with a Z-standardized (Zstd) greater than positive 2.0, thus exceeding the “fit” criteria. The PAQ items were therefore reduced to 40 (Table 2).

Table 2 The item characteristics of the final Pragmatic Abilities Questionnaire

Unidimensionality

To examine unidimensionality, the Rasch PCA was performed on the residuals of the items. As shown in Table 3, 63.8% of the variance in the PAQ was explained and the criteria for unidimensionality were met (>40%). The 40-item PAQ was therefore deemed unidimensional.

Table 3 The main component analysis of the Pragmatic Abilities Questionnaire based on the Rasch model

Construct validity

The construct validity of PAQ was examined so as to reduce the misfitting items based on the Rasch model. Item reduction was performed on the 80-item draft of the PAQ to reach an appropriate and valid subset of items. The infit and outfit mean-square values indicate the degree to which the items and persons fit the model. Infit mean-squares are sensitive to inliers and outfit mean-squares to outliers, which suggests unexpected response patterns.⁶³ The infit and outfit values for the 40 items suggest that the PAQ meets the criteria for the construct validity assumption of the Rasch model. Construct validity is also supported by the positive item (point biserial [PTBIS]) correlations (Table 2).

Reliability

The reliability of the PAQ was excellent given the item measure reliability of 0.99, which suggests that the items can be discriminated from one another based on their difficulty. The person measure reliability was 0.97, which suggests that the scale can discriminate between pragmatic abilities based on their estimated pragmatic level. The item separation index of 10.91 indicates that the PAQ items have the potential to be categorized into ten levels. The person separation index was 6.03. This index is an estimate of how well one can differentiate persons in terms of the measured pragmatic abilities; that is, it estimates the replicability of person placement across the other items measuring the same construct⁵⁹ and shows that the participants can be divided into six groups based on their pragmatic abilities level, although these groups are not the same five groups of different disorders (ie groups of children with ASD, ADHD, ADD, SLI and SPCD) and children with normally developing group. In other words, there are six response patterns, but it is not the case that each response pattern is related to only one type of neurodevelopmental disorder. For example, children with autism may have response patterns similar to children with ADHD. The separation index cannot be solely attributed to the type of disorder (ie ASD, ADHS, ADD, SLI and SPCD); rather, it is affected by factors such as sample pragmatic ability variance (which depends on the disorder, gender and age), length of the test, the number of categories per item, and sample-item targeting.⁴⁵

The person-item map (Wright Map)

The person-item map demonstrates a continuum of pragmatic abilities generated by the measurement of the individuals’ ratings of each item (Figure 1). This map indicates that items and persons are appropriately targeted.⁴⁵ The items are placed on the right-hand side and the persons on the left-hand side of the dividing line, and the number of persons are indicated by the symbols #=2 and .=1. The item estimates range from approximately +2.5 to −3.2 logits. The more difficult items appear toward the top of the map (item 36 is the most difficult item in this set) and the easier items toward the bottom (item 6 is the easiest one). There are no gaps between the items. The person ability estimates show a wide range of abilities. The mean measure for person ability is higher up on the left-hand side of the dividing line; that is, the mean person abilities is higher than the mean item difficulties.

Figure 1 The person-item map of the Pragmatic Abilities Questionnaire.

Discussion

Cultural and linguistic diversity and valid and reliable instruments are important factors in assessing pragmatic abilities. This study was conducted to develop and test the reliability and validity of a new instrument for assessing pragmatic abilities in children with neurodevelopmental disorders aged five to nine years, especially those with normal structural language skills; for example, children with SPCD.

The PAQ developed in this research consists of 40 items that assess pragmatic abilities and have good psychometric properties. The infit and outfit mean-square values and the positive item-measure (point biserial [PTBIS]) correlations prove the construct validity of the instrument. The results showed that the PAQ has an acceptable construct validity (Table 2). The dimensionality (ie the structure of a specific phenomenon) of the PAQ was investigated as a necessary stage in gathering evidence to support the validity of interpretations based on the total score of the instrument, particularly when assessment development and analysis are carried out within the Rasch measurement framework.⁶⁴ All the residual items of the questionnaire measured a single construct and explained 63.8% of the variance (Table 3), which is an important finding because unambiguous interpretations require scores that represent a single defined attribute. That is, scores obtained on a scale that is used for measuring a single variable should not be appreciably affected by varying levels of one or more other variables. In addition, if scores do not represent a common line of inquiry, then it will be unclear whether two individuals with the same score can be considered comparable.⁶⁵ Unidimensionality (the existence of one underlying measurement construct that accounts for variation in examinee responses) indicates an instrument that is less ambiguous, more precise and ready for use with promising statistical methods, such as structural models.⁶⁶

The PAQ has an item measure reliability of 0.99 and a person reliability of 0.97 with a separation index of 6.03. Person separation and item separation indexes are a superior way for assessing an instrument’s function, since they are not associated with any ceiling effects (ie their minimum value is 0 and they have no maximum values). Item separation and person separation are additional techniques by which one can assess (1) how well a set of items is able to differentiate between respondents and (2) how well the set of items is able to be differentiated by the group of respondents.⁴² The person separation index (6.03) shows that a set of items is able to differentiate between six groups, and the PAQ was found to be sufficiently sensitive for distinguishing between high and low performers. The item separation index (10.91) for this instrument showed that the person sample is large enough for confirming the item difficulty hierarchy of the PAQ or its construct validity.

In the person-item map, the person scores and the difficulty level of the items showed that the strong persons answered the easy items correctly and the weak persons answered the difficult ones wrongly. It is worth noting that the difficulty level of one item was lower, such that the weakest person was also able to answer it (item 6). Additionally, there were no items for the persons with the highest ability measures.

The findings of this study suggest that the PAQ has a good construct validity and all its items measure pragmatic abilities and that the Rasch measurement model is a useful method for developing new assessment measures to use in psychological and psychiatric settings. Similar results have been reported by some researchers.^59,67–69

Conclusion

It is imperative for professionals working with children to have access to effective pragmatic assessments, because pragmatic impairment may cause psychological problems, difficulties in social relationships and difficult attainment of educational and academic goals.

This study demonstrated that the PAQ has acceptable construct validity and reliability. Further research to validate the PAQ in a broader sample of children with neurodevelopmental disorders is required.